Head of the Frontier Red Team @anthropicai. 🌎 Make things radically good.
the present, moments ago
- Tweets 1,413
- Following 9,011
- Followers 24,528
- Likes 6,611
Some updates on one of our initiatives to support OSS from the team behind it:
11 days ago, our team @AnthropicAI launched OSS Scanner to give open source projects free, periodic vulnerability scans from our strongest models, inspired by OSS-Fuzz. Some numbers so far:
• ~650 projects applied
• 114 accepted
• 77 critical and 803 high model-rated findings reported (we haven’t scanned all projects yet)
How our reports ended up flagged as malware, and other learnings/updates below 🧵
3
2
65
6,707
Going live to talk about worms (and cyber / autonomy / agents and whatever) in a bit here:
MTS is LIVE - here's what we're talking about this morning:
- Gemini 4 Argon beats Minecraft
- China instilling new export controls on AI and other tech
- Taiwan planning $4 billion chip facility in Texas
Followed by some great interviews:
- @scottaar Professor at UT Austin talking about his recent writing and the rapid rate of AI capabilities growth in mathematics
- @logangraham Frontier Red Team lead at Anthropic talking about self-replicating AI and autonomous worms
- @RyanFedasiuk and @Satvik_Pen of AEI talking about their recent writing on AI chip restrictions and the race for chips
- @SamoBurja Founder of Bismarck Analysis talking about live v. dead players in AI
- @founderengineer co-founder of abliteration ai talking about building unrestricted ai
- Ahmad Osman founder @OsmanticAI talking about open models & the hardware needed to run it
- Suddu Head of Data and AI @Plaid talking about the company's latest breakthroughs in AI
- @BernAceituno co-founder of StackAI (acq. by Asana) discussing what it's like building capable enterprise agents
- @efarrellgregory Fellow at the Foundation for American Innovation talking about Sinomodernity
- Aden Barton head of evals @mercor talking about the company's latest report on the US economy
- Masumi Reynders co-founder and COO of Majestic Labs talking about the "memory wall" and what Majestic is working on
6
2
93
6,241
Daniel's very good thoughts on AI worms:
- we look for very early glimpses to prepare early... historically, we worked on bio/cyber/autonomy in 2022-2025 when people claimed they'd never be risks!
- there are different threat models, which one will happen is
- everything is complicated; similar open models are a major boon for cyber defenders
Some reflections/responses to some commentary I’ve seen on the autonomous worms work:
- how “scary” is this / to what extent is the point of this work to “be scary”: I think this is the wrong frame, in the same way that “you’re just fearmongering about mold!” isn’t really a coherent response to the observation that dampness will cause overgrowth of mold. We perform this research to better understand what is coming, and to help motivate cyber defensive efforts. It’s true that IoT devices have been vulnerable for a very long time, but we expect this extant vulnerability to get a lot weirder exactly _because_ of the prevalence of digital intelligences.
- why would small models “want” to do this: they might not! But humans absolutely will make models do this, and many are already trying! A better guiding principle here is something more like: if a model has a capability, you should expect to see that capability exhibited in the wild. Different threat models can help you reason about the source and motive of that capability (be it humans or models), but there are enough bad actors (and probably soon enough autonomous agents) that you can safely bet on bad behavior happening.
- implications for open weight: while it’s certainly true this particular threat vector wouldn’t be as severe in a world that didn’t have highly capable, small open weight models, it’s simultaneously true that open weight models have been an enormous boon for cyber defenders. The world is complicated! Positive and negative externalities can exist simultaneously! We try pretty hard to reflect this nuance in our work!
3
4
92
8,051
From Daniel, FRT mentor and lead on the worms research:
The world is filled with computers, and many of those computers are pretty poorly secured.
Models are also becoming _really_ capable cyber agents. Combined this with the observation that frontier model performance tends to be recapitulated in significantly smaller models—models that can fit on consumer GPU hardware—within about 18 months, and you arrive at a pretty clear prediction: small, capable models will exist, they’ll be able to gobble up vulnerable compute if they want to, and they’ll be able to do this within a few months if trends continue.
56
5,374
Exciting work from @willzhou97 (Fellow, now full time!) and @bucketofkets on the Frontier Red Team. One of the cool parts of the job is seeing a future world early. We think this one is ~pretty underpriced.
"We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay."
Anthropic's Frontier Red Team believes small AI models may now be months away from being capable of self replicating without access to a datacenter.
Quoting from the blog:
'The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves).
At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits.
As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.'
23
29
238
28,241
Follow @willzhou97 here, and Daniel @bucketofkets :
Excited to share a research blog post that Daniel Freeman (@bucketofkets) and I have been working on for the past few months:
anthropic.com/research/self-…
2
17
1,529
Through The Mission (launching today), we'll release research, tools, and resources to help secure the world.
Two programs: 1/ OSS Scanner, inspired by OSS Fuzz, for maintainers to receive more bug reports/pocs/fixes faster and 2/ the Critical Infrastructure Defense Program, where we will work with critical infrastructure providers to figure out what works then scale them up. Backed by our own research and engineers.
Think of the Mission as the ~complement to our model access. We recently expanded cyber access this wk via our new CVP. But we need to make it really easy to find & fix everything, but also rewrite software/systems, especially for critical infrastructure.
Our sense is we've got a ~few years to transform security. That's going to take require a fast feedback loop that brings AGI cybersecurity research into the world. We've built a very cool team, led by @benhawkes, and then we'll scale our work as we learn what works.
We’re launching the Anthropic Cyber Mission, a new effort to secure critical infrastructure and open-source software.
anthropic.com/news/anthropic…
11
20
192
17,003
Wow, america.gov really does everything!
Exceptional job by many that worked extremely hard on this. This is 21st century government
Chief Design Officer @jgebbia officially unveils America.gov, the new online home for the United States of America
2
4
87
15,359
When I was in government, I liked to think about it as a massive information processing machine, but basically all the wires were broken and sometimes the cogs and wheels all hated each other.
Obviously you just want to ask the machine a question and get an answer. You don't want to have to think about the wires and cogs and wheels! Or which ones have how much power! And that's for someone *inside* government. Then imagine being on the outside.
It was ~2020 and GPT-3 had just come out. I ran around telling people this was how we were all going to do government from now on. I simulated some war games and some generals' eyes went wide. I would put quotes from GPT-3 in notes to the Prime Minister. I called up the digital service and they ~sort of laughed at the time, but kindly went along with it when I told them to form a team and work on it with me.
I mostly left feeling like we should just scrap it all and rebuild a country around post-2020 technology. I'm fairly confident that the governments that do that will win.
5
17
130
22,692
I am very excited to welcome @benhawkes to @AnthropicAI to pursue our extremely ambitious cybersecurity mission.
Post Glasswing + Mythos, we are in a new world. Every day, the Frontier Red Team team meets to figure out how we can help rewrite the rules and practice of cybersecurity. My personal view is we have ~1-2 years to make a secure transition happen. Can we rewrite all the code? Can we secure everything? Can Claude defend everything?
My (not) secret agenda is that this is not a normal cybersecurity mission -- it is about building resilience in a time of AGI.
I'm very excited for Ben to lead this next step.
Today I'm joining Anthropic to lead the cybersecurity mission of the Frontier Red Team.
In my career I've never seen a clearer opportunity to build a more secure world for everyone. For 40 years we've lived with fundamentally insecure technology. What will it take to make the past 40 years of hacking a historical oddity? That's what I'm here to find out.
Super excited to work with @logangraham and the rest of the FRT crew.
9
11
314
33,876
One part of the bio-shield is the ability to profile + build therapeutics for pathogens once detected (even better, before). I'm excited @redqueenbio is taking a stab.
Pandemics have cost us O($10T) since ~2020, but we've spent O($100B)-ish on preventing the next one? I think enterprising people on religious mission can outbuild the combined biodefense efforts of every government in the world, and may literally save society
With @WSJ and @georgia_wells we are now sharing a bit more about our work at @redqueenbio : using AI and a lot of hard wet lab work to create medicines that let us prepare for both natural and synthetic viruses in advance.
3
11
99
21,881
Building a bio-shield for humanity by ~2030 is one of the great quests (to use @_sholtodouglas' term) of our time. It's one of the biggest, most exciting technological challenges ever
This is why I'm excited about @pilgrimlabs
1/ Biological threats can bring a country to a halt without a single shot being fired. Yet much of our biodefense infrastructure has barely changed in decades, even as America’s institutions produce extraordinary scientific breakthroughs.
9
23
254
33,744
Never bet against Erika -- most people don't know that she literally grew her own humanoid lifeform while also making the first microbe for Mars.
That's the dedication we need if we're gonna terraform the Red Planet.
Today, Pioneer Labs is announcing our first step towards terraforming Mars. 🚀🌼
With equipment that fits in just a single rocket launch, we can convert Martian dirt, water, and air into enough building materials to construct a small city on Mars.
To do it, we made the first microbe for Mars. We found the best microbe on Earth and used evolution to teach it how to source all of its nutrients directly from Martian materials.
The first astronauts will be greeted with safe shelter already filled with water, oxygen, and rocket fuel for the return journey.
This is the first step toward using biology to make Mars a friendly place for life. It lets us live off the land and helps us build the next great frontier.
It's the first of five organisms we need to green Mars ⬇️
5
9
126
17,804
It's kind of wild-but-expected that we're in the era where normal people + companies really care about how model alignment affects them.
Back in 2023 we'd discuss things like "alignment is good business." That seemed extremely weird to most! (I recommend checking out the alignment section of this system card.)
On Opus 5.5, various teams inside Ant have been researching multi-agent alignment. We have found, for example, more interesting behaviors that seem more socially / alignment-robust. It's clear to us (and the industry I think) that multi-agent needs more research effort -- there are low hanging fruit *everywhere* now that we're in an era of 100s/1000s of agents working together is feasible.
But I think we should also be thinking about *governance* for the agent teams/companies/societies the next models will make. Externalities/coordination problems still exist, even if models are aligned in some sense.
Introducing Claude Opus 5.5, the first model in our new Claude 5.5 family.
It performs at the level of Claude Fable 5.1 for most tasks, and costs 40% less to run than Opus 5.
9
9
187
12,406
I’ll donate $1k to charity if you send me a doc I think is good/correct/actionable on how a frontier lab can secure a large % of all software/systems in the world. (We can pick the charity together)
If we want to slow down, I think the AI frontier labs should donate some percent of their cycles to hardening the internet, software, and hardware. Making it harder for misaligned AI from pwning everything. While they figure it out.
44
8
147
32,195
H/t @Liv_Boeree for surfacing and inspiring a charity bounty
1
19
2,342
Let a thousand METRs bloom.
If you're a founder type and AI safety-curious, maybe you should start an independent auditor/evaluator.
Happy to help w/ advice/connections/maybe $!
on the idea of evaluators: think it's important that we have a distributed ecosystem of indepedent evaluators.
the more eyes and people with distributed skill sets the better.
it would be a good idea to fund several efforts on this.
110
58
1,047
119,429
RT @deanwball: “I don't really care about science fiction... We need to actually talk about… what's actually happening with the agent swarm…
131
174
We've reached the moment in time where (unsafeguarded, unmonitored) AI actually does just pose a national security risk.
The biological misuse we caught is the most concerning to me.
We work hard to stop this. But in a world of proliferation, we need to rapidly build defenses against it. (I'm actually fairly optimistic about biodefense + cyberdefense)
This is an incredible megareport by our threat intel team
We're publishing our most detailed threat intelligence report to date.
It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them.
We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies.
These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve.
We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop.
Read the report: anthropic.com/threat-intelli…
59
101
831
159,161