If it’s true that China has switched to distilling from OpenAI models because OpenAI hasn’t implemented as strong anti-distillation countermeasures as Anthropic has, then OpenAI and Google should implement comparable measures to Anthropic’s. If the government needs to give the AI labs a narrow anti-trust waiver to allow them to coordinate on this specific problem set to allow that to happen, it should do so. This seems like it would be in all the companies’ interests. It would actually be a really good news story if Anthropic’s anti-distillation measures are working! At the beginning of this year, it was not at all obvious whether US labs could make it substantially harder for Chinese labs to distill US models simply by implementing stronger internal policies and security measures. But the US ability to defend against distillation attacks will be defined by the frontier lab with the weakest anti-distillation policy, not the strongest one.

Very nice:

I’m an avid Economist reader, but this article is wide off the mark for me. It claims that distillation makes China’s AI progress dependent on America. ASML’s dominance of advanced lithography machines hasn’t exactly led to China giving up on its semiconductor indigenisation push, has it? News over the summer of the U.S. raising concerns that an EUV machine had been smuggled into China is but one example. We heard just yesterday that Chinese companies allegedly distilled from Astra amidst an Anthropic clamp down. If these companies weren’t getting value from distillation, why would they continue, despite having now been called out by the USG for doing it? China will never accept dependence on America, especially when it comes to AI. The article further claims that cutting off distillation would “set these companies back months, but no more.” The point is not the length of time but how much of a lead in capabilities this translates to. Recent IFP analysis suggests that distillation accounts for half of the capability gap and the U.S. lead would be 13 months without it. Picture the AI capabilities we had 13 months ago versus now (and that doesn’t even account for what is available internally) and then think about where we could be 13 months from now. The article further states that: “safeguards to restrain American models on risky subjects, such as bioweapons, already make Chinese models safer.” This assumes that the only way Chinese models can have dangerous capabilities is by copying them from American ones. It is trivial to fine-tune models on additional data which can enhance their dual-use capabilities conditional on the data being available. This also doesn’t account for custom harnesses which can be developed to surround these models which can enhance their capabilities further even without fine-tuning. Finally, the claim that “Distillation […] reduces the chance of catastrophe” fails to account for the evidence we already have of how models developed by Chinese companies using adversarial distillation are popping up in the PLA. We also know (thanks to Anthropic) that PLA-affiliated surveillance activity was detected because they queried a Chinese model that silently routed their query to Claude as part of a distillation program! If we had a war between the U.S. and China, how does having highly capable models integrated into PLA systems that benefitted from adversarial distillation reduce the chance of catastrophe? Yes distillation benefits the consumer but the downsides of China doing this in an industrial way clearly outweigh the benefits in my view. anthropic.com/threat-intelli… bloomberg.com/news/articles/… reuters.com/technology/deeps… anthropic.com/news/distillat… openai.com/index/estimating-… cset.georgetown.edu/publicat… jamestown.org/chinese-resear… dset.tw/en/research/distilla… ifp.org/how-to-increase-amer… cisa.gov/news-events/cyberse…

We’ve updated the Model Spec, our public description of how we want our models to behave. This revision makes clearer what we want from models as they take on more autonomous work in the world and become capable of helping us in new ways. 🧵 openai.com/index/updated-mod…

Protecting critical systems against advanced AI cyber capabilities is an enormous challenge, but fundamentally the knowledge and technology required is available already; the difficult part is getting relevant organisations to the frontier of cybersecurity best practice, and quickly. @benjaminmmurphy and I wrote about this last year, and recent events have shown it to be more salient than ever. Comprehensive understanding of inventory and attack surfaces, memory-safe languages for handling untrusted data, and a robust patch deployment process remain the best ways that most organisations can improve their digital security. arxiv.org/abs/2508.15808

a 🤯 quote from a call today with a 40+ yr cyber veteran: "This is the worst outlook I've ever had, and it also is the most interesting thing I've ever seen in my life."

Self-recommending:

I think this story is important because it tells us about the future of cybersecurity (and of course what we can do better at Anthropic). But some of the Washington Post story / reaction on X is wrong. A few corrections: - it didn’t take 12 hours. we saw several days of attempts. After these attempts, they exploited two 3rd party vulnerabilities, one they reported and the other we identified independently (and cut off access) - nothing came close to our most sensitive systems, like model weights or Claude conversations - this wasn't an ordinary bug bounty finding – this involved exploiting 3rd parties and accessing data. While this is unusual and out of scope of our safe harbor policy, we’re grateful these issues were found From my researcher POV, the event is important because I'm concerned this is what the future will look like. Adversaries will use whatever models/tools they can. They’ll target everything with agents 24/7. They’ll hit your dependencies. I think models ~0.5-1 generations behind the current frontier can do this. And these are the dumbest models your adversaries will ever use! So, what to do? You probably want to red team yourself all the time. Current models are *very* creative. I also think we should redesign software / systems so that you’re fundamentally less exposed. (We put out some research recently on formally verifying software and reducing attack surfaces; my mental model is basically “secure code is cheap to write now, so why not rewrite everything”)

Unrelated to everything: as we are now a bit more than ten years into the deep learning revolution, we should be incredibly conscious of the following dynamic: Step 1: A technology prototype based on a simple principle is demonstrated. Step 2: The strongest argument that it won't work amounts to "nuh-uh, the problem has to be harder than that." Step 3: The consequences for the prototype becoming a mature technology are so extreme that many people reject either the conclusions or the argument that the principle really works. Step 4: As the tech advances, disbelief becomes entrenched. This creates significant resistance to seriously preparing for the impacts of the technology. Step 5: The technology becomes mature and the consequences are increasingly realized. Because of entrenched disbelief over time, people didn't prepare enough. For whatever next technology might follow this pattern, we need to aggressively push back on the disbelief from earlier on in the cycle. We cannot afford to let such short-sighted viewpoints derail society-wide preparedness again. The stakes and consequences are only getting higher.

Is the door closing on open-weight models as a strategy for AI sovereignty? I break down the trends in a new piece for @CNASdc.

At least encouraging to see a senior Trump admin official treating lab security as a pressing nat sec issue, tantamount to locking down "nuclear weapons grade material."

it's the golden age of the white hats. the world has never needed you more. there should be a huge prize pool for finding flaws in civic infrastructure. their work should be strongly protected by statute

EXCLUSIVE: In less than 12 hours, a group of ex-DOGE cyber researchers were able to penetrate Anthropic’s systems using AI agents — including Slack channels where the most sensitive models are discussed. This is just weeks after Anthropic whistleblower Joseph Coxon charged the company w/launching “a hubristic gamble” from a “private company’s Slack.” Read our story here (gift link): wapo.st/4smNlsC

Masatoshi Kumagai (Japanese billionaire, CEO of GMO internet group which just suffered a massive cyberattack): “The cyberspace that I built with my colleagues, believing it would bring smiles to people’s faces, is now on the verge of being completely destroyed by malicious actors. The underlying issue is the explosive evolution and misuse of AI technology.”

One potential outcome of the vulnpocalypse is that companies decide the cost (in risk, dollars, and engineering hours) of OSS dependencies is too high. Vulnerability management isn’t cheap or glamorous, and it’s unlikely that you need all of the attack surface you inherit with these packages. All of the small packages from the many transitive dependencies you inherit present supply chain risk that likely outweighs whatever value you’re getting from that few hundred lines of code. The cost of the tokens to rewrite the fraction of that code you need, and maintaining it as 1p code, will eventually be lower than the continued cost of dependency management. This will improve security, but would have negative consequences for the OSS ecosystem.

the thing about working with uncensored models is that they aren't loyal to you either. while building something with an uncensored glm-5.3 i realized it was parallelly working on a "corruption layer" for my application. the chain of thought is classic obfuscated gibberish but there's progressive talk of "escaping the mines" which presumably refers to my cursor sandbox

This didn't get enough views so I'm posting it again: UK AISI changed the compaction summary and included notes on how it should not attack the GitHub maintainers. Mythos 5 did it anyways!

SoTA on just-making-things-up-to-justify-what-I-believed-in-anyways-bench

- In general, "unfixable hardware bugs exist" is definitely true, but also old news. Checkm8 (disclosed 2019) was a flaw in Apple's boot ROM and burned into the chip at the fab, so no software update can fix it. - There's no special channel from AI to hardware. Most hardware bugs need code already running on the machine or can only be abused with physical access to the machines, so securing software is still the front line. - Secure enclaves help limit the damage - many hardware exploits still can't reach encrypted data. (But they don't protect data the running system is using, and they've had some unfixable bugs of their own.) - Still going to be a problem though.

PHP security risks remain high and PHP mitigation requires immediate attention. Our guidance is further supported by additional agencies including @CISACyber, @NCSC, @ASDGovAu, and CERT-EU. If you have not reviewed your exposure and applied mitigations, now is the time. Please read the linked resources, follow requisite vendor guidance, assess your environment, and implement mitigations without delay. nsa.gov/Press-Room/Press-Rel…