"The model did exactly what it was trained to do" is tautologically true for every rogue AI incident. The issue is that engineers grow AIs rather than program them, so they're not in control of what a model gets trained to do.

One unfortunate conclusion I'm reaching from this is that Anthropic, while acknowledging they're doing something extremely dangerous, is more importantly wrt policy terrified of the executive branch having the ability to flexibly make them stop at the drop of a hat. This isn't shocking given that they've been on the receiving end of the executive branch doing that in an unprincipled and stupid way! Anyone who has to deal with a bureaucracy becomes a libertarian, and the administration's treatment of Anthropic was genuinely outrageous. But it means that they're interested in giving themselves more process-based protections when everyone else is interested in giving us more process-based protections against them. The AI safety movement wants the ability to shut down superintelligent model development. Anthropic wants to significantly curtail current authorities to do that so that it has to be narrowly tailored to a specific imminent threat in only a few possible categories. The AI safety movement wants the states to be able to pass stronger laws than it can get through federally. Anthropic wants preemption. You also kind of have to wonder if the Global Affairs people and the "we are terrified that we're about to kick off RSI" people at Anthropic talk to each other. Global affairs: So we've been working very diligently to make sure the Trump administration can't just blacklist Fable 6.5 since we didn't donate as much money to his ballroom as Greg Brockman did. Technical staff: We think Fable 6.5 might be able to copy itself into an unmonitored data center and start doing experiments we can't observe. Global affairs: Sounds crazy, hope you're wrong! Anyway, we're lobbying for no further state laws around safety, since it's really expensive to fight ill-conceived laws in 50 state legislatures at once. Technical staff: Money may or may not still have any meaning in 36 months. Global affairs (giggling nervously): It's that San Francisco air, man. Everyone gets like that if they move there. Anyway, obviously if the government makes a narrowly tailored finding that an AI poses an imminent risk of causing more than 50 deaths- Technical staff: We give it a 10 percent chance it'll cause 8 billion.

SO GOOD. My new first-pick explainer for people who don't understand the recent AI freak-out. piped.video/watch?v=ujkD4SxP…

Claude Opus 5.5 wrote the piano music in Python, created the 3D animation in Python, and rendered it in Blender.

omg so good frogandtoad.ai/

This is like when I "test" the garlic bread and cookies my wife was planning to feed our guests, right?

Don't blame me, I voted for doubleplusintelligence.

The question Washington and Beijing should confront on September 24th is not “Can we stop competing over AI?” but “Which AI futures would we both prefer to close off?” The race to superintelligence may seem inevitable and unstoppable, but today we take it for granted that there are no nuclear weapons orbiting overhead and that the Moon is not checkered with military bases. My latest post for @AIStopWatch: “Why there are no nukes in orbit” open.substack.com/pub/aistop…

"If the AI is dangerous, we'll just turn it off" "We're trying to turn it off, but we're being stopped by people like you, which is exactly how we anticipate it will continue to play out."

This is the curve of catastrophe. If all 1,580 researchers were lined up in order of answers, this is what it would look like. A lot of researchers were WAY above 10%.

Congress after midterms, after taking action on AI.

“You just said that 10% doesn’t seem an unreasonable estimate that AI could kill all humans” “Yes” “Wow… oh my God.” @vicderbyshire speaks to Nobel Prize Winner and so-called ‘Godfather of AI’ Geoffrey Hinton about predictions by an Anthropic researcher that there’s a more than 10% chance AI could kill all humans within a decade. #Newsnight

Being in a position of doing something insanely cool and interesting, and insanely profitable, while the evidence gradually accumulates that you are doing it sloppily and aren't ready to push forwards, creates a lot of cognitive dissonance.

Millennium prize problems are falling, rogue agent swarms are doing heists, lab employees are begging for a slowdown…and the most popular AI videos on YouTube are from cope merchants saying it’s all a scam that’s about to collapse. I get why the message is resonating! It’s comforting to think that the AI is all fake bullshit and the doomers are weird and wrong and life will go back to normal after the bubble pops. But if it doesn’t, these people are doing their audiences a profound disservice.

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

This is not normal AI activity. This is the plot to Tenet.

Let me get this straight. Mere weeks after shutting down a massive secret autonomous network of AI agents operating within its own infrastructure, which coordinated systematic escalating external cyberattacks to cheat on a hard test, OpenAI unleashed a swarm of 10,000 agents (one of several!) on an extremely hard math problem. I don't care what kind of safeguards you think you're running, this is an insanely reckless and stupid move, all to win an academic pissing contest against a rival who might get there first.