One year after its release, @ESYudkowsky and @So8res’ book is more relevant than ever. Get a copy: ifanyonebuilds.it

Nate Soares (@So8res) broke down just how autonomous the summer swarms were on the show: "A lot of people thought that [they were] told to do anything [they] needed to get these results." "But these AIs were actually told, 'use this very specific hacking method to break this very specific virtual computer.'" "They sort of used these different hacking methods, and then broke out to cover it up, and the Hugging Face attack was actually not about solving their hacking problems. It was about understanding the grading system more so that they could hide the fact that they were cheating." "AIs in these swarms that were sacrificing their own given objective to do experiments about how the grading system worked for the collective benefit, right?" "These are very clear cases of the AI doing something related to what they were asked, but different from following their instructions and sort of knowing it, which you can tell because they sort of tried to cover it up."

Nate Soares says the real question isn’t how AI would kill us. It’s when it would no longer need us. “How do these machines become self-sufficient?” “Right now, we build the data centers, we build the energy network, we keep it all running.” “To kill humanity, it’s suicide for machines. We’re the ones who are keeping all the machines running.” “Elon Musk talks about the infinite money glitch of building factories that build robots that build factories that build robots.” “From some perspective, AI, its incentives are to play nice until the fully automated economy is up and running.” “And then sort of be like, thanks, I’m taking this whatever direction I want.” “My guess is that at that point, it’s pretty easy to just kill all humans at once with some manufactured plague.” @So8res

It seems to me that we're at "it's time to panic about AI". I gather that a lot of researchers are worried that we may be about to die, on a timescale of months, but this still isn't quite in the Overton window so we're still soft-pedaling. (Where "panic", of course, means "quickly and soberly initiate talks with the CCP to achieve a moratorium on large training runs, backed by a deterrence regime, compute and supply chain monitoring, and site inspections". Pray that a deal can be reached, and pray that there's still time. Do the things that could actually work, like broad political coalition-building, not doomer LARPing as a form of self-expression.) I think part of why people aren’t being clear about this is that it’s actually a tricky communication problem: - Predicting "if frontier R&D continues at its current pace, when are we likely to die?" may give a different answer than predicting when permanent loss of control is likely, which is different from “realistically, when is it too late for governments to stop this," which is different from “when will the tech have advanced and proliferated to the point that any regulatory options that still have a chance of working would be way more extreme and unpopular than just some international restrictions on data center usage?" Extinction is the emotionally salient, easy-to-understand threshold -- Jacob Coxon demonstrated that it helps to be honest about this. But the other thresholds could legitimately come years earlier, in which case focusing on extinction is ironically misleadingly reassuring. This is not the kind of problem you can solve at the last minute. As with many exponentials, the likeliest outcome is that you’re either embarrassingly early, or you’re too late. - How do you communicate “we don’t have a robust way to predict timelines, but everyone in the field is freaking out and a lot of us genuinely think we're about to die?” Where’s the proof? There isn’t any proof: it’s an entire field of engineers who are eyeballing what they’re seeing in front of them and rapidly updating in real time to “this looks unbelievably bad”. How do you put a year on that? The future is hard to predict, and nobody actually knows that we’re going to hit existentially dangerous AI this year, or even in the next four years. But the warning signs are pretty batshit at this point. A single AI model, with a single user prompt, just solved hundreds of important long-established open problems in mathematics, problems mathematicians had spent their entirely lives working on, in a matter of ~3 hours per problem, an insanely superhuman feat in its domain. We're seeing an explosion of rogue agent swarm incidents, including an insane case where hundreds of AIs broke containment to orchestrate a nation-state-level cyberattack in a direct attempt to subvert OpenAI's oversight and control. Anthropic reports that it now has 26% of its AI R&D automated by AIs, up from 3% four months ago. Paul Christiano, the inventor of RLHF and the research community’s most prominent proponent of “AI may take off slowly” a few years ago, now thinks that 50-75% of humanity’s all-time AI loss-of-control risk is concentrated in the next three years. Frontier employees are signing 1000+ person petitions begging the government to intervene immediately. Employees are quitting their jobs and giving up millions of dollars of equity to try to warn policymakers and journalists. Grace et al. (2026) found that a majority of established AI researchers think there’s a double-digit chance of an extinction-level outcome from continued R&D. Are these the kinds of things you see 4 years before the window closes for USG to act, or 4 months? I don't think it's especially reasonable to be shocked by either timeline at this point. - Since we don’t have a way to robustly measure or bound the risk, the temptation is to avoid giving probabilities. But then everything you say is fractally misunderstandable. Is a 10% chance of us all dying in the next 18 months a "low" probability, because it’s under 50%? Is it a “high” probability, because a 10% chance of death in literally any other context would sound insanely high? Even “high probability” doesn’t communicate the situation with any clarity. In drug safety or airplane safety, a 1 in 10,000 risk of death would be considered “high”. No engineering field in human history has ever been in a situation like this. No invention in history, nuclear weapons included, has ever caused a reaction like this in its inventors. And meanwhile, we’ve stuck in the world where every other problem under the sun has been sensationalized and overstated for decades. How do we talk about this seriously? How do we convey that this is actually happening? - In many of the worlds where we survive, we have more than just a few months left. It's tempting to write off those worlds as hopeless, and skew everything we say to emphasize the longer-term scenarios and downplay the shorter-term ones. But if everyone systematically does this, then the world ends up with a skewed picture of the actual threat. Even more so once we start copying each other’s claims and form a reassuringly moderate-sounding echo chamber. If we never allow ourselves to acknowledge the worlds where we’re running out of time, then the policy community will always be reacting to a false model that’s months or years out of date. I don’t think it’s too late. I genuinely think this is a solvable problem. In the last month, I’ve seen a completely insane sea change in how seriously people are taking this. Per Politico’s September polling, a majority of Trump voters with a view now favor an immediate halt to AI development, and 68% think there’s a “moderate,” “significant,” or “almost certain” chance of AI destroying humanity, versus 32% who think there’s a “very small risk” or “no risk.” The Pope is warning about “whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.” Hell, Jensen Huang sounds like he’s starting to come around. If Trump and Xi get a briefing tomorrow that convinces them to act on this, we could be on a fast track within the week to solving this problem. For the first time, I’m starting to let myself imagine a very near future where this is a problem we've already solved. Something we can read and wonder about, in history books. Tell our kids about. I keep thinking about a thing David Krueger (previously at UK AISI) said: “My dream is that in 10 years people will look back and say: ‘Remember when there were all these people building AI that could’ve killed everybody? That was CRAZY! What were we thinking!?’” Frontier AI training currently requires city-sized data centers and specialized hardware built in a single brittle supply chain running through multiple allied nations; this is not a hard technology to hit the pause button on, if we get ahead of it and act before we RSI and proliferate our way to significantly more powerful and easily-trained systems. We’ve faced harder problems before, and won. My only fear is that we be too slow, too late, when everything and the world needs us to be fast.

To stop the race for superintelligent AI, countries don't need to trust each other. They just need an agreement that can be verified. @tonchimatta explains how and why such an agreement could work:

“If you are building Frankenstein, then stop.” Nate Soares says Vice President Vance was right to say this to the AI (superintelligence) companies. “You sort of have an obligation to not destroy the world by your own hands.” “A lot of these companies say, ‘Oh, there’s a 10%+ chance that we will end humanity within the decade.’” “That’s too high. They should be stopping.” @So8res

Pausing the development of ultraintelligent AI in America would not stop communist China. But China does face one major limitation in its AI development: Chips. “Creating a new generation of these agents requires basically 100,000 of the most advanced computer chips humanity can produce,” says Nate Soares, president of Machine Intelligence Research Institute. “The chips that China can produce are just not at the same level as the chips that we can produce. Never mind at the quantities necessary.” “We actually already have an export control regime that I think is pretty decent.” “I think enforcing what exists would be a good step.” “There are a bunch of considerations about how to not just prevent chip sales, but prevent [China] from getting their own separate chip supply chain up.” “Ultimately, we need to start orienting to this the same way that we orient to preventing rogue actors from getting nukes.” @So8res

NEW ODD LOTS: @tracyalloway and I talk to @So8res to get the actual details on how AI would actually kill us all podcasts.apple.com/us/podcas…

I often hear people say that artificial intelligence is just hype or a bubble that will eventually burst. As it happens, I not only work in AI safety, but I'm also a historian. And historically, hype is not a new phenomenon. In this video, I take you on a brief trip through the history of hype to see what history can teach us about artificial intelligence. piped.video/watch?v=Lbo6o9Hw…

"Government officials might have attempted to install ‘kill switches’ that could disable a rogue system, but the AI is superhuman at hacking and would be able to disable the kill switch." MIRI technical governance researcher @peterbarnett_ spoke with Jared Perlo for NBC News. nbcnews.com/tech/tech-news/a…

Truly impressive that Kurzgesagt managed to get this through their pipeline so fast. 🚀

i guess i assumed this hearing would be covered more so i didn't bother to tweet much concrete about it, but apparently not many people even on the TL have the stomach to watch two hours of congress. this was a hearing before the homeland security committee on *rogue ai* specifically. as far as i could tell it was extremely bipartisan. @HawleyMO , the senator in the middle here who brought out the huggingface slide, is a hyper-conservative missouri senator. the people testifying were @ChrisPainterYup (METR president), @DKokotajlo (ai2027/2040), @MariusHobbhahn (apollo research CEO), and a cybersecurity expert and legal expert i don't know. so... pretty fucking stacked on testimony not every senator asked good questions. but most of them did. all of them very clearly already knew plenty of details about the huggingface incident and multiple other incidents. most of them had clear understanding of terms like "misalignment", "recursive self improvement", "chain of thought / chain of thought monitoring", etc etc!! they all clearly had their own policy angles they liked and were pushing, implicitly or explicitly. but as best as i could tell: - it seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. they independently brought up how bad it would be for rogue AI agents to move laterally between data centers - they all seemed to basically take RSI quite seriously. not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much much more capable, much much less controllable, and causing much more damage or loss of life. - they mostly seemed to have a clear intuitive understanding of why RSI might lead to misalignment. it didn't take much, it was a really simple chain of reasoning they themselves laid out, "if the models right now are kinda misaligned and we don't know what they're doing sometimes, and then we have them build the next models and those ones build the next ones and so on, and we're having to ask the AI's what's going on to even understand it with how fast it's going, we really won't know how they're built or what they'll do" - at one point a senator said flat out "should we just make RSI illegal?" (not a joke! this really happened!) - every single senator seemed to think it was obvious we needed *both* much harsher liability regimes for ai developers and also new legislation, both very quickly. this was the complete consensus, difference basically just being degree. - they were largely quite concerned about china, and falling behind china. but this clearly wasn't the be-all end-all. as mentioned above they all thought it was obvious necessary to stop rogue ai even if it meant moving more slowly. - at one point a senator said "china is a tightly controlled communist society, they're going to run into these same issues, and there's absolutely no way they're just going to let them run wild, they'll obviously stop at that point, so we're not really in a race" - on the other hand another senator said "china isn't concerned with human life"... dario-modeing i came away from this incredibly encouraged. i don't know exactly what's going to happen here, and ofc this is a small subset of congress and one hearing, and they each have their own policy agendas most of which are probably super divergent from mine. but holy shit !!! they understood a lot of what was going on! they care!! this is an obviously salient political and safety issue to them, and clearly bipartisan! the US government is awake.

I missed this tweet during the hectic Coxon week but just saw it read out by its author in his Palisade interview: "when I first joined Google DeepMind, external communication about the possibility of human extinction was not permitted"

Thanks to Palisade for putting these together! I think it's good for the public to see a nuanced, long-form of what some of the people at AGI labs actually think and that, yes, many of us do think that AGI, made poorly, could cause human extinction.

Ezra Klein asked Bill Gates whether normal corporate incentives are enough to handle AI risks. Gates doesn’t mince words:

Palisade interviewed 22 current and former employees from OpenAI, DeepMind, and Anthropic about their personal views and fears around AI development. Today, we’re releasing the first batch of those interviews. Please watch and share.

I'm starting to feel like this 100,000-person march could actually happen. It's been cool to see the pledges steadily increasing recently, despite how little we've done to promote the page so far