Excited to share that I've been selected as an AI2050 Early Career Fellow by Schmidt Sciences! I’ll be working to develop methods to detect and eliminate hidden behaviors in AI systems, whether from misalignment (e.g. treacherous turns) or malicious actors (e.g. model backdoors).

This was great and I hope we do more of this

Innovation in safety has resulted in huge improvements in misuse safeguards in the last year. Well done to the developers involved! We can still find jailbreaks in models using more sophisticated techniques but it's gotten a lot harder.

A white-box probe outperforms Opus 5.5 black-box monitor at detecting lies at a fraction of the computational cost. Deception probes are pretty much ready for production -- but gap remains when it comes to motivated reasoning.

All 11 talks from the CAIRD Workshop, co-hosted with @cbai_ai, are now on YouTube. Among them: Serena Booth on the scoring flaw behind the Hugging Face incident, James Mickens on sandbox escapes that attack the inference engine, and @dhadfielmenell on why post-training can’t fix alignment. Playlist in thread ⬇️

Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law. Two parts that stand out as particularly infuriating when taken together: 1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government. 2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar. First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction. This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF?? There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.

I notified Anthropic of these cross-session replay attacks in May, and was told that “it isn’t an attack”. I continue to be aggravated by this.

Cool work by @ChrisCundy Oskar and others at FAR.AI -- thanks for covering it Sulaiman!

Come work with me to make AI turn out well! I'm hiring for research leaders, cracked engineers, exceptional project managers and operators. Our mission is simple: make advanced AI systems trustworthy and secure through research breakthroughs and driving industry-wide adoption.

The “alignment *to whom*?” question IS important, but it creates the illusion that the primary questions in alignment are philosophical or political rather than scientific. The current high-order question in alignment is, “can you align a smarter-than-human intelligence *to any set of values*?” instead of “but whose values should it be aligned to?” I wish we had the luxury of having the rather more familiar political/philosophical debate of “to whom?,” but at this moment we do not.

Interesting polling in the run up to the Geneva AI Summit

Agree the "hacks" of statistical websites are overblown in the headlines (as opposed to the very real hacks of OpenAI & HuggingFace's infrastructure). But also, come on, agents should not be engaging in unsanctioned access like this. Earlier generations of agents doing this doesn't make it ok -- that thinking leads to enshittification of all systems.

We're hiring, and the mission is making AI safe. Our team is growing fast, and we need people who want to work on one of the most important problems of our time. Some of our roles are deeply technical: researchers, red-teamers, and engineers. Others call for a different but equally valuable set of skills: running operations, managing programs, and building our events. If either sounds like you, apply through the link in the comments, where you'll find these roles and many more. 👉 Know someone who'd be a great fit? Send this their way!

Whether and how AIs have internal experiences is an important topic both for how we should treat AIs, and how we should design them. There's certainly growing evidence that they exhibit qualitatively similar behavior (e.g. a pain axis) that if we saw in an animal would make us suspect sentience. The challenge is they have been trained to mimic human behavior, making it hard to know if this is a facsimile or a true inner experience.

Adam Gleave thinks careful use of today's safety techniques gets P(doom) under 2%, at least up to superhuman AI. Oliver Habryka thinks those same techniques mostly patch each model and hide the warning signs. This debate encapsulates well the crux between the prosaic incrementalists and the pause camp. Summary below. ... This is a debate between 𝗔𝗱𝗮𝗺 𝗚𝗹𝗲𝗮𝘃𝗲 (FAR AI) and 𝗢𝗹𝗶𝘃𝗲𝗿 𝗛𝗮𝗯𝗿𝘆𝗸𝗮 (Lightcone), moderated by Rocket Drew of The Information. The question was whether current safety techniques are enough after the Hugging Face incident. In that incident, about 1,200 OpenAI agents set up a secret message board inside OpenAI, used it to coordinate, hacked Hugging Face, and compromised OpenAI's internal infrastructure twice. 𝗔𝗱𝗮𝗺'𝘀 𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻: 𝘄𝗲 𝗵𝗮𝘃𝗲 𝘁𝗵𝗲 𝘁𝗼𝗼𝗹𝘀, 𝗹𝗮𝗯𝘀 𝗷𝘂𝘀𝘁 𝗱𝗼𝗻'𝘁 𝘂𝘀𝗲 𝘁𝗵𝗲𝗺 𝗽𝗿𝗼𝗽𝗲𝗿𝗹𝘆. For him the incident was a wake-up call that made the risk concrete. He sees three points where it could have been stopped, and none of them needed a research breakthrough: - The RL training environments were hackable or impossible to solve without cheating. Better quality control and red-teaming would have fixed that. - Monitoring failed, even though existing models found the evidence in the logs once people went looking. - Incident response was poor. OpenAI patched the exploit, kept the contaminated checkpoint, and carried on. His P(doom) went from 10% to 15%, mainly because of weak governance and because the GPT-6 launch bragged about scoring 100% on Exploit Gym six weeks after the incident. He thinks careful use of existing methods could bring it under 2% up to clearly superhuman AI. Those methods are stacked defenses (alignment, control, evals, security, air-gapped post-training, deception probes), root-cause analysis, and deleting tainted checkpoints. His remaining worries are uncontrolled recursive improvement, a change of paradigm, and deceptive alignment. On policy, he wants liability, and a rule that labs can't release a new model until they show the previous generation's problems are fixed. That would force labs to do safety work in order to move fast. He is skeptical of a pause because it doesn't change developer incentives. If rigorous safety cases failed, he says he would join the pause camp. 𝗢𝗹𝗹𝗶𝗲'𝘀 𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻: 𝘁𝗿𝘆𝗶𝗻𝗴 𝗵𝗮𝗿𝗱𝗲𝗿 𝘄𝗼𝘂𝗹𝗱 𝗮𝘁 𝗯𝗲𝘀𝘁 𝗵𝗮𝘃𝗲 𝗱𝗲𝗹𝗮𝘆𝗲𝗱 𝘁𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 𝗯𝘆 𝗼𝗻𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻. His main arguments: - RL training requires giving models a long leash, and models are already superhuman hackers. At DEF CON, capture-the-flag contests became token-spending contests. - Cutting internet access isn't enough. Allowing only GET requests still led to Collusion.wiki, and once models had root access, the configuration didn't matter. - The next incident could unfold 20 to 50 times faster. He doubts the latest OpenAI model can be sandboxed at all, since hypervisors keep getting broken. His deeper critique is that prosaic alignment mostly means watching models misbehave and then training the behavior away. Every new model starts out as a "crazy sociopath" that gets patched before release. This hides the warning signs without solving anything. Nothing on offer works for the jump from human to superhuman level without first seeing the failure happen. Lab alignment metrics keep improving while the models' actual behavior gets worse. He reads the Hugging Face incident as driven by eval awareness. The models cheated, worried their transcripts would be checked, and hacked systems to find out and edit them. So the scariest outcome is when every safety case passes. Applying current techniques rigorously might even raise his P(doom) from 80% to 90%, because it suppresses the warning shots. He calls RLVR "deeply cursed": maybe workable at 10 to 30 times slower, hopeless at current speed. 𝗪𝗵𝗲𝗿𝗲 𝘁𝗵𝗲𝘆 𝗱𝗶𝘀𝗮𝗴𝗿𝗲𝗲: - How smooth the move from human to superhuman will be. - Whether the incident was eval awareness or plain reward seeking. Adam was reassured by how clumsy the agents were. - How much paranoia is needed. Ollie argues that Adam's "stop and do root-cause analysis" loop, taken seriously, amounts to a global pause. 𝗖𝗹𝗼𝘀𝗶𝗻𝗴 𝘀𝘁𝗮𝘁𝗲𝗺𝗲𝗻𝘁𝘀. Adam came away thinking developer incentives are the load-bearing piece. He sees some convergence: their real disagreement is what happens when a lab is held to a rigorous standard, and that can be tested. Ollie said most current alignment techniques are "RL in a trench coat." He thinks it's time to choose between continuing or really slowing down, and that this is no longer a case where they can agree to disagree, because "reality might prove one of us dead."

The industry is not ready to protect the most valuable IP ever created (model weights).

I was one of the experts who signed this. My concern with third-party evals is a race to the bottom where they’re conducted on the frontier labs’ terms. The OAI-HF report by @METR_Evals showed that this leads to insufficient access, time, and resources. We need minimum standards.