I'm so happy to share that I’ll be joining @UofT as an Assistant Professor of Statistical Sciences and Computer Science, with an appointment at the @VectorInst, in 2026! I'm recruiting postdocs and PhD students: timrudner.com! Please help me spread the word! 🧵(1/5)

Wow, actor Jeremy Strong has been xrisk-pilled: “We are in a moment of existential civilizational crisis.” "The issues of control and alignment are not being figured out and are being outpaced by the speed and acceleration of the technology." "We need to wake up and regulate and do something about it."

After reading AI-written slop, my brain feels so exhausted (or grossed out) that I have to cleanse the palate by reading something written before the invention of the internet.

Over 400 researchers and I have signed this open letter calling for the AI and security communities to come together to improve alignment and containment approaches in order to protect our critical infrastructure and digital systems. ai-security-letter.org/

Over 400 academics in 30+ countries signed our open letter calling for immediate changes in AI development and deployment practices to ensure that cybersecurity incidents do not continue to occur, as future incidents will be far more damaging to our critical infrastructure.

The last few years have been tough for those applying for faculty positions. I'm happy to announce that my department has an opening in AI and ML. AI has historically been one of our strengths. Statistics is literally just two floors below, and we have centers and institutes like the AI in Science Institute and the Center for Machine Learning. Perks include: a great research environment, a politics-free work environment, proximity to the tech hubs in LA and Orange County, ... did I mention the beach? recruit.ap.uci.edu/JPF10341

Advancing AI safety and AI alignment is more important than ever. Apply to the @aims_oxford PhD program at the @UniofOxford if you want to join my group and work with me on AI safety and alignment! 🌐More details: timrudner.com/joining

When you're young you need to work hard and do annoying stuff, yes even if the task is genuinely dumb and your boss is being unreasonable. slowboring.com/p/feed-the-li…

Front page of every Australian newspaper rn. On the one hand it's *almost* reassuring that all these new incidents are from the same May-July period when OpenAI's training setup was clearly fucked... meaning maybe they've now fixed it. OTOH, what might be happening now—at OpenAI or elsewhere—that we're not going to learn about until December..??

What a great conversation -- highly recommend! If you want to learn more about why AI agents may go rogue and become 'misaligned', @hlntnr and I wrote a non-technical explainer paper on this back in 2021: cset.georgetown.edu/publicat… It's more relevant today than ever!

OpenAI was hacked by three people using Claude Opus 5. It doesn't take much creativity to imagine what kind of damage near-future rogue agents would be able to cause as capabilities keep improving---and we need better scalable oversight and verification methods to prevent that.

My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why. Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not. We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up). METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website. Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.

Great list of understudied and important research problems highlighted by @timrudner. Fill in the form – what are you waiting for?

I strongly second and agree with this path forward. E.g.,

We are looking for people interested to work on AI safety!! Please see more details below

We don't just need more evaluator organizations, we also need better methods for effective frontier AI oversight. Fill out this form if you want to work together on improving scalable oversight for frontier AI: forms.gle/VpWyH6gTqqcMot7e9 Topics of interest (what's missing?): - Formal verification and runtime verification for AI agents and multi-agent systems - Scalable oversight for very large multi-agent systems - Statistical guarantees for scalable oversight - Mechanism design for frontier AI multi-agent systems - Collusion in frontier AI multi-agent systems - Emergent cooperative norms in frontier AI multi-agent systems - Covert communication (steganography) in frontier AI multi-agent systems - Oversight alternatives to CoT monitoring - AI control

If we want this to succeed, we need many more well-qualified independent evaluators – which in turn requires substantial funding and more talent than is available now. The ecosystem of organizations providing AI safety research training is growing, but we need many more researchers who have the experience, training, research taste, and judgment to act as managers and actually lead oversight organizations as well as smaller oversight teams.

I appreciate the proposal, and if enacted across frontier labs, this could make a real difference – but it’s not enough: We need fundamental advances in AI alignment and scalable oversight along with a strong ecosystem of independent evaluators in order for this effort to move the needle. A few key points from the proposal + my thoughts on what’s needed for successful alignment and scalable oversight: "Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. [...] Anthropic is unilaterally committing to this step now." This is an important step towards successful scalable oversight. But to do this successfully, we need many more well-qualified independent evaluators – which in turn requires substantial funding and more talent than is available now. The ecosystem of organizations providing AI safety research training is growing (e.g., @MATSprogram, @SPARexec, @KairosAIS, @ERA_Cambridge, etc.), but although there is an influx of new talent, we need many more researchers who have the experience, training, taste, and judgment to act as managers and actually lead oversight organizations as well as smaller oversight teams. "We’ve made clear progress in alignment — training models so that they remain safe, ethical, compliant with our guidelines, and genuinely helpful (the principles that are embedded in Claude’s Constitution)." It’s fair to say that there has been some progress on alignment, but virtually all advances in alignment made by the frontier labs over the past few years have been "ad-hoc", i.e., trying some empirical trick and seeing if it has the intended effect – as opposed to approaches rooted in first principles that have solid theoretical foundations. That is not how we get to reliable goal and value alignment, and what we need are fundamental, theoretically well-founded advances. That's the kind of research the Alignment Journal (@AlignmentJrnl) is trying to support and amplify, but we should be clear-eyed about the fact that progress in alignment by the frontier labs is far from where we need it to be. "The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then. The question was always: what would you do with the extra time? The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks." Putting “pausing or slowing” in the same bucket here is misleading. Slowing down development and dedicating resources to non-ad-hoc alignment research to let alignment meaningfully catch up was a reasonable proposition then, and remains one today. Since we need fundamental methodological advances (e.g., formal guarantees) to perform alignment and scalable oversight successfully (unlike the ad-hoc attempts at oversight and alignment that are being used by frontier labs today), slowing down earlier would likely have been useful. It’s also worth remembering that Anthropic started driving much of the adoption of agentic AI through Claude Code at a time when capabilities had reached the point where slowing down for alignment and oversight was the reasonable call. The hour is late, but now is the time to act. The proposal – if enacted across frontier labs – could make a meaningful difference, but we need fundamental advances in AI alignment and scalable oversight along with a strong ecosystem of independent evaluators in order for this effort to move the needle.