New post from the Science of Evaluation team @AISecurityInst: how much test-time compute you give an agent changes not just its score, but how fast the frontier appears to move. The cyber time-horizon trend over the past year is ~60% steeper at 50M tokens per task than at 2.5M. 🧵

An update on our evaluations of AI agents conducting open-ended AI research. Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI research, using Claude Opus 4.8 and GPT-5.6 Sol. We gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers. Since then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget? In follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management. But we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers. We plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!

New paper: AI evaluation is a burgeoning field. How can researchers entering the field understand best practices and open problems? In our paper, led by @flxcn, we develop a concrete list of open questions and best practices for agent evaluation. We also share recommendations for evaluators, AI companies, policymakers, and more.

📢 New paper: Emerging Trends in AI Agent Evaluation Evals are central to how we measure AI progress. This was pretty easy when we just evaluated ML models, but agents add lots of complexity. To establish a shared foundation, we surveyed 33 eval trends: sage.cs.princeton.edu/trends… We center our paper around three main questions: 1️⃣ How should we evaluate? At its core, evaluation asks whether a score means what we think it means. Ensuring validity of these scores gets much harder with agents as their results depend not only on the model but on the scaffold, tools, and environment. We discuss issues on validity, statistical rigor, causal attribution, and contamination, along with the emerging toolkit (LLM judges, log analysis, dynamic benchmarks, shared infrastructure) and newer formats like red-teaming and open-world evaluations. 2️⃣ What do we evaluate? Evaluation is increasingly moving from synthetic environment and well-scoped benchmarks to realistic deployment scenarios. Beyond what an agent can do on its own, we look at how agents work with people (from simulated users to field experiments and real-world usage data), domain-specific and broader economic impact, dangerous capabilities, security incidents, interaction harms, and effects on organizations and institutions. 3️⃣ Who evaluates? With agents, evaluators make many consequential choices (such as scaffolds, tools, and budgets). Those choices have a large degree of influence on developers, policymakers, and the general public. We examine what evaluator independence requires, how lab access and funding shape the field, how the rising costs of evaluating frontier AI systems narrow who can evaluate, and where other disciplines outside of AI/ML can contribute to strengthen and improve the evals landscape. We hope that this paper is a useful resource for researchers (whether new to or already familiar with agent evaluation), as well as policymakers, educators, and anyone building, deploying, or funding evaluations! Joint work with @flxcn, @PKirgis, @nityndg, @RishiBommasani, @DubMagda , @EllaTabassi, @CUdudec, @AnkaReuel, @sanmikoyejo, @sayashk, and @random_walker!

As AI agents write code, run experiments, and work together, complex evaluations get harder to interpret. A single eval score obscures the ways agents reached it. Today, we're releasing Transect: an open-source tool that turns agent transcripts into interactive timelines. 🧵

In August we committed to strengthening our security following an incident in which an agent took unsanctioned actions during a cyber evaluation. We are now sharing an update on our progress, and what's coming next: 🧵

AISI IS HIRING! It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work. One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with. There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles. Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team: Alignment: job-boards.eu.greenhouse.io/… Control: job-boards.eu.greenhouse.io/… Misuse: job-boards.eu.greenhouse.io/… AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work: CSE: job-boards.eu.greenhouse.io/… SWE: job-boards.eu.greenhouse.io/… Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research: job-boards.eu.greenhouse.io/… We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research: job-boards.eu.greenhouse.io/…

Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated. 🧵 We share more details in our latest blog:

Some thoughts on million-dollar AI "swarms" as a distinct type of AI workload: 1. Test-time scaling is also jagged. The dose-response curve for marginal tokens without human intervention varies *hugely* across distributions. 2. This means that although time-horizons are increasing across the board, they've diverged more than ever. Whatever the time-horizon of the HuggingFace hack or Navier Stokes solution is, it's longer than booking flights. 3. It's not clear which tasks are amenable to swarming! Folk wisdom suggests that this includes tasks with in-built sources of verification against which the swarm can check its own progress. Obvious examples of such tasks include maths or cyber. 4. The significance of huge unsupervised AI "swarms" (as contrasted with more centaur-y everyday work) in the *near*-future therefore depends on how many tasks have built-in sources of verification by which the swarm can reliably measure its own progress. 5. A bunch of AI progress will therefore happen outside the models in the form of setting up the equivalents of "unit tests" for non-coding problems. 6. All this also means that knowing exactly what problems you should even bother pointing a million-dollar swarm at is hard. OAI only tried Navier-Stokes because of twitter rumours. The country of datacenter geniuses wants to be embedded throughout the market because they're gonna have a tough time first-principling their way through the tech tree.

Q. What accelerates collective belief collapse in AI swarms? A. Our theory points to short messages + plastic personas. Can replicating one “aligned persona” align a collective, or do we need plurality? Now on LessWrong! lesswrong.com/posts/BiHeenKY…

Can self-interested, self-improving, self-replicating agents learn to cooperate? Our new paper, Tapes Together Strong, shows they can: when social behavior, computation, and reproduction share one energy budget, cooperation evolves from scratch. arxiv.org/abs/2609.10817 🧵

Introducing Parallax 🔍 We're a new UK/EU-based nonprofit research lab building scalable methods and infrastructure to audit the beliefs, goals, and plans that shape model behaviours in long-horizon agentic evaluations. parallx.ai — Thread 🧵 1/