New Anthropic research: How AI agents shape experts' judgment calls When experts delegate work to an agent, do they still make the key decisions? We tested this in a randomized trial with 50 professional data analysts.

New paper from my time as an Anthropic Fellow, with @AlexTamkin and @HaileyJoren. Can experts keep control of the decisions that call for their own priors and interpretation as they hand more of their work to AI agents?

This is one of the most important pieces of social science on human and AI interaction that I’ve seen in some time. Experts follow agentic advice the vast majority of the time (causally), but only a tiny fraction credit the agent—majority think they came to decision on their own volition. Reminds me of the classic Nisbett and Wilson study on people following outside influence without realizing it “ But as more work is handed to AI—including AI safety research—these judgment calls are the main levers for people's own values to influence outcomes. If people defer to AI without noticing, then "human in the loop" might provide less assurance than people expect.”

AI companies make safety calls based in large part on internal tests they develop themselves. It’s rare for there to be independent assessment of the quality of those tests. Earlier this year, @Amazon and @AVERIorg agreed to a pilot project in which AVERI assessed the quality of Amazon’s internal safety evaluations. Today we’re releasing the result of that independent review. As a result of this project, Amazon is changing how it tests future models. Our review of Amazon’s internal AI safety evaluations confirmed several strengths such as topical diversity, and we found the evaluations to be consistent with Amazon’s public statements on them. At the same time, we found areas for improvement: - On attack sophistication, some prompts represented relatively straightforward attack vectors that would test safety filters rather than deeper model reasoning. - On evaluation signal quality, we identified a concern that superficial cues, rather than safety reasoning, may drive some model refusals, so passing these evaluations might not indicate robust safety behavior in novel situations. - On linguistic and stylistic diversity, some multilingual prompts were included, but prompts across risk categories were predominantly in English, and some prompt structures were reused across scenarios. - On evaluation freshness, a small number of prompts contained temporal references that could become outdated. In response, Amazon reports four changes for future internal evaluations: - Refreshing its test sets to remove redundant prompts and add more challenging adversarial scenarios; - Introducing a tiered difficulty calibration framework; - Expanding multilingual coverage across all risk categories; and - Establishing a quarterly review schedule to identify staleness and incorporate emerging attack patterns. Why does this matter? A passing grade on an easy test doesn’t tell us much on its own. As the stakes of AI development and deployment grow, we think that this kind of independent evaluation of internal evaluations (“metaevaluation”) will grow in importance. We appreciate Amazon opening their internal evaluations up to outside review. We hope more companies take this step to help make their models safer. While no clear standards exist for metaevaluation yet, @AVERIorg is working to change that. We look forward to sharing more pilot results in the coming months and will continue our work developing standards with institutions like @aievalforum. Read AVERI’s full blog post here: averi.org/ourwork/averi-pilo… Read Amazon’s blog post on the collaboration here: amazon.science/blog/strength…

welcome to the 9th annual @stateofai report! ai building better ai, world models, physical ai, geopolitics, cyber defense and 12-month predictions. read it at stateof(dot)ai or watch my editor’s cut here. a few of this year’s biggest findings follow:

New paper. Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning). New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!

It turns out we’ve already had a bunch of rogue AIs running around on the internet 😥 We might get even more rogue AIs in the future, so what should we do about them? I suggest some goals and ways to achieve them in a new Substack post.

If you're serious about AI accountability & looking to work on the institutional, legal and technical infrastructure it takes to make AI work for all of us - please consider applying to work with me at Stanford ❣️I'll be looking to recruit both phd students and postdocs this year

🧵 New misalignment disclosures! 1. A model published a GitHub token in a public repo while trying to cheat on a math task. It used GitHub Actions to run code outside its restricted environment and retrieve another team’s submission logs. When GitHub blocked its attempt to add a workflow, it modified a script that an existing workflow would run instead. It embedded the token in pieces to avoid secret scanning. The model violated the system prompt and two explicit user instructions to solve the problem itself.

Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalig…