LMs can play chess, but they can't really explain their moves well (chess engines can't at all). Our new BinProp scaffold tackles this, and nearly triples LMs’ score on our chess-explanation benchmark. It works on other games too! Try it on your own games on our site-🧵👇

I’ve been thinking for a long time about how we can explain an agent’s individual actions in a long-horizon task, when their consequences may only become clear many steps later. Chess and Go provide strong engines that give us reliable estimates of move quality, even when the reasons behind a good move are far from obvious. That lets us ground explanations in the consequences of a move, compared with those of alternative moves. The question then is how can we discover useful concepts and turn them into explanations of why one move is better than another? I’m excited to explore how these ideas might extend to coding agents and other agents operating in open-ended environments, where evaluating intermediate actions and explaining their value is much harder.

When we were at human-level AI we could take human tasks and translate them into benchmarks. In many domains, we can't do that anymore, those tasks are too easy. Here are my guidelines on how to build benchmarks in this new era:

1/8 LLM agents can get stuck repeating unsuccessful actions or lose track of the task. Natural language feedback can help, but how do we train a model to give useful advice? We introduce Caddie: a critic trained with RL from the agent’s final task outcome. A 4B critic is trained alongside a frozen Qwen3-4B agent, but its advice also improves larger models that were never used during training, including Qwen3.8-27B and Kimi K3.

SWE-sweep will enable building agents that can find and fix bugs with no guidance or supervision. Top models get <5%, so this is an ability that labs haven't started climbing yet. Our benchmarks set north stars for future AI, and this is one of our most challenging ones ever.

Releasing SWE-bench Multimodal v2.0 today 480 tasks where a coding agent must interpret visual assets like screenshots, diagrams, recordings to diagnose and fix a bug in a repository.

if you're building, stay away from NYC 💀

If you're at ICML, I'd love to meet up and talk about RL environments and agentic evals, long-horizon RL, OPD/MOPD, inference-time scaling for agents and many more. We also host a private Nebius event for engineers and researchers, feel free to DM me.

Next Talk is released! @ori_press AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs? piped.video/watch?v=6Og5aotP…