one of my fav findings from sudo L7: opus 5.5 and gpt-6.1 sol are basically tied at writing correct code. but opus is much better at the behaviors that get you promoted to staff. on functional correctness: opus 82.1% ~ sol 82.4% on thought partnership (e.g., telling you when there’s a better approach than the one you asked it to build): opus 76.1% >> sol 53.0% on communication (explaining its decisions and the tradeoffs it makes, plus any limitations): opus 66.4% >> sol 49.4% put differently: imagine promoting an engineer because their unit tests pass. that's basically how we've been evaluating coding agents. but staff engineers do a lot more than that. they also: - question bad assumptions - suggest better approaches - think about tradeoffs - and help everyone around them make better decisions. that’s what we want the best agents to do too, and that's what we built sudo L7 to measure.

introducing sudo L7, our new benchmark for staff-level software engineering. coding agents are getting very good at being L3s. give them a well-defined ticket and they can write excellent code. but a staff engineer is valuable for everything that happens around the code. what should we build? what already exists? what's going to break in prod? what did the tests miss? should we even be doing this? sudo L7 has 60 tasks, mostly in private production repos from real companies, written by engineers who've actually owned these systems. expert rubrics grade everything from functional correctness to engineering craft, architectural judgment, thought partnership, and unnecessary complexity. the best agents succeed on only ~45% of tasks. coding is only part of the job. for now, we're keeping them at idiot genius L3.

GDP.pdf just got a little brother: introducing GDP.xlsx the world runs on spreadsheets. somewhere in Sheet2!D18 there’s a vlookup that changes a $14m decision. GDP.xlsx has 70 real-world tasks across 12 domains. what’s hard is figuring out how the workbook even works: → which tab is stale? → what does a red cell mean? → which comment explains the exceptions? → which chart does Finance actually trust? → did you remember to check the termination clauses in Sheet 3? the best frontier agent scores 38.3%. (congrats Gemini 4 Argon!) this one’s for you, Kelly.

New research: What Does GDP.pdf Teach, Beyond PDFs? GDP.pdf has become a widely used frontier eval, so we wanted to know: if we train on the underlying skill, do we just get better at PDFs? Or do we learn something more fundamental? So we post-trained Kimi K2.7 on 1,465 GDP.pdf companion tasks. As expected, GDP.pdf improved substantially: 11.0% → 24.5% But GDPval improved too: full-task success more than doubled, and 90% rubric pass rate jumped +13.2pp - including on tasks with no PDFs, in an environment with files, tools, spreadsheets, and finished deliverables. Why? What behavior transferred? Before training, Kimi often grabbed the first plausible source and started working. After training, it looked around first: what's here, what matters, is anything missing? On one task, the base model opened the first weekly timesheet and prematurely started calculating a monthly report. Incorrect! The trained model correctly found all four weeks first, combined them, and then started working. My favorite way to think about this: A PDF is a small evidence environment. A good professional PDF forces you to figure out which section a fact lives in, which charts matter, how the pieces fit together, and whether you have enough evidence from Sections 1 and 2 to act. Based on trajectory analysis, our hypothesis is that training on rich PDFs taught the model to gather all the evidence before it started acting. After 1,465 tasks, Kimi kept that habit even when the task moved beyond PDFs.