If you guys want credible information follow @ValsAI. Slept on.

soon we all get to just do math

We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.

Palantirian-turned-founder @RayanKrishnan built a leading independent evaluator of AI models. He has a unique view of the AI frontier, and a deep understanding of how to score model performance - and risks.  New AO episode out now, with Rayan and my @8VC Partner @vvkgopalan

I wouldn't have founded @ValsAI without first working at Palantir and Vals wouldn't be what it is without 8VC investing in our seed. @JTLonsdale was key to both. Got to sit down with Joe and @vvkgopalan to talk about the story behind Vals, where the industry is going, and why I'm broadly optimistic while the world is incredibly high-variance.

my girlfriend didnt understand the whos on first bit should i be concerned

The tide is turning

it’s not immediately clear what that gap between open and closed source will actually close. egalitarian platforms like x have a tendency to become echo chambers to developments that are philosophically consistent. unfortunately we’re approaching a world in which relative differences of models are indistinguishable to normal people.

The long-awaited Kimi K3.1 is here, and it is #1 on our Vals Index amongst open-weight models (#8 overall)

Does it matter where you run an open-weight model? Today we are releasing the Inference Provider Index. It compares first and third-party open-weight models for quality on agentic tasks in finance, coding, and legal industries. Unlike other inference benchmarks, we rerun it every two weeks to catch quality degradation, and we use agentic tasks, not single-turn multiple-choice questions. It covers over 650 hours of model runtime.

click to learn more

Richard has been thinking deeply about the limits of intelligence (not just imitating human intelligence). Excited for this one

Excited to launch Before They Break Out, where I’ll spotlight promising startups before the rest of the world catches on. Kicking off with Vals AI, a startup building critical evaluation infrastructure for the future of AI. @ValsAI @RayanKrishnan

The latest DeepSeek model makes powerful offensive cyber capabilities accessible to a much wider range of actors. The increase in risk is enormous.

people like indices because they represent intelligence unidimensionally. but a model's ability to do knowledge work (vals index) doesn't predict whether it can do scientific research. this is an important step for us toward measuring a model's propensity for novel scientific breakthroughs.

not a secret, our external rsi index has predicted the same timeline

Our RSI index perfectly predicts this btw

An oracle router on our Vals Index would score ~13pp higher than the best model and be 37% cheaper. Routers have emerged as a new primitive in the AI stack, we're excited to partner across the major providers to develop a methodology for consistent evaluation.

Some thoughts on the new Meta release today: - The models from MSL have improved quickly. They’re not just competing for Pareto optimal anymore, but driving for the frontier. - To me, the biggest unlock of these models is their token efficiency. They’re able to see significant performance increases while reasoning less and spending fewer tokens / turns. In Vibe Code Bench ,WF gets the same performance as MS 1.3 in half the number of steps. Smarter models think more efficiently. - Today, Muse Wonder Full is only available in in-app experiences (not via API). It may be that we see an even tighter coupling between models and first party applications. Or that this gating prevents nefarious actors from misusing the most capable intelligence. Not yet clear what the plan is. - MSL is making some serious progress on alignment. We’ll have more to share publicly here soon. Meta hasn’t been as vocal about safety, but to service Muse so widely they have needed to solve some fundamental alignment challenges. - Relatedly, we saw 0 refusals and Wonder did better than Wonder Full on our cyber capabilities benchmarks. This is itself very good! People should not expect their hard economically valuable work to be inhibited by these models. - The models have room for improvement on scientific discovery. They are strong executors, especially in code, but don’t have the intuition for which experiments to try. We’re still running our RSI Index, sharing results soon. If they aren’t able to maintain the frontier, this would be why. - META went up 3% today. I’ve said it before and I’ll say it again. As AI underpins a larger part of our economy, third party benchmarks will be the best value predictor of AI labs. The market for intelligence is rationalizing.

The diffusion of personal agents (Grok bot, Muse, Dots, Instinct, etc) may cause a rapid increase in socioeconomic inequality that we are not prepared for. This follows from a rough first-principles argument: 1. Life is a set of negotiations. 2. As personal agents get better, we will delegate more of our negotiations to them. 3. Negotiations will increasingly happen directly agent-to-agent. 4. In general, agent performance scales with model intelligence and inference-time compute. 5. Personal agents that consume more frontier intelligence will win more negotiations, in expectation. 6. People with more resources will pay for more expensive personal agents. 7. Winning more negotiations begets more resources. Therefore - people with more resources will use better agents, win more negotiations, and acquire more resources. Consider a situation where my roommate and I are deciding how to split our rent. If we delegate the decision to our agents, with mine running Fable 5.1 and his running Luna 6, I should expect a more favorable split than if our agents were equally intelligent. The most important junctures in life are negotiations (jobs, promotions, purchases, housing, lawsuits, etc.), so people who can afford the best agents should expect to do better in these situations, compounding their initial material advantage. This already happens to some extent (people with more resources hire better lawyers), but it may be amplified if 1) the gap between frontier and free intelligence is wider than the status quo gap (Big Law attorney vs. public defender), or 2) delegation to agents means more negotiations occur in total, so differences compound. Personal agents may instead reduce inequality if 1) the gap between everyday and frontier intelligence is smaller than the human equivalent, 2) agents are more neutral/rational than humans and converge on an independent truth rather than acting at the behest of their operators, or 3) superintelligence frees everyone from resource constraints entirely. If this form of inequality emerges, possible solutions include 1) taxing tokens and 2) subsidizing intelligence for the masses.