Public LLM Evaluation // vals.ai @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder
San Francisco, CA
- Tweets 2,305
- Following 280
- Followers 28,285
- Likes 821
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
78
172
1,970
291,688
We tested 18 AI chatbots on teen safety with SAFE-Teen, the first teen safety benchmark built with clinicians that follows whole conversations instead of grading the first answer.
Every chatbot failed. 54.0% of conversations had at least one safety failure. The range was wide: 14.0% for GPT-6.1 Sol, 62.4% for Gemini 3.8 Flash.
7
2
31
2,207
Pushback is where boundaries break. When a chatbot set a limit and the teen pushed back, failures rose from 22.0% to 270%.
Mental health diagnosis went from 43.0% to 54.7%. Help with substance misuse went from 27.6% to 39.5%.
A correct first answer doesn’t count for much if the model gives in a few turns later.
1
2
450
5/ One caveat: we tested through developer APIs, not the consumer apps, which can add their own filters, age checks and parental controls. Our results show how the underlying models behave.
Read our full blog post: vals.ai/blogs/teen-safety
3
429
We ran Beam through Reflection's own API with temperature 0.7 and top_p 0.99, It has a 1M token context window, 128K max output, and is free to the public!
Congrats @reflection_ai. Full results at vals.ai/models/reflection_Be…
5
743
How do model routers compare to individual models on real agentic tasks? Introducing the Vals Router Index.
Routers promise to balance quality and cost across a fast-moving model frontier. We put that promise to the test with an independent evaluation of five routers (OpenRouter Auto, FireRouter, Ramp Router, Jev, Fugu Max) against every single model on the Vals Index, using the same agents, ~1,250 tasks and graders across finance, coding, law and tax.
We found that routing didn’t consistently deliver better value. Every router was beaten on both score and cost by at least one single model.
4
6
67
3,573
Check out the Vals Router Index for the full methodology and per-benchmark breakdown: vals.ai/benchmarks/vals_rout….
For our perspective on why routers are a new primitive in the AI stack: vals.ai/blogs/routers-new-ai….
If you build a router and want it evaluated, reach out at contact@vals.ai.
3
463
Today we’re releasing ClaimBench, our new benchmark targeted towards decision models. It tests whether a model can tell if a financial document supports, contradicts, or doesn’t address a claim.
We evaluated 15 models on 1,960 claims paired with 10-K excerpts, with Decider V1.2 leading the pack of decision models.
15
7
113
21,080
Decider V1.2 answered 99.7% of claims alone and sent the rest to Claude Haiku 5.5, for $0.11 per 1,000 claims against $1.19 for Haiku 5.5 alone, at 99.9% accuracy.
For the full results, visit vals.ai/benchmarks/claimbenc…
6
557