I won’t fall for the SF psy op I won’t fall for the SF psy op I won’t fall for the SF psy op I won’t fall for the SF psy op I won’t fall for the SF psy op

We’re launching leaderboards for HealthBench and HealthBench Professional, two benchmarks evaluating how AI models respond in realistic health conversations HealthBench and HealthBench Professional are benchmarks from @OpenAI that grade model replies to realistic health conversations against physician-created rubrics. HealthBench tests models on 5,000 health conversations with everyday users and health workers. They cover seven themes, from emergency referrals to global health, and each rubric criterion falls under one of five axes: accuracy, completeness, communication quality, context awareness and instruction following. HealthBench Professional is a smaller set of 525 conversations which, unlike HealthBench, are all with clinicians. These cover consultation, writing and research tasks, and include adversarial and difficult conversations designed to challenge models. We report both the raw score (the share of rubric points earned) and the length-adjusted score that OpenAI reports. The length-adjusted score lowers the score of replies longer than 2,000 characters and raises the score of shorter ones. Consistent with OpenAI’s original implementation, replies are graded by GPT-5.4 (low) (HealthBench Professional) and GPT-4.1 (HealthBench). Key results for HealthBench Professional: ➤ GPT-6 Astra and GPT-6.1 Sol top the leaderboard: GPT-6 Astra (max) scores 66.0% on the length-adjusted score, comparable to GPT-6.1 Sol (max, 65.1%) and ahead of Claude Sonnet 5.5 (max with fallback, 63.0%) ➤ OpenAI models make up the entire score vs. Cost per Task Pareto frontier: The Pareto frontier consists of GPT-6 Luna (max, 61.9%, $0.004 per task) through GPT-6 Sol (max, 62.3%, $0.036) and GPT-6.1 Sol (max, 65.1%, $0.047) to GPT-6 Astra (max, 66.0%, $0.27). GPT-6 Luna (max) scores 4 points below GPT-6 Astra (max) at under 2% of the cost ➤ Claude Opus 5.5 and Claude Fable 5.1 receive the largest length penalties: Both average ~5,900 characters per answer, nearly 3x the 2,000-character threshold, and lose over 11 points after adjustment. Claude Opus 5.5 (max with fallback) has the highest raw score (71.4%) but drops to sixth (59.9%). GPT-6 Sol (max) is the only model averaging under 2,000 characters (~1,600), and the only one whose score rises Key results for HealthBench: ➤ Claude Sonnet 5.5 tops HealthBench: Claude Sonnet 5.5 (max with fallback) scores 62.1% on the length-adjusted score, ahead of DeepSeek V4.1 Flash (max, 60.9%) and GPT-6.1 Sol (max, 58.5%) ➤ GPT-6 Luna, DeepSeek V4.1 Flash and Claude Sonnet 5.5 form the score vs. Cost per Task Pareto frontier: GPT-6 Luna (max) scores 54.6% at $0.002 per task and DeepSeek V4.1 Flash (max) scores 60.9% at $0.005 per task. Claude Sonnet 5.5 (max with fallback) has an average Cost per Task of $0.08, ~16x more costly than DeepSeek V4.1 Flash for a ~1 point score increase ➤ Shorter answers trade completeness for communication quality: Each HealthBench rubric item is tagged with one of five axes. Completeness checks whether a reply includes all the important information, while Communication Quality checks whether it is clear, well structured and pitched at the right length and level of detail. GPT-6 Sol (max) writes the shortest answers (~1,000 characters) and has the highest raw Communication Quality score (81%), but the lowest Completeness score (34%). Claude Sonnet 5.5 (max with fallback) leads on Completeness (63%)

Recursive self-improvement is being hyped to the moon and back. I wanted to know how good frontier models actually are at a less-studied piece of it: curating pretraining data. So I built PretrainGym at the @PrimeIntellect RL Residency 🧵 justin-chae.org/blog/pretrai…

born to frolick around forced to think about docker network policies

INTELLECT Policy Optimization (IPO) In async RL, trainer and inference never agree exactly on a token's probability. IcePop masks on the ratio, which misjudges both tails IPO masks on the probability difference instead

Whelp. Every inch given. public-inspection.federalreg…

Distillation has always been part of the explanation for why China is so close behind the U.S. in model benchmark performance. But one of my updates after joining openai is that distillation probably plays a bigger role in the explanation than I realized.

i used iCloud as my mail service for over a decade ai assistants might be the last drop in the bucket to move it all over to gmail

I've got a theory Since Anthropic started blocking distillation and patched most exploits Chinese models really slowed down it might be the reason why we haven't seen new releases since GLM 5.3 and Kimi K3.