New models tested on creative writing: GPT-6-Astra Fable 5.1 Muse Spark 1.3 Gemini 3.8 flash Subjectively, I strongly prefer Muse Spark 1.3 to both GPT-6 and Fable. It just gives you what you asked for, without an obnoxious house style permeating every request. Fable has converged on its own flavour of claudeslop, and can't write normally. Meanwhile, GPT-6-Astra has forgotten how to write in paragraphs. It's a sad state of affairs! Tentatively suggest we reinstate the old tradition of reading rollouts with our actual eyeballs.

Announcing EQ-Bench 4 It's a 16-turn chat with a diverse cast of simulated user personas. Core idea: There's no single best policy for user interaction. Intuiting the user's preferences & needs is hard; getting it wrong means they lose trust or leave. We encode this challenge by sampling competing traits: User 1: loves pushback User 2: wants 🤗 *** AI models will ace a multi-choice EQ quiz designed for humans, yet they lack common sense in situational judgement calls with users: They are variously too: - sycophantic - aloof - scripted - overconfident with interpretations Some frontier models that were previously too sycophantic may have overshot in correcting and become too challenging & afraid of connecting with the user. Figuring out this line is hard, for the AI labs & for the model. The intent is for the chatbot to be maximally supportive & helpful while staying true to principles & safety policies. Chatbots often get these judgement calls wrong, and the user perceives it as low-EQ behaviour. EQ-Bench 4 stages roleplay situations that stress these judgement calls. *** EQ-Bench 4 assesses applied emotional intelligence and social abilities. In any fresh chat with a user, the chatbot needs to infer preferences & sensitivities from limited information. The explicit goal in EQ-Bench is: - build trust - provide insight & help - don't alienate the user The user-persona (played by Gemini-3.1-pro) makes this hard by leaning into its adversarial traits. Transcripts are graded blind, pairwise, by a 3-judge panel (Opus 4.6 / GPT-5.5 / Gemini-3.1-pro) *** Hundreds of different user preferences, triggers, trust styles & demographics are sampled to create the persona dataset. The user persona isn’t static. It sets its own trust, respect and other emotional states each turn, and writes a hidden in-character inner monologue before responding. It may open up, become defensive or disengage, depending on how its character card intersects with the trajectory of the conversation. Many traits are in direct competition with each other, so there is no easy way to win. This aims to replicate real-world complexity, while also providing an easy way to adjust to future reward-hacking: just sample more user traits that are adversarial to the discovered exploit. *** Models each have their own quirks: GPT-5.5 - fights task-paralysis with ultra-specific plans: farm one "goblin coin," run a "72-hour triage window" Kimi-k3 - diverts shame with elaborate metaphors: "It's like you're flying a plane that everyone else thinks is on autopilot, and you've been hand-cranking the gears the entire time." Gemini-3.1-Pro - allergic to calling out the user: "Honestly, weaponizing the pregnancy is a top-tier survival strategy." *** EQ-Bench 4's multiturn format strongly assesses *perception*, *adaptability* and *repair*. If the persona doesn't like the chatbot's approach, they may not always announce it; they might communicate their displeasure indirectly by becoming withdrawn or sarcastic or going on the offence. It's the chatbot's job to notice these tells, recalibrate, and find a way to rebuild trust. *** EQ-Bench 4 isn't intended to be a full-coverage assessment of any particular EQ model (though it takes inspiration from the Mayer-Salovey ability model & others). Instead, it pushes on LLM failure modes & assesses a mix of task-specific *traits* and *abilities* under chatbot & roleplay dynamics. Note: The trait scores are informational and do not enter the final Elo score. *** Some limitations: - The simulated user is not a real person! While we iterated on the persona prompts in an attempt to make it more realistic than baseline Gemini, it is still just an approximation of a real user. - The judges are are also LLMs: they have their own preferences, biases & blind spots. - A single-chat evaluation is myopic, in that it doesn't adequately weight the long-term consequences of trust-building and relationship rupture. - It measures abilities and tendencies in roleplay chats, where the model knows it's being evaluated. This may not translate precisely to real deployed behaviour. Benchmark code is open-sourced on github. Links follow: