deepseek flash update pretty decent on the connections eval. very cheap, but pretty slow (typical problem with these OSS models is they don't have good ways to control how many tokens they consume)
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated.
Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
After Bobby learns to make burgers on a charcoal grill, Hank accidentally joins the local DSA chapter when attending a “No Kings” protest he mistook for “No Kingsford”
This is hilariously predictable. These healthy sodas somehow taste worst than a classic Coke etc and also worse than lightly flavored seltzers. Like who are they for?
Poppi isn't meeting expecations for Pepsi after buying it for $2 bn: "Poppi is underwhelming: Pepsi now expects to pay almost none of the $300 million bonus it promised the sellers if the brand hit its targets" wsj.com/business/its-too-lat…