Day 16 on the
@droid Max plan ($200)
Wanted to know if GLM 5.3 Flash and Nemotron 3.5 Super are actually worth using in Droid, since they're both super cheap in the picker (0.06x and 0.08x). So I benchmarked them.
23 Python tasks from
@VulcanBench, 3 runs each, and hidden tests decide if a task counts as solved. Ran DeepSeek V4.1 Flash and GPT-6.1 Sol next to them so I had something to compare against. 690 runs total and my laptop lowkey almost died doing it 😅
Sol is on top at 87%, but it costs 26k to 46k credits a task.
The surprise was GLM 5.3 Flash on low. 80.6% solved at about 5.7k credits a task. That's half of what DeepSeek low costs and about a fifth of Sol low. Easily the best value in Droid right now.
DeepSeek V4.1 Flash on high is still the sweet spot if you care about getting it right. 86.8% for about 14k credits, so almost Sol level for around half the price, and it's fast too.
Nemotron 3.5 Super was the disappointing one. Same score as GLM on high but almost 3x the credits. Honestly couldn't find a job where it wins.
So this is how I'm splitting it now. GLM 5.3 Flash low for exploration, mechanical edits and anything high volume. DeepSeek V4.1 Flash high as my default coding worker. Sol only for the hard stuff where getting it right matters more than credits. GLM max barely helped (82.3% for almost 3x the credits of low) so I'm skipping it.
It's one Python suite, so take it with a grain of salt.