Pinned Tweet
Recent KV cache compression research has focused on small models with dense attention. We scaled these techniques to GLM 5.3, a frontier sparse attention model. We adapted KVzip and Attention Matching, reducing KV cache size by 80% while retaining over 90% of full-context accuracy on the QuALITY benchmark.
9
22
199
21,429
It checks its work much earlier. Argon makes its first check 44% of the way through an attempt. For Opus 5.5, it's 77%.
1
4
390
Gemini 4 Argon is the new top model on Ramp Accounting Bench. It scores 61.1%, 10.6 points ahead of Opus 5.5. Its lead is largest on the hardest tasks.
7
3
75
4,561
We had early access to @NVIDIAAI Nemotron 3.5 Super VL and post-trained it for one of our highest-volume workflows: receipt extraction. One epoch of supervised fine-tuning on 10,000 examples made it competitive with leading open and closed-source models on our receipt benchmark.
5
5
149
15,413
At 750 requests/min, the post-trained model delivered ~68% lower p50 and p95 latency than DeepSeek V4.1 Flash. At 1,500 requests/min, receipt throughput was ~31% higher.
1
5
477
Model selection is as much a systems problem as a quality one: latency, throughput, and cost matter. See more of our work at labs.ramp.com/research
5
390
Two new challengers to the Pareto frontier on accounting work. First Sonnet 5.5, then GPT-6.1 Sol: each one scores more than any model at its price or lower. GPT-6.1 Sol then pushes both GPT-6 Sol and Sonnet 5.5 off the frontier.
6
5
104
13,112
Both list at $2 in and $10 out per million tokens. On our accounting tasks, GPT-6.1 Sol scores higher and costs 45% less per attempt. Sonnet 5.5 is about 2x faster.
1
12
1,392
Recent KV cache compression research has focused on small models with dense attention. We scaled these techniques to GLM 5.3, a frontier sparse attention model. We adapted KVzip and Attention Matching, reducing KV cache size by 80% while retaining over 90% of full-context accuracy on the QuALITY benchmark.
9
22
199
21,429
For KVzip, we use dense scoring and compare shared selection rules that balance importance across heads. For Attention Matching, we compare HAK and OMP selection, then fit per-head biases and shared latent values against GLM’s sparse attention. We fit one layer at a time, accounting for changes introduced by earlier compression.
1
5
675
We had early access to GPT 6 Sol and Luna and ran them on Ramp Accounting Bench. Sol did better than GPT 5.6 Sol at 58% lower cost. Luna did better than GPT 5.6 Terra at 87% lower cost.
7
5
154
23,528
We had early access to Opus 5.5 and ran it on Ramp Accounting Bench. It performs close to Fable 5.1, at 61% lower cost and 1.7x faster.
4
9
182
20,128
Compare Opus 5.5 with every model on Ramp Accounting Bench. Explore the scores, costs, runtimes and behavior breakdowns. labs.ramp.com/ramp-accountin…
9
1,052
Introducing Ramp Accounting Bench. We partnered with accounting professionals to create 137 tasks and grading rubrics grounded in everyday accounting workflows. Even with three attempts, the best model achieved only 21% accuracy. Reliable agentic accounting remains an open challenge.
28
38
470
88,187
We wanted to understand where models tend to go wrong, so we analyzed their mistakes by type. Among the five most common categories, data interpretation and accounting judgment errors were the most frequent.
1
7
2,208