Every time an AI types one word, a chip somewhere re-reads the entire model from scratch. For a 70B model that's 140 gigabytes, about 35,000 songs' worth, for a single word. Then it does it all again for the next word.
When you hit enter, the GPU reads your whole prompt in one shot and the math units stay busy. Then it starts writing the answer one token at a time, and every token needs the full set of weights dragged through the memory bus before the next one can start. The math takes microseconds, and the rest of the time the chip is waiting on memory.
So the industry's answer has been HBM, the stacked memory bolted onto every top-end accelerator. And HBM is sold out. SK hynix makes around half the world's supply and has said its 2026 HBM capacity is already spoken for.
Now stack agents on top. A coding agent doesn't make one model call per task. It makes dozens, sometimes hundreds, each waiting on the last. Fifty calls with a few seconds of decode each and you've spent minutes watching a cursor blink. Cut the wait on each call and the whole task gets dramatically faster.
I figured the answer was just more GPUs. Turns out you split the job. GPUs take the prompt, a memory-first chip writes the output, and you call one endpoint under one contract. That's what
@general_compute is doing with
@positron_ai's Atlas systems, with the Titan generation behind it, available Q1 2027.
The bottleneck moved from math to memory a while ago, so whoever moves 140 gigabytes fastest wins inference.