Since my nanoGPT record was published, the question I’ve been asked the most is, “Does it scale?” The true answer is, while some aspects of the speedrun scale, the goal of this specific endeavor was not to scale it up. The point of the challenge is to utilize clever engineering and creative ML intuition to increase efficiency. But while nanoGPT is just about hill-climbing one benchmark, the rest of my work has been focused on scaling to the frontier. The main projects I’ve worked on are some architectural changes and ANVIL III. ANVIL III is a significantly stronger optimizer than ANVIL II, the optimizer I used to beat the nanoGPT world record. In fact, for the record, we removed the parts of ANVIL III that scale. ANVIL III is significantly stronger than Muon, while nearly matching its speed. For the same token budget (same compute), ANVIL III reaches final validation loss 20-28 millinats (.02-.028 nats) lower than Muon, holding at all scales from 124M to 1.2B, and different overtraining depths. As we scale up the model size and training depth, the 20-28 millinats becomes worth a larger fraction of compute. At deep overtraining (8x-Chinchilla) and 1.2B parameters, the fitted scaling curves imply that ANVIL III reaches the same loss numbers at 50% less compute. Of all the scales we tested, the gap is largest at 1.2B, staying between 26-28 millinats. The first remarkable aspect of ANVIL is hyperparameter consistency. The optimal hyperparameters remain highly consistent across different scales and training depths. In most of our tests, varying hyperparameters rarely changed ANVIL III’s loss, or did so within noise. In fact, as shown in the attached image (from the blog’s appendices), we tuned Muon significantly more than we tuned ANVIL III. We performed over 3 times the aggregate amount of runs of Muon than ANVIL III, and at each scale our loss margin uses only the optimal parameters for both. At 1.2B, we used 51 tuning runs of Muon (including muP proxies), compared to only 7 for ANVIL. We wanted to guarantee that our improvement was not a tuning artifact, so we meticulously tuned Muon’s parameters, and had to apply far less tuning to ANVIL III. Our hypothesis is that the high SNR of ANVIL III makes it very resistant to parameter changes. That SNR is the second remarkable aspect of ANVIL III. We generalize the Adam SNR (SNR = m̂ / (√v̂ + ε)) for each singular direction (uᵢ, vᵢ) in an update, and measure how strongly held-out gradients (G_b = gradient on held-out batches b) point along it: SNRᵢ = mean_b(uᵢᵀ G_b vᵢ) / rms_b(uᵢᵀ G_b vᵢ) In our tests, Muon takes a lot of steps with low SNR, but ANVIL III doesn’t, despite being a Muon variant. For example, at 10% of the training tokens into a 1.2B (approximately Chinchilla) model, 90% of singular directions in ANVIL III’s steps go downhill on held-out data, while Muon gets only 67% (random = 50%). Pretraining Blog: hyperstition.cc/cutting-pret… Another recent post from Hyperstition contextualizes the importance of this SNR gap. In The Signal and the Noise, we generalize Adam (in which the update is just a per-coordinate SNR) to every direction in parameter space. Almost every direction is nearly orthogonal to the gradient history, so averaging that over all directions just collapses back into plain momentum. But the single best direction, C⁻¹ḡ, reaches SNR=1 whenever recent gradients are linearly independent, which is almost always true in LLM pretraining. By solving that with a s-by-s Gram system with recency weighting, we get a basis-free optimizer that matches AdamW at 286M parameters. While this is far from optimal, it represents one avenue for possible gains in pretraining optimization that are yet to be realized. hyperstition.cc/the-signal-a… In another fun but mostly unrelated blog, Stuck in a Loop, we show that attention in pretraining creates feedback loops and self-reinforcement. A key’s gradient is proportional to the attention it already gets, so keys with early inflated attention receive stronger gradients, overinflating them. Breaking that loop spreads attention out, with the average entropy of attention heads goes from 3.75 to 4.69 nats (effectively 43 vs 109 tokens), but it makes the model worse. Similarly to The Signal and the Noise, this suggests that putting more weight in high-signal directions can beat spreading it out broadly. hyperstition.cc/stuck-in-a-l… These discoveries are just a small part of the upcoming releases from @hyperstition_cc. Can’t wait until we release more. Big things are coming!

New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it. Specifically: -(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time. -Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used. -Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2. -Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step. -Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params. -Hand-rolled flash attention for 64 dim heads. There are several additions that add accuracy too: -(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15. -(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application. -A couple additional dynamic skip connections in the network. The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large, only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead. github.com/KellerJordan/modd… As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…

New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it. Specifically: -(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time. -Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used. -Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2. -Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step. -Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params. -Hand-rolled flash attention for 64 dim heads. There are several additions that add accuracy too: -(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15. -(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application. -A couple additional dynamic skip connections in the network. The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large, only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead. github.com/KellerJordan/modd… As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…

Excited to have some of my pretraining results published! 1. We’re announcing ANVIL III, the new pretraining optimizer that builds on ANVIL I and ANVIL II (the optimizers I created for the nanoGPT speedrun). ANVIL III takes a new approach, and is much more suited to large-scale models and overtraining, where the advantage grows as we scale up. It features several changes that @Everule2 and I made after the publication of ANVIL II in the nanoGPT world record. ANVIL III, in our internal testing, shaved even more time off the nanoGPT speedrun, but we are not willing to publish the IP at this moment, as it is an optimizer that can make a massive impact on the frontier of pretraining. A SNR distribution curve of ANVIL III compared to muon is attached. ANVIL III also has remarkably consistent optimal parameters across model scales and overtraining regimes, making it exceptionally well-suited to frontier pretraining. 2. Over the past 4 days, I used ANVIL III and architectural changes I had come up with back in July, to train a LLM primarily on web data. These combined to create Feather-1.7B, a 1.7-billion parameter base (only pretrained) model, trained entirely on 200B tokens of public datasets (decontaminated thoroughly across all benchmark test sets). At 180x less compute, it outperforms Qwen3-1.7B, the frontier base model at its size, on math, while matching it on reasoning. It is competitive with Qwen3-4B on math, despite being trained with 414x less training compute. We are unfortunately compute-constrained, so we did zero parameter tuning at 1.7B scale and did only one full training run of this model. Despite likely being far from its optimized frontier, we still outperform Qwen3-1.7B and compete with Qwen3-4B on math benchmarks, without losing any language/reasoning capabilities. 3. Last but not least, the nanoGPT speedrun world record I created and submitted, taking the record from 73.889 to 39.914 seconds. This record marks a 1.851x speedup, the largest ever and larger than the last 45 world records combined. Really excited to continue the work, and looking forward to some of the other insane results we will be publishing soon @hyperstition_cc! Link to pretraining blog: hyperstition.cc/cutting-pret…

Today, we release the first series of our findings. 1 - We cut pretraining costs by 62% at frontier scale (8x Chinchilla), and we further cut all AI inference costs by 30% (Faster decode). 2 - Introducing our model ‘Feather’, which decisively overtakes Qwen 3 as the leading frontier 1.7B math model, using 180x fewer total training tokens, even matching Qwen 4B at potent benchmarks in math. 3 - With Anvil II, our state of the art LLM Optimiser, we record the largest pre-training efficiency jump on the NanoGPT Speedrun, leaping the previous by 34seconds - Our record singularly, is a greater percentage drop than the past 45 World Records combined. The website : hyperstition.cc