Early results from Recursive 🚀🚀 SotA results from our open-ended knowledge discovery system: 1️⃣NanoChat 5min pre-training (0.9372 bpb -> 0.9109 bpb, 2.8% lower Bits-Per-Byte than long-standing community SoTA) 2️⃣NanoGPT SpeedRun (79.7s -> 77.5s, 2.8% faster than long-standing community SoTA) 3️⃣GPU kernel optimization (Overall 7.8% better than SoTA performance in SOL- ExecBench, hosted by NVIDIA) To achieve that, our system automatically finds and combines innovations together to create better solutions than current ones carefully designed by expert humans in various domains. We have open-sourced resulting artifacts found by our system so you can check the output yourself. See a full breakdown and technical writeup: recursive.com/articles/first…

A great read. The blog post captures years of progress in mechanistic interpretability and offers an optimistic vision: perhaps we really can open the black box and understand how neural networks work. I share that optimism, but with some reservations. I'm skeptical of explanations that stop at individual neurons, accompanied by compelling visualizations that seem to "explain everything." Humans are remarkably good at convincing themselves. It's easy to cherry-pick evidence that supports a hypothesis, especially when the visualizations look convincing. I'm also not convinced that "Recursive Generalization" is fundamentally different from hierarchical representation learning. As representations become more abstract, they discard details while preserving invariances. Isn't that essentially what hierarchical representations have always been about? IMHO, we need to move beyond a largely descriptive, behavioral approach toward a principled mathematical framework. And we have a natural starting point: gradient descent. Rather than just asking what a network has learned, we should understand why learning dynamics produce these structures in the first place. Of course, this is extraordinarily difficult. Look at OpenAI's 719 mathematical proofs. We see plausible proofs tackling problems at the Fields Medal level, yet no comparable breakthrough that cracks deep learning theory. That contrast is striking. Part of the difficulty is that there are countless plausible theories of how deep networks work. Proving even one can be mathematically formidable. And a proof alone isn't enough. Each theory needs empirical validation, often requiring substantial GPU compute. It's a daunting combination of mathematical and experimental challenges. Still, I believe this is the way forward. Interpretability gives us observations and hypotheses. But ultimately, we need a theory that explains why these phenomena emerge from the learning process itself. That's how we turn compelling stories about neural networks into actual understanding. And hopefully recursive self-improvement can help with that 😀

Thanks for the interview @ziv_ravid! It is a fun discussion 😀

Will be in CoLM conference this Friday afternoon. Won't be able to give a talk (too busy!) but will still join the panel in wdlctc.github.io/efficient-r…. See you all~

We got exact the same comments when we published Coconut😀. However, even with explicit CoTs, the model can still learn to hide the true thinking process. The ultimate solution is always to understand how the model works.

😅 so our StreamingLLM (the original attention sink paper) strikes back again? 😅 -- Caveat: it is in post-training setting.

Interesting findings by our automatic system😆 It finds that some FlashInfer kernels have numerical issues, provides a PR and accepted by human experts🤓

赠Zeyuan Allen-Zhu🫡 世言大模惟精调，君执初心问真章。 动力学中观涨落，表达性里辨行藏。 万里独征云和月，所慕岂在功与赏。 他年若问登临处，格物致理照新航。 突き破れ、扉の向こうへ！ （source: 鋼の錬金術師 ED，《扉の向こうへ》）

We are hiring engineers working on sandbox and LLM inference. If you are interested, please send your CV to talent@recursive.com

🚨A novel way to do RL in LLM post-training! Inspired by our previous path-not-taken work (arxiv.org/abs/2511.08567), we dig deep into the learning trajectory of RL and find that optimizing singular vectors (i.e., rotation) of weight matrices suffices for good performance in RL. The resulting “isospectral optimization” reaches matched scores with substantially fewer training steps. Great work from @zhu_hanqing666 and the co-authors!