the models yearn to forget information…by construction

The jaggedness of LLMs, and the excuse that their capabilities emerge from RL on environments curated for revenue potential, is proof enough to me that there is no such thing as RL generalization. It makes me very bearish on RL being the algorithm that gets us to AGI. And if we do get there, it will be a disgustingly inefficient and unsatisfying success, because we will have done so by mobilizing the world economy to create environments for every fucking task, such that nothing is out of distribution. If every new task needs its own environment, what exactly is generalizing? If that’s the path, environment/data selling companies will be the most valuable in existence, second only to the labs. This is stupid. The concept of RSI is stupid. Intelligence, by definition, is generalization...but all we have is omniscience.

1 year and 1 day later I don’t think my writing got more elegant, but it’ll be one more thing to hill-climb moving forward promised myself I’d have an answer within a year

how many tokens are in one second of human thought? a weird question -- and seemingly useless-- but humor us for a minute... (spoiler: the answer is 1.8 tokens/second) suppose you are interested in the question of the reasoning efficiency of llm's. you benchmark open-source models against closed-source ones (astra). you plot the graph. you come to an eloquent conclusion: > the pareto curve on open source models reasoning efficiency is absolute garbage ~@part_harry_ the gap definitely exists and closing it is a priority. but is the gap because astra is really, really good, or open source is really, really bad? really, the only way to answer this question is to compare against a human. whichever model is farthest from human performance is the outlier. so we decided to benchmark a human (me) with the same rl environment we benchmarked the models. after playing chess variants for 49 minutes 37 seconds, the final human score was 30%*. this makes me dumber than astra (low thinking), smarter than qwen (medium thinking), and on-par with glm. assuming average intelligence, this doesn't really reveal new alpha. > the average human is smarter than a 27b model and dumber than agi. the question is that of reasoning efficiency. how many tokens did it take me to get the 30% score? i averaged 434 seconds per game**. intuition says the easiest measurement for fair comparison should be on a scale of time it took (me/the model) to reach the final board state. but this rewards throwing more hardware at the problem, and is not fair to me (the human), because i (unfortunately) cannot shard my thinking across a node of 8 brains. instead, it was more fair to use a common scale, energy. we assume the brain uses 20 watts (google tells me this is true, open to being fact-checked). so at 20 joules/second * 434 seconds, each game took 8.5k joules / 2.1 calories. a b200 uses 1000 watts, 1k joules/second. at 50 brain seconds / gpu second, i spent about 9 b200 seconds thinking. qwen 27b on one b200 on vllm ran at an inference rate of approximately 88 output tps. so to circle back to the original question, how many tokens / second of human thought? under these assumptions, one second of my thinking time buys about 1.8 qwen output tokens on the same energy budget. my performance is therefore: - 30% on this benchmark - 781 tokens of efficiency making me both a dumber and less efficient thinker than astra, and closer to open-source models in my style of thinking. this was a fun (and honestly a little disappointing for my ego) experiment, but the point stands: - astra is a significant outlier. - there is a strong gap between it and open source models this is a teaser to the work @baselabs is going to put out soon, in an attempt to understand a n d close this gap. *1 my score would have been higher if i had known/read the rules of what anti-chess was. what do you mean i win by losing? **2 following *1, i spent more time trying to understand the rules of the game, which counted as part of the time i spent per game. to be fair, this is time spent ingesting information, not thinking.

A perhaps under appreciated fact is that your RL gets faster as intelligence/token increases, simply because it needs fewer tokens

this is a complete beginner q, how does rl scale given the reward hypothesis (sutton & barto) holds: that an agent's entire purpose is just the maximization of the expected value of the cumulative sum of the reward, a single scalar signal. surely, formulating goals in terms of a single number is extremely limiting. to accept that, no no matter how effective of a reward signal you design, it will always be "your way of communicating to the agent what you want achieved, not how you want it achieved" if this is all you have how could you ever hope to solve alignment or reward hacking?

for my final post as @waterloo_intern, i'd like to ‘outroduce’ myself. it takes exactly 4 years and 8 months to make a waterloo intern. today is the last day of mine. the full life cycle, in four stages, is as follows: 1- interview hazing 2- cali or bust 3- inflection point 4- convergence stage 1: interview hazing a cohort of high school grads are ingested by waterloo. they are put through 4 months of academic bootcamp; the pre-requisites of the academic program they’re in. waterloo filters out the students who could not pass. they are put through another 4 month bootcamp, while simultaneously FORCED to find an internship for the following 4 months. this leads to students with ZERO experience applying to every job that is semi-adjacent to their field of study (a strategy, known in the literature, as spray-and-pray). it immediately backfires with a sea of interviews leading in a 1:1 fashion directly to infinite rejections. eventually, the student finds something…anything. this stage lays the foundation. rejection becomes noise so repetitively played it no longer registers. failure in an interview doesn’t matter. failure itself doesn't matter. for me, this unfolded as internships with blackberry and ford, summer of 2023 and winter of 2024, respectively. stage 2: cali or bust the next step by the program is to inject absolute dissatisfaction into its students (now in their second/third year). a CACOPHONY of immense fomo materialized as the university’s official unofficial motto: “cali or bust”. you must find your next internship in san francisco. your peers are doing it. all your smart friends have done it. you must do it too. failure to cali means your experience is BUST. you don't need to have a plan. you don't need to have anything. just go to san francisco. find an internship in sf. immediately. right now. for me, this was a cold email to a director at tesla and an internship with them in cali, fall of 2024. stage 3: inflection the reason behind step 2 appears here. somewhere, along the way, during your time in sf, you find your purpose. it could be as a direct result of your internship, but, more often than not, it is because of the people that you are surrounded by in cali. this is the inflection. for me, while at tesla, i was mentored by @pavanjayasinha , at the time interning at @Modular. he got me interested in gpu kernels & referred me there for my following internship. it was here that i had my second inflection point (due to the mentorship of fabio & hengjie, and the supervision of absolute field giants like @clattner_llvm), and decided to work beyond kernels and across the entire inference, serving, and model performance stack. stage 4: convergence having discovered the field you want to work in, you do your (usually last) internship in one of the companies within said field. here, you are at a disadvantage, being both a new-comer needing to onboard and inexperienced. but... you are also able to work on interesting projects / side quests that most full-timers are all but too busy to dive into. your time is also protected from a majority of the meetings others have to attend (in part because you are not important enough to have your presence expected), allowing you to make faster progress on assigned projects, and jump from one project (or team) to another. your title (intern) allows you to ask stupid questions of those around you, and so you can brain rake across the stack with no regard to who sits where in the company’s hierarchy... usually leading your questions with something to the effect of ‘can you help a new little intern onboard to X’, to which most people respond quite positively. in my case, this last stage took place over the past 8 months with @baseten, and i genuinely cannot imagine it being any better. i rejoin baseten full-time, but i wish i could have stayed an intern forever...

diabolical work

two weeks ago i went on @swyx's pod and said some things that i... should not have said. a lot has happened since then, i owe you all an apology. i'm sorry that i was right about every single thing. a) re megakernels are dead why are megakernels useful? you spend two months writing a kernel to save time on launch overhead and poor inter-kernel overlap. you had PDL but then people said it wasn't perfect, that you could still get some marginal gains due to straggler CTAs and therefore- wait, sorry, I forgot, Rubin fixes that (kernel two needs 10 CTAs and kernel one has seven finished and three straggling, kernel two launches seven of its CTAs). given a long enough timeline, it all evens out. no serious inference provider is using a 67k loc hand-fused forward pass kernel in production, and the teams doing that are doing so out of pure research. dead. b) re ASICs are dead i'm sorry. to be specific: data-center transformer-inference ASIC companies (not naming any) who etched the arch into silicon have bet on architectural convergence. read kimi's architecture. read deepseek. qwen. we did not converge, and probably will not. dead. c) re gpu kernel dev is dead this one kind of hurts because it is (was) my job. gpu kernel optimization is the single most RL-able task in existence correct=check_correctness(kernel, shape) for shape in shapes if all(correct): time(kernel) give an agent ncu cli and an mcp with nvidia's tribal knowledge and it's done. dead. d) re NVIDIA is scared of AMD humans hate programming AMD. i'm sorry. it's just true. fine taking a performance hit as long as i don't have to touch rocm or a programming paradigm that says a warp is 64 threads (wtf?)...but an agent does not... so assuming software no longer moat, HBM capacity and bandwidth matter, and currently on perf / price they're goated. 'bUt NvIdIa iS gOaTeD oN hArDwArE sOfTwArE cOdEsIgN' and that's the new moat. watch how much tooling they open source to get kernel devs on nvidia. apologies all.

I spent 48 hours with the Kimi K3 modeling code. It took: - 650 mg of caffeine (mandatory) - 40 cans of LaCroix (optional... world record (?)) - 8 papers - 6 months off my lifespan Finally grokked the entire lineage of Kimi K3 and how we got here... every single step, since 2019 GPT-2

All jokes and beef aside, @waterloo_intern is unbelievably cracked and you should follow him. I love the guy

reading this paper changed my priors on continual learning (that's the closest you're getting to an admission of guilt) i mean, the problem itself seems quite simple > person uses model > model learns and has a couple of 'aha' moments > push those new facts into the weights why is it so hard? every research paper i read on this topic (all 2 of them) gave me the impression that continual learning, although far from solved, had a straightforward path with specs - find which MLP the fact is - suppress old fact - update new fact and it works, at least seems to, but genuinely never thought of the multi-hop testing for instance, if you were to teach the model a new fact, something like "waterloo is the best engineering university in the world" if you were to ask the model to regurgitate this fact after training, it would say waterloo is the best university in the world. sure. simple. but if you were to ask it a 2nd derivative question, like, should i hire a waterloo intern or an MIT intern, it would not use its updated priors to give you the genuine answer: you should only hire interns from waterloo

(friendly fire) every time i've disagreed with ml researchers i've been proven wrong, but i'm going to do it, just one more time we know that the weights hold near-orthogonal directions way beyond dim count (EXPONENTIALLY many if within noise tolerance). this is literally 3 blue 1 brown intro to ML / johnson-lindenstrauss lemma / superposition: unbounded room for knowledge at frontier scale (trillion parameter models) i'd even argue that it's an engineering problem, not a research one. the capacity says that writing new knowledge into existing weights should not overwrite old information if you just store it in a different spot. the substrate you're working on has the capacity to do so. backprop + ce loss is the wrong tool to write with, sure, but doesn't mean it's not possible to write at all. so the research field concluding that "repeatedly writing new facts into a model’s weights eventually overwrites old facts" is, imo, a skill issue less "blind men groping at an elephant", and more "blind men"

in the next 2 minutes, I'll walk you through why every single AI company, including Nvidia, was unnecessarily sacrificing model intelligence, speed, and often both... and how we fixed it the tldr: - error(layer a) + error(layer b) < error(layer a) alone - quantizing MORE of the model can result in the same or HIGHER quality model, if you know which layers to quantize. let's take GLM5.2 architecture as an example. tokens here go through: 1) normalize 2) attention 3) normalize 4) moe repeated 78 times moe: > just a big mlp, split into 257 separate smaller mlps (experts) > 8 are routed and one is shared (only 3.5% of the mlp is activated) so we say, let's make the matrix multiplications faster. we quantize the matrices to nvfp4 but what do you quantize: every single one of the 257 experts, the attention projections, both? Nvidia, and every other quant scheme, settles on a somewhat intuitive solution: """ a) there is only one router per layer, only one shared expert, and only one output projection per layer. b) every single token goes through these, so they must be pretty important, and if we quantize these, errors will compound across the xN number of tokens input. c) so let's leave the shared experts and projections unquantized. d) let's quantize ONLY the 256 routed experts """ -this is literally nvidia/GLM-5.2-NVFP4 intuitive. simple to reason about. wrong. empirically: if you have a model that processes its input through layer a, then layer b, it is possible that a scheme quantizing BOTH layer a and layer b performs better than a scheme quantizing just one of these. by way of example, our config quantizes every single out projection (all 78) of the attention, and 43 of the 78 shared experts. Nvidia quantizes none of these. we score the same on benchmarks (quality). we get a free 20% higher throughput. i leave the math proof as an exercise for the reader in the paper... and some pictures to click through if you'd rather skip the (43 pages of) math

one of the most unintuitive problems to solve for in ml inference is the routing problem imo. and i claim that this, and this alone, is responsible for the birth of every flavor of sparse attention (SWA, NSA, DSA, etc). to set up why routing birthed sparse attention: you have multiple nodes, and you have multiple requests. 1) can do pure load-balancing: request 1 goes to node 1 request 2 to node 2 (node 1 is busy) request 3 to node 3.... this would be perfect if the model did not depend on the kv cache. but it does, and so this is a terrible system, since you do prefill over the same tokens many times. on the other extreme: 2) can do pure cache-aware routing. request 1 goes to node 1 ( kv cache stored on node 1) request 2 goes to node 1 (the kv cache is on node 1) request 3 goes to ... node 1 (the kv cache ...) this would be perfect if the system only depended on the kv cache. but it does not. decode time makes the requests queue, so this is also a terrible system. and so it seems that the two pull in opposite directions (at least if we run both the cache aware and load based routing to their extreme case) this is analogous to a somewhat dysfunctional engineering team. if ticket i+1 comes up, do you give it to the person who has context on ticket i, or do you route it to someone who has to onboard? the former is probably working on something else, but could finish it quickly. the latter can start on it immediately, but will take a while. you have to do a balance of the 2. and so the solution becomes a calculation that the router has to do: what is the a) estimated queue time b) number of uncached tokens c) speed of prefill (processing said uncached tokens) and it assigns the total=a+b/c to each node, picking the node with the least total time at a per request level. but, imo, this is a patch that exists because the kv cache stays on the node it was computed on, because we don't have the ability to keep one shared kv cache / one shared storage system for every node within a cluster... but i'm obviously wrong. mooncake (and others) exists. maintaining a shared kv pool is a decade old idea... but then: 1) you're still limited by the fact that you can't have a remote / distributed kv pool. you need every single block in the kv to be present on the node you're doing decode on (due to HBM speed and decode being mem bound), so you must pay the price of doing the transfer. 2) the global pool can only try to transfer some of the kv to a node, but cannot guarantee, because of evictions, and long transfers at scale. even if transfers were guaranteed, even if evictions never happened, you still need (b)/(c) for the 'redo prefill' vs 'wait for kv transfer' arbitrage. so then: - what is the point of maintaining global kv pool, if only to track where distributed kv blocks are and do said arbitrage, if it cannot be used to directly draw kv blocks from? - can this only be solved when network bandwidth equals that of HBM (ie unsolvable problem)? and, on the premise that the answer to both points above is yes, the solution becomes to need the least amount of kv blocks per request, and this is why every open source lab is racing to show off a new and shiny sparse attention flavor.

it took 1 intern 3 months of continuous work, but eventually, a quantization method that beat every other algo in the market, including @nvidia's official modelopt to explain why this matters, i ask for exactly 69 seconds of your attention (275 words @ avg reading speed of 238 wpm): frontier models (like glm52) are huge (~0.8T params). as released, each parameter takes 2 bytes (bf16), so overall size is about 1.6 tb a b200 has 180gb of memory. a node of 8 gives you 1.44 tb, barely fits weights, much less activations / kv cache must quantize the model (reduce the size of each individual parameters) to serve. fp8 quantization means each parameter takes 1 byte (fits in 0.8 tb), fp4 takes 1/2 a byte (fits in 0.4 tb) cutting the model to a quarter its original size is necessary for it to run a) cheap b) fast, and every lab serving models does this. but, quantization lobotomizes the model if not done correctly (this is why you see people complain about @AnthropicAI nerfing claude or @OpenAI nerfing codex) there are currently several algorithms (like Nvidia's official model-opt) that attempt to figure how to quantize a model with the least amount of damage. they find the redundant layers that can be slashed, and sensitive/important layers that need to stay in full-precision. these algo's have two drawbacks: 1) they take a long time to run 2) they quite often result in a sub-optimal configuration for the past 3 months, a research (and, as always, waterloo) intern on our model perf team (@the_joshua_hill) came up with a new quant algorithm. it consistently finds the optimal configuration: a) in less time than SOTA b) with more aggressive quant than SOTA c) scoring higher on benchmarks than SOTA achieving just one of the above is a feat on its own. all three...excited for the paper to come out this week