we have received a lot of dms asking: how 2 people, without a research lab, trained hobbylm using roughly 100 - 400× fewer gpu-hours than many models in this space so here is the short answer: we spent 76 hours training on 4 h200s, bringing the full base run to roughly 304 gpu-hours. for context: sarvam-1: ~122,880 gpu-hours zoho zia 7b: ~153,600 gpu-hours llama 3.2 1b: 370,000 gpu-hours llama 3.2 3b: 460,000 gpu-hours i will share the engineering decisions, failed experiments, what worked, and the full training story over the next few days. built with @hariharanms95.

hangaroo 🦘 was my all-time fav game in the 2000s, so i made my 1B model (which i built from scratch) play it testing hobbylm in public this week: what can a tiny model actually do? been asked my many found that it is really good at one of the thing: knowing what text is likely. so instead of asking for the answer, i made it rank answers: few-shot prompt: 6 examples like "category: movies → Answer: inception" → constrained decoding: it can only come up with answers that fit the blanks → beam search: keeps the 8 best guesses going, checking the top 400 options each step → probability vote: the letter shared by the most likely answers gets guessed → if nothing fits, it admits it and falls back to common letters (e, t, a…) so it never sees your answer, only the blanks. runs right on my mac and the results were good : 38/60 puzzles solved. a dictionary solver got 11, guessing common letters got 2 what it can't do: idioms, lost to IRON_MAN, here's the vid of my fav hangaroo by hobbylm

life update jan: " gib job " oct: 4 VCs call booked, 3 client approach, 1 NDA to sign, 150k impressions for the launch

the most dangerous number in ml training is 100% gpu utilization. ours showed it for 29 minutes while training nothing. here's how we @hariharanms95 got from 6.1s → 2.5s per training step. when we first ran it, nvidia-smi showed the GPUs running, but there was no loss and no error. nvidia-smi was showing 100% utilization the whole time, but each card was only drawing around 120W, when an h200 actually training pulls somewhere around 600 to 700W. that's when we realised utilization only tells you a kernel is sitting on the GPU and power draw was the number telling us the truth. since we were on 4 GPUs, our first guess was the usual one, that NCCL was stuck waiting on some GPU that never responded, so we ran it again on a single GPU and it hung the exact same way at the same 120W. that ruled out the multi-GPU side completely and left us with two suspects, torch.compile or one of our custom kernels. what finally worked was a compilation change. our FP8 custom ops had their own compile wrappers nested inside a model we were already compiling as a whole, so we removed those and compiled the full model once, and the next run trained normally. but honestly, the hang happened on other platform but the run that worked was on jarvis labs, a different setup. the FP8 path was optional anyway, so we can't say for sure which change fixed it. once it was training, the log showed 6,352 ms per step at step 10, which looked slow until we realised it was a running average that also counted step 0, and step 0 alone took 8,700 ms just for startup. taking that out: ((6,352 × 11) - 8,700) / 10 ≈ 6,117 ms (refer image 2) so the real starting point was ~6.12s and not 6.35s. but if you compare configs using averages with different startup costs, you end up comparing warm-up time instead of training speed. for every test after that we kept one thing fixed, 1,048,576 tokens per optimizer step on 4× H200, because changing the micro-batch also changes how many gradient accumulation steps you need. • seq 2048, micro-batch 4, eager: ~6.12s (32 accumulation steps) • seq 1024, micro-batch 32, eager: ~4.98s (8 accumulation steps) • seq 1024, micro-batch 32, torch.compile: ~2.48s (refer image 3) going from 32 accumulation passes to 8 meant each pass did more work and the GPUs wasted less time on overhead, but since we also changed the sequence length in the same step, we can't really split the credit there. the second jump was only compilation, and that alone cut step time in half. one thing that almost fooled us again was that the first compiled step took around 31 seconds to build the kernels, so any early average made the compiled run look slower than eager. we also tried micro-batch 64, which was a bit faster at ~2.41s per step, but memory per GPU jumped from ~76.5GB to ~131.9GB, which left almost no room for longer sequences or validation, so for a 3% gain it wasn't worth it and we stayed with 32. what that speedup was worth over the full schedule (95,367 steps on 4× H200, step time only): • ~6.12s per step → ~162 hours → ~648 GPU-hours • ~4.98s per step → ~132 hours → ~528 GPU-hours • ~2.48s per step → ~66 hours → ~262 GPU-hours (we made it here) the real run took about 76 hours end to end, with validation and checkpoints included, and cost us roughly $1178. at the speed we started with, the same run would have needed about 2.5× the GPU time, so somewhere around $4,500 to $5,000.

p.s - these models differ in size, hardware, token budgets, and objectives, so this is a comparison of raw accelerator-hours, not normalized efficiency.

p.s - the hardware, token budgets, and objectives differ, so this is context rather than a direct efficiency ranking

We are releasing HobbyLM from @Fuellabsai : a sparse MoE trained from scratch, with 1.037B total parameters and ~305M active per token, on ~100B tokens. Here's how we built 🤞

Data: FineWeb-Edu, DCLM, code and FineMath for the main run; Cosmopedia-v2 and FineMath-4plus for annealing. ~100B is tokens consumed by the schedule, not unique documents. We relied on the upstream maintainers' filtering and deduplication.

Training: ~100B tokens on 4× H200. ~66 hours of main pretraining, then ~10 hours of annealing. About 304 GPU-hours end to end. Weights started from initialization, not from another model's checkpoint.

The architecture: 20 decoder layers. Layer 0 is dense. Layers 1 to 19 are mixture-of-experts: each token goes to 8 of 64 routed experts, plus one shared expert that always runs. So 1.037B parameters in total, ~305M used per token.