Check out this post by my amazing co-author @chandavidlee with more details on Lafa!

Some new work where we tried to understand how much changing the optimizer can reduce forgetting when training on offline data 👇

Just read a fun paper where they adapt off-the-shelf Qwen models with under 30B training tokens, and the result runs at 2.54× the throughput of a same-size autoregressive model at comparable quality. The idea is to have an LLM write the way image generators draw: it still goes left to right, but a chunk at a time, and each chunk starts as random noise that gets refined into real tokens all at once. Everything else works like a normal LLM, which is why they could just convert an existing model instead of training one from scratch. Each token in a chunk gets cleaned up at its own pace, and the model learns those paces itself during training. What it ends up learning is that earlier tokens should come into focus first. Nobody told it to write left to right. It figured that out on its own. Learning all this also made it a better normal LLM. Compared with the same Qwen model fine-tuned the usual way on the same data and budget, its average score across six benchmarks went up at every size (0.5B: 28.7 → 31.6, 7B: 58.4 → 60.9, 8B: 56.8 → 63.1). The authors think it's a multi-token prediction effect: practicing on whole chunks makes it better at predicting the next token too. That normal-LLM side is also what makes it fast, because the model can check its own drafts. It takes one quick shot at a whole chunk, then checks it the normal next-token way and keeps everything up to the first token it disagrees with. Since every token gets checked, the output is exactly what it would've written one token at a time (greedy decoding), just faster. With the 7B model on a single H100, that's about 2× faster and a bit more accurate with 4-token chunks, or 2.54× faster at about the same accuracy with 16-token chunks. For comparison, other speedup tricks they tested, like DeepSeek-style multi-token prediction and MARS, keep the quality but top out at 1.4×. The diffusion LLMs they compared against all lose quality, and lose more the faster you push them.

Next step: Scale to strong models such as Qwen3.8-27B!

Large Flow Language Models Lafa shows continuous flow models can scale to 8B parameters by adapting pretrained autoregressive models with under 30B tokens, matching autoregressive quality across instruction following, reasoning, and coding benchmarks. arxivb.org/abs/2610.28559

Another blockwise continuous diffusion language model at 8B scale, adapted from a pretrained autoregressive model, with flow map distillation and self-speculative decoding for faster inference. Cool😎

Saying solved is a stretch :) but we are excited to identify the failure cases of SST and push continual learning ahead.

This is some really great work on training large scale flow based language models. They show you can adapt an existing autoregressive backbone, like Qwen, and turn it into a non autoregressive flow model.

Also check out the lead author @ArnavChavan6's explanation!

Tagging some people who might be interested: @sedielem @PatrickPyn35903 @jdeschena @zhihanyang_ @ArashVahdat @ArnaudDoucet1 @iNikitaGushchin @osclsd @FEijkelboom @WeiGuo01 @pengzhangzhi1 @rohit_dilip_ @ssahoo_ @linluqiu @JeongjaeLee0