看到开头写着“清北的编程课”，一时间完全没绷住。 似乎他心中觉得清北编程课很好一样 😅😅😅

期待我们完成 SGLang Omni MLX 的 TTS stack 后，继续支持关伟老师的力作！

A few days ago, our SGLang-Omni team experimented with MiniMax M3.1 in a cascaded system. Today, seeing the official release of M3.1 Flash, I spent some time studying its architecture, particularly MiniMax Sparse Attention (MSA). I found the design genuinely interesting! 🤔 M3.1 Flash is a 428B MoE model with only 23B activated parameters per token, supporting up to 1M tokens of context. Beyond MoE, three architectural decisions caught my attention. 1. MiniMax Sparse Attention (MSA) replaces expensive full attention with a lightweight Index Branch that selects relevant KV blocks, followed by a Main Branch performing attention over those blocks. With Top-16 selection and 128-token blocks, the main attention computation can focus on roughly 2K historical tokens instead of the entire context. What I particularly appreciate is that sparsity is incorporated into training through KL alignment and full-attention warmup. The model learns to operate under sparse attention rather than relying on post-training approximations. 2. Quantization-Aware Training (QAT) brings low-precision computation into training for routed experts and attention paths, helping address the substantial memory and bandwidth costs of a 428B MoE model. 3. DSpark Speculative Decoding uses a draft head to propose candidate tokens for parallel verification, reducing the number of expensive sequential decoding steps when acceptance rates are favorable. Meanwhile, the model maintains competitive agentic performance, including 61.7 on OSWorld 2.0 and 73.7% on DeepSWE v1.1. I really like this direction: model architecture and inference efficiency should be co-designed from the beginning, rather than optimized independently after training. Congratulations to the MiniMax team!!!

8,192 concurrent sandboxes and ~1.3 million rollouts. Really impressive engineering from @IFM_AI! One thing I find particularly interesting about long-horizon agentic RL is that infrastructure failures can directly affect training correctness. If a sandbox crashes halfway through a trajectory and the system treats it as a failed task, we're effectively penalizing the model for something it never did wrong. At this scale, distinguishing environment failures from actual model failures becomes part of the learning algorithm itself. RL360 also demonstrates why modular RL infrastructure matters. With Miles handling rollouts and optimization, Harbor managing agent environments, SGLang serving inference, and Megatron-LM running training, each component can evolve and scale independently. I'm especially happy to see Miles and SGLang being used together in such a large-scale, real-world RL system. The lessons about asynchronous execution, distributed trajectory management, and sandbox reliability are incredibly valuable for the entire open-source community. Huge congratulations to the IFM team! github.com/ifm-ai/RL360

这是错的，因为我会在推荐别人的时候，让人在简历上面写上自己的 PR，而且我会把他们写的好的 PR，非常认真的给这些公司的朋友讲，为什么我觉得这个人写的 PR 是好的。 如果你想要用开源 PR 的方式来证明自己，那么请你对这些 PR 保持审慎，同时感到自豪。

Perhaps unsurprisingly, something rather absurd has happened in the AI industry over the past two or three years: young researchers have increasingly impressive resumes, more top-tier publications, and more internships at leading model companies, yet their bargaining power in the job market has started to decline. Just a few years ago, companies were willing to offer million-RMB annual salaries to young researchers who had yet to demonstrate any significant industrial achievements, betting that they might eventually become exceptional talents. Today, even PhD graduates with stronger research records may struggle to get an interview at those same companies. I think the most interesting part of this change is that AI talent hasn't suddenly become less valuable. Rather, the industry is shifting from paying a premium for intelligence and potential to paying for capabilities that have already been demonstrated. The problem, of course, is that young researchers often need an opportunity to prove themselves before they can demonstrate those capabilities. From an economic perspective, this shift is understandable. Over the past few years, foundation model companies were racing to build their teams while technical directions remained highly uncertain. They were willing to pay a premium for intelligence and long-term potential, essentially purchasing an option on someone's future contributions. Now that these teams are maturing, their needs in Pretraining, Post-training, RL, Inference, and other areas have become much clearer. Hiring is no longer simply about bringing smart people into the company and figuring out what they should work on afterward. Instead, companies first identify the problems they need to solve and then look for people capable of solving them. This is certainly good for short-term efficiency, but it also means that young researchers without specific industrial experience can no longer rely on potential alone to get opportunities. Meanwhile, traditional signals such as academic credentials, publications, and internships are rapidly losing their ability to distinguish candidates. Having several top-tier papers or participating in the training of an important model is not necessarily the same as possessing independent technical judgment. I've always believed that one of the most important ways to evaluate engineers and researchers is to understand why they chose a particular technical approach, whether they can explain their experimental results, and whether they can independently solve unfamiliar problems. These qualities are difficult to assess from a resume. You often only begin to understand them after actually working with someone. As a result, personal recommendations and previous industry collaborations are becoming increasingly valuable. There are perfectly reasonable explanations for this, but it also risks reinforcing existing professional circles. And this brings me back to something I've said before: whether in hiring or academic peer review, our AI community seems to be gradually moving backward from the imperial examination system to the Nine-Rank System of ancient China, where social standing and personal connections played an increasingly important role in determining opportunity. To me, this is a step backward. What's even more unfortunate is that talent development itself depends heavily on opportunity. If a young researcher has never had the chance to participate in large-scale model training, how are they supposed to demonstrate that they are capable of doing it? And if hiring increasingly favors candidates who already have industrial experience, we end up with a self-reinforcing cycle: those who entered core teams earlier continue to grow because they have access to compute and important projects, while those who come later struggle to obtain those same resources because they lack the required experience. We tend to attribute past achievements entirely to individual ability, overlooking how profoundly the environment itself shapes the development of talent. I've always believed that open-source communities offer something uniquely valuable here. In AI Infrastructure, many engineering capabilities can be demonstrated through public PRs, benchmarks, design discussions, and sustained maintenance. Even without an internship at a prestigious model company, someone can establish their reputation through actual contributions. Of course, open source cannot entirely overcome the compute barriers in areas such as Pretraining, but it can at least create additional opportunities for young researchers outside established circles. I think it's perfectly reasonable for the market to become more selective, and I don't believe companies should be expected to invest unconditionally in every promising young researcher. But I still hope that, while pursuing people who can contribute immediately, the industry retains some patience for developing young talent. After all, many of the highly experienced researchers being pursued by companies today were once young people with no industrial experience, who needed someone to believe in them and give them a chance.

不出所料，过去两三年，AI 行业发生了一件颇为荒诞的事情：年轻人的履历越来越漂亮，顶会论文越来越多，头部模型公司的实习经历也越来越丰富，但他们在人才市场上的议价能力反而开始下降。前几年，公司愿意开出百万年薪，赌一个尚未做出工业成果的年轻人未来能成为顶尖人才；如今，履历更加优秀的博士生，可能连进入同一家公司的面试机会都难以获得。我觉得这背后最值得思考的，并不是 AI 人才突然不值钱了，而是行业正在从为天赋和潜力支付溢价，转向为已经被证明的能力支付溢价。可问题恰恰在于，一个年轻人要证明自己的能力，往往需要先得到一个证明自己的机会。 某种角度上，这是一个可以被理解的经济因素选择。过去几年，各家大模型公司忙着组建团队，技术路线又充满不确定性，公司愿意为年轻人的聪明才智和未来潜力支付溢价，相当于购买一个长期的期权。现在，团队逐渐成熟，大家对于 Pretraining、Post-training、RL、Inference 等方向的技术需求也更加明确了。 招人不只是先把聪明人招进来，再想办法给他找事情做，而是先确定自己有什么问题需要解决，再找一个能够解决问题的人。这对公司的短期效率当然是好事，但也意味着那些暂时缺乏特定工业经验的年轻人，很难再单纯依靠潜力获得机会。 学历、论文、实习经历这些传统信号的区分度正在快速下降。一个人有几篇顶会论文，参与过某个重要模型的训练，和他真正拥有多少独立的技术判断能力，毕竟是两回事。我始终觉得，对于工程和研究人才，一个很重要的评价标准是他为什么选择某个技术方案，能不能解释实验结果，以及面对未知问题时有没有独立解决的能力。 这些东西在简历上很难体现，只有真正一起工作过，才会逐渐了解。于是熟人推荐、工业界的合作经历越来越有价值。这有它合理的一面，但也很容易进一步强化已有的圈子。 当然，这就是我说的，无论是招聘人才还是审稿，我们的 AI 圈子已经逐渐从科举制回退到了九品中正制。这是历史的倒车。 更让人有些遗憾的是，人才的成长本身就高度依赖机会。一个年轻研究员如果从来没有机会参与大规模训练，怎么证明自己具备大规模训练的能力？而如果招聘又越来越看重已经积累的工业经验，就很容易形成一个循环：先进入核心团队的人，因为拥有算力和项目而继续成长；后来者因为缺乏这些经历，越来越难获得同样的资源。我们很容易把既有的成果全部归因于个人能力，却忽视了环境本身对于人才成长的塑造。 我一直认为开源社区在这里有一种独特的价值。对于 AI Infra，很多工程能力可以通过公开的 PR、Benchmark、Design Discussion 和长期维护来证明。一个人即使没有明星公司的实习经历，也有机会通过实际贡献建立自己的信誉。当然，开源无法完全解决 Pretraining 等领域的算力门槛，但至少能给一些不在核心圈子里的年轻人提供额外的机会。 市场开始更加理性地选择人才，我觉得是正常的，也不认为公司应该无条件为年轻人的潜力买单。但我还是希望，大家在追求即战力的同时，能够保留一些培养年轻人的耐心。毕竟今天那些拥有丰富工业经验、被各家公司争抢的优秀人才，很多也曾经是一个没有任何工业经验、需要别人愿意相信他的年轻人。

Proud of the team. 297 agent memory systems, 82 benchmarks, and 5 modalities. Really impressive work by Minghao and the team! I've become increasingly interested in agent memory, especially as we move toward continuously interactive AI systems. An agent that can see, listen, speak, and operate software will accumulate enormous amounts of experience over time. But simply storing all those interactions is unlikely to be enough. The harder question is how an agent decides what is worth remembering, how to represent it, when to retrieve it, and how to update its memory as new experiences arrive. What I particularly appreciate about this survey is its distinction between static memory reuse, dynamic memory updates, and self-evolving memory management. The last category is especially interesting: an agent could potentially improve not only its decisions using past experience, but also the policies governing how it remembers and learns. I think this will become an increasingly important research direction for Personal Agents and Interactive Intelligence. Memory needs to be evaluated through its impact on downstream behavior and long-term task performance, rather than retrieval accuracy alone. Congratulations to Minghao and all the collaborators! Looking forward to reading more.

Yesterday, we announced SGLang on Google TPUs. Today, we're bringing SGLang to AWS Trainium2 and Trainium3! This has been quite a week for open-source AI infrastructure. Built in close collaboration with @awscloud, our new TorchNeuron-based plugin brings inference serving and RL rollouts to Trainium, with initial support for Qwen3, DeepSeek, and GLM model families. What particularly excites me is the heterogeneous RL direction. Rollout generation and model training have very different computational characteristics, and there's no fundamental reason they must always run on the same type of accelerator. We're now exploring how Trainium can handle rollouts while @radixark's Miles connects them to the broader RL training workflow. As someone working on AI inference infrastructure, I find this direction especially meaningful. Supporting a new accelerator is just the beginning. Making scheduling, serving, model execution, and RL workflows work efficiently together across different hardware is where much of the interesting engineering begins. It's still an early beta, but I'm excited to see how far we can take this collaboration. Huge thanks to the AWS engineers and everyone in the SGLang and Miles communities who made this possible! More to share at the SGLang Summit next month!

Some of the biggest improvements in LLM inference come from figuring out which computations we shouldn't have to perform twice. SGLang HiCache is a great example. What started as CPU offloading in early 2025 has evolved into a hierarchical caching system spanning GPU HBM, CPU DRAM, and external storage. As agent conversations grow longer, retaining reusable history becomes increasingly important. And making this work efficiently involves much more than moving KV cache around: we need prefetching, overlapping data transfers with computation, smart eviction policies, and correct recovery of hybrid-model states. The results are impressive. In our DeepSeek-V4-Flash multi-turn benchmark, adding a Mooncake L3 tier increased effective input throughput from 9.4K to 145.5K tokens/s, a 15.5× improvement, with nearly 98% cache hits. Importantly, this measures effective throughput including reused cached tokens, rather than raw GPU computation speed. Really happy to see @YichiZ03 put together such a thoughtful and beautifully visualized technical deep dive. These are the kinds of engineering details that make open-source inference systems genuinely useful at scale. Huge appreciation to Yichi and everyone who contributed to HiCache. Great work!

This is one of the most interesting AI-for-research benchmarks I've seen recently. Huge congratulations to @MangQiuyang, @wenhaocha1, and the entire FrontierCS team! The design is particularly thoughtful. Take 48 published CS papers across 13 research areas, remove their key contributions from the original repositories, and ask AI agents to develop better solutions without access to the papers or the web. The final implementations are evaluated against the original human contributions on hidden workloads, and passing requires beating the reference on every checked metric. As someone working on open-source AI inference infrastructure, I really appreciate this evaluation philosophy. Producing plausible code is relatively easy to demonstrate. Developing an optimization that actually improves performance, preserves correctness, and generalizes beyond the workloads used during development is a much harder problem. The initial results are fascinating: the strongest agent passes 58% of its runs, while none of the evaluated agents passes 16 of the 48 tasks. Of course, reconstructing and improving an existing research contribution is different from identifying an entirely new research direction. But I think this is a meaningful step toward measuring how much genuine R&D work AI agents can accomplish. Really excited to see how this benchmark evolves, especially for ML systems research!

I’ve said this more than once: building large models is an art of management. I strongly dislike the claim that “top talent doesn’t need management.” Low-ego teams + consensus + clear rules and processes. Every good company I’ve seen works this way.

我不止一次说过，大模型是管理学的艺术，非常反感“顶尖人才不需要管理”这种说法。 low ego 的团队 + 共识 + 制度 我见过好的公司全然如此。

你能不能算算，高中生进 OAI 的比例再说？孤例不证这种事情还需要人来教？

Meta is running SGLang in production on Google's TPUs. And the stack is open source This is one of the things I love most about working in open-source AI infrastructure. SGLang-TorchTPU is the result of a remarkable collaboration between engineers at @Google, Meta's @PyTorch team, and our SGLang team at @radixark. What's especially exciting is that we're bringing much more than basic model execution to TPU. Prefix caching, speculative decoding, KV cache offloading, disaggregated serving, and the rest of the machinery needed for production agentic workloads can now run through a PyTorch-native TPU backend. The custom MegaKernel also delivers a reported 770 tok/s decode for Muse Spark Open at batch size 1 on InferenceX 8K/1K. We're also opening a community TPU capacity pool at @radixark, offering free compute to developers contributing to SGLang-TorchTPU. I hope this makes it easier for more open-source engineers to experiment with TPU kernels, model support, and serving optimizations. Huge thanks to everyone at Google, Meta, RadixArk, and the SGLang community who made this happen. It's genuinely exciting to see what collaborative open-source engineering can accomplish!

Hi everyone, today we’re releasing SGLang Omni v0.1.9! This release includes 153 merged PRs and welcomes 7 new contributors, who contributed 11 merged PRs. We’re officially releasing full Session Graph support for Interactive Superintelligence Systems. We also delivered day-zero support for FishAudio Drama 3, the Ming-3.0 family, and Sortformer, while expanding multimodal generation capabilities for Cosmos3 Nano and LLaDA2-Uni. 1. Infrastructure for interactive intelligence: A unified session graph supports native full-duplex models, turn-based half-duplex models, and streaming cascaded systems. Full-Duplex-Bench provides a common scoring pipeline across all three architectures, alongside improved evaluation of concurrency, playback continuity, and listening quality. sglang.io/blog/omni-interact… 2. Models and hardware: Added support for FishAudio Drama 3 and the Ming-3.0 family, along with native Sortformer support. Intel XPU support now extends to Ming-flash-omni-2.0, Nemotron VoiceChat, and Audar-TTS-V1 Turbo. On NPU, Qwen3-ASR now supports updatable encoder graphs. sgl-project.github.io/sglang… 3. Performance: Continued optimizations for Qwen3-ASR/TTS/Omni, Whisper, CosyVoice3, MOSS-TTS, and PersonaPlex cover prefill CUDA Graphs, reference encoding, sampling, caching, kernel fusion, and streaming vocoder batching. Whisper and Cohere Transcribe also gain native FP16 inference paths, further improving model computation and audio decoding efficiency. 4. Serving: Added memory limits for streaming backlogs and stage offloading for Music3. Voxt improves hotkey permission checks and OpenAI API format compatibility. Voxtral gains true streaming TTS, while Nemotron ASR improves streaming recognition. 5. Runtime: Batched message routing between stages in the same process and event-based waiting for idle stages reduce thread wakeups and message dispatch overhead. We also added streaming backlog memory limits, upgraded Music3 stage offloading to preserve stable memory addresses, and improved observability for CUDA Graph replay and fallback, request cleanup, startup, and cancellation. Together, these changes improve resource management and stability under high concurrency. 6. Voxt voice input: Fixed real-time transcription stalls during repeated recording sessions and improved timeout handling, cancellation, and failure fallback. Added a choice between the Responses and Chat Completions protocols, ensured reasoning and output settings are passed through correctly, and strengthened global hotkey permission checks and Escape cancellation while running in the background. If you like our work, please give us a star on GitHub. Thank you for your support! github.com/sgl-project/sglan…

大家好，今天我们发布 SGLang Omni v0.1.9。本版本共合入 153 个 PR，迎来 7 位新贡献者，首次贡献者共合入 11 个 PR。 我们正式发布对于 Interactive Super Intelligence System 的完整 Session Graph 支持。此外。我们在 Day 0 接入了 FishAudio Drama 3、Ming-3.0 系列及 Sortformer，并扩展 Cosmos3 Nano 与 LLaDA2-Uni 的多模态生成能力。 1.交互式智能基建：通过统一的 session graph 支持原生全双工，轮次式半双工和流式级联系统；评测上 Full-Duplex-Bench 统一三种交互式智能的评分流程，同步完善并发、播放连续性和听感质量评估。 sglang.io/blog/omni-interact… 2. 模型与硬件上新增 FishAudio Drama 3、Ming-3.0 系列和原生 Sortformer；Intel XPU 扩展至 Ming-flash-omni-2.0、Nemotron VoiceChat 和 Audar-TTS-V1 Turbo，NPU 新增 Qwen3-ASR 可更新 Encoder Graph 支持。 sgl-project.github.io/sglang… 3. 性能上持续优化 Qwen3-ASR／TTS／Omni、Whisper、CosyVoice3、MOSS-TTS 和 PersonaPlex，覆盖 Prefill CUDA Graph、reference encoding、采样、缓存、kernel fusion 与流式 vocoder batching；Whisper 和 Cohere Transcribe 新增原生 FP16 推理路径，进一步优化模型计算与音频解码效率。 4. Serving 上加入流式积压内存上限、Music3 stage offload，Voxt 完善快捷键权限检测和 OpenAI API 格式兼容；Voxtral 新增 true streaming TTS，Nemotron ASR 完善流式识别。 5. Runtime 通过同进程 stage 批量消息路由和空闲 stage 事件等待，减少线程唤醒与消息分发开销；加入流式积压内存上限，升级 Music3 稳定地址 stage offload，并完善 CUDA Graph replay／fallback 观测、请求清理、启动与取消路径，增强资源管理和高并发稳定性。 6. Voxt语音输入修复反复录音时的实时识别卡顿，完善超时、取消与故障回退；新增 Responses／Chat Completions 协议选择，正确传递推理与输出设置，并加强全局快捷键权限检测和后台 Escape 取消处理。 如果您也喜欢我们的工作，欢迎给我们点上宝贵的 github star，感谢！ github.com/sgl-project/sglan…

In SGLang Omni, we have been focusing on interactive superintelligent systems for half a year, and this is our first official release. The most interesting is that a full abstraction with duplex support, runtime powering up, interactive models, diffusion models, and even physical AIs. More to come!

感谢作战群里面 20 余位同学的全力合作，花了一个半月的时间，我们终于解决了包括 full duplex，turn based half duplex and cascaded system 三类完善的 interactive model system，都实现了非常好的流式体验，打断智力表现均在线。 很长一段时间，完成 interactive model system 以实现 interactive intelligence，是 SGLang Omni 项目的立项初衷。但是苦于长期受限于开源模型的能力水平和相关技术领域的缺乏，我们一直没有很好的进展。 直到 3 个月之前，有幸与一些研究者大佬建立了联系，我们便很快开始 move 了起来。经过了一段时间的密闭开发，终于可以很荣幸的给大家分享。more to come，欢迎大家读取我们的博客。 sglang.io/blog/omni-interact…

Huge thanks to the 20+ contributors in our project group for giving this their all. After a month and a half of work, we finally have complete interactive systems across all three architectures: full-duplex, turn-based half-duplex, and cascaded systems. All three deliver smooth streaming experiences, responsive interruption handling, and strong conversational intelligence. Building interactive model systems that enable interactive intelligence was the original motivation behind SGLang Omni. But for a long time, limitations in open-source models and gaps in the surrounding technology made it difficult for us to make meaningful progress. Then, three months ago, I was fortunate to connect with some outstanding researchers, and we quickly got moving. After an intense stretch of heads-down development, we’re finally excited to share what we’ve built. More to come. Check out our blog!