(1/9)Suppose you're training an LLM on web data with limited math data. Should you place all your math data at the end of training, or mix it with web data earlier? The answer flips with the learning rate (LR) schedule! At fixed compute and mid-training (MT) token budgets, earlier mixing gives lower MT loss when LR decays throughout training. With constant LR, placeing all MT data for the end works better. We then vary when LR decay begins. The best tested strategy: keep LR high after MT data enters, then decay near the end. See our new paper: "Data Schedules and Learning Rate Schedules Should Be Designed Together" !

🚀 Very excited to introduce TTR-Compressor -- A test-time regression framework for sequence compression! I particularly like this work for its fundamentally different perspective on sequence compression. Unlike popular pooling and projection, TTR-Compressor formulates sequence compression as a test-time regression problem. Instead of directly aggregating tokens, we fit regression keys derived from the input tokens to learnable targets, using the fitted coefficients as the compressed representation. This leads to a bi-level optimization with an exact closed-form solution to the inner test-time regression. The closed-form solution captures second-order inter-token correlations, allowing TTR-Compressor to reduce redundancy and better preserve distinctive information, going beyond conventional first-order aggregation in pooling and projection. Strong gains in two applications: ✨ Block-wise sparse attention: +12.24% in RULER retrieval accuracy over attention-based compression. ✨ Visual token merging: Comparable multimodal performance with 55.6% fewer visual tokens than standard MLP merging. 🤔 Takeaway: Sequence compression can be viewed as input-adaptive regression rather than static aggregation. This provides new insights to building more efficient language and multimodal models. 📄 Paper: arxiv.org/abs/2610.31574 🌐 Project: spherelab.ai/ttr-compressor/ 💻 Code: github.com/Sphere-AI-Lab/ttr…

Can LLMs manage their own context natively without any external harness? We are teaching LLMs the ability of Native Context Editing. Our method achieves considerable performance gain on popular long-horizon agentic benchmarks (Terminal-Bench, SWE-bench) while reducing token usage significantly. 🤔 This can be viewed as a form of verbalized context management, which leverages LLMs' inherent capabilities to optimize context. Our prior work, Verbalized Machine Learning (arxiv.org/abs/2406.04344), similarly explores context optimization to enhance downstream task performance. Project: dengyl20.github.io/native-co… Paper: arxiv.org/pdf/2610.25550 Code: github.com/dengyl20/native-c…

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains? We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer. In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans. 📝 xingjianbai.com/reactive-opt… 💻 Try it on your task: github.com/xingjian-bai/opal 🧵

Video-Index 1.1 comes! Happy to see agentic video understanding reach higher accuracy with far less context. Letting the model decide what to watch beats feeding it 512 fixed frames. What is the next vision task that is still hard for agents? 🤔

Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do. SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%.

🔁 𝐖𝐡𝐚𝐭 𝐢𝐟 𝐥𝐨𝐨𝐩𝐞𝐝 𝐋𝐌𝐬 𝐜𝐨𝐮𝐥𝐝 𝐮𝐬𝐞 𝟑× 𝐟𝐞𝐰𝐞𝐫 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫𝐬 𝐚𝐧𝐝 𝐚 𝟑× 𝐬𝐦𝐚𝐥𝐥𝐞𝐫 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 𝐚𝐧𝐝 𝐬𝐭𝐢𝐥𝐥 𝐛𝐞 𝐜𝐨𝐦𝐩𝐚𝐫𝐚𝐛𝐥𝐞 𝐭𝐨 𝐚 𝐬𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫? Introducing 𝐑𝐞𝐭𝐡𝐢𝐧𝐤𝐢𝐧𝐠 𝐚𝐭 𝐅𝐢𝐱𝐞𝐝 𝐏𝐨𝐢𝐧𝐭𝐬. We argue that looped LMs don't have to pay for every loop. They can scale by adding FLOPs at constant memory. The key is the fixed point of the loop. We introduce: – 𝐅𝐢𝐱𝐞𝐝-𝐏𝐨𝐢𝐧𝐭 𝐒𝐡𝐨𝐫𝐭𝐜𝐮𝐭𝐬: what fixed points buy a looped model in training (pre-training, post-training) and inference (prefill, decoding) – A 𝐥𝐞𝐚𝐫𝐧𝐞𝐝 𝐝𝐞𝐩𝐭𝐡 𝐩𝐫𝐢𝐨𝐫 and 𝐨𝐫𝐭𝐡𝐨𝐠𝐨𝐧𝐚𝐥 𝐢𝐧𝐣𝐞𝐜𝐭𝐢𝐨𝐧: two ways to shape those fixed points 🧵 Long thread ahead (0/n).

GPT-5.5 + a geometry-grounded agent can give GPT-6 Astra a run for its money in single-image 3D reconstruction 👀 Our HARMONY combines hierarchical reasoning with geometric refinement: competitive visual quality and better geometric alignment in our evaluation. Check it out 👇

We’re teaching a new course at Penn: CIS 6280 · World Models 🌍 cis.upenn.edu/~cis6280/ Every day, LLMs amaze me with one more thing they can solve. But I’ve always felt there’s more to learn than what humans have written down—from observing the world, interacting with it, and experiencing what happens next. To me, that’s what world models are about—and why I see them as one of AI’s next big bets. However, I struggled to find a systematic course on them—so we are building one! Topics will include: representation learning, generative models, simulation, model-based RL, video and 3D generation, world models for robotics, reasoning, and code-based world models. Hands-on work will include: - Building an environment - Training a world model - Learning a policy and a final research project with leaderboard. We’re already six lectures in. Slides, demos, and readings are public, and we’ll keep adding materials throughout the fall. Big thanks to our TA team— @TongMutianTMT @hagsaeng_bag @EnxinSong @KeelyAi04 —for helping bring this course to life!

It’s been fun to see how research interacts organically with production this summer at @togethercompute

Images, video, audio, actions — generative modeling has converged on one recipe: compress into continuous latents, generate in latent space, decode. Language is the lone exception: still generated token by token, as a long discrete stream. Latent Thought Flows: We compress 256 text tokens into 8 continuous latents, generate them with a one-step flow model, and read them out as text using an autoregressive decoder. Result: a better inference-compute vs. generation-quality Pareto frontier than a tuned autoregressive baseline — thanks to compression and one-step generation. Co-led w/ @ZhengyangGeng 🧵

DeepSeek V4 (huggingface.co/collections/d…): their long-context recipe shares a lot with directions the video understanding community has been exploring for years. I had doubted these aggressive compression strategies would carry over to language tokens; they do. MovieChat's (arxiv.org/abs/2307.16449v4) long/short-term memory ↔ V4's compressed and local-window KV. AuroraCap's (arxiv.org/abs/2410.03051) token merging ↔ V4's KV compression. We also tried bringing NSA into video with VideoNSA (arxiv.org/abs/2510.02295). Language tokens may not be as dense as we assumed. Keeping all of them isn't strictly necessary. Congrats to DeepSeek on the release. please deliver more!

I’m excited to present VideoNSA at #ICLR2026 tomorrow (Pavilion 4, Poster #3307, 3:15–5:45 PM). Feel free to stop by and chat about sparse attention and long-video understanding!