What Happens Inside an AI Model When a Problem Becomes Too Difficult? New research reveals a fascinating pattern: the harder the problem, the sparser the AI's internal representations become. We know that large language models perform remarkably well on familiar questions but often struggle with complex reasoning, conflicting information, and unfamiliar problems. But what actually changes inside a model when the difficulty increases? A new research paper, Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs, investigates exactly this question. And the findings could change how we understand AI reasoning, model robustness, and the way we design more intelligent inference systems. 1. The surprising discovery: harder problems produce sparser representations Researchers investigated how the internal activations of LLMs change when they encounter increasingly challenging inputs. They examined models from the Qwen and Llama families across different reasoning tasks and model sizes. The central finding was remarkably consistent. As task difficulty increases, the model's final hidden-state representation becomes more sparse. In simpler terms, more of the representation's activation energy becomes concentrated in a smaller subset of dimensions. The researchers observed this pattern across four different types of difficulty: Complex reasoning: Harder mathematical problems produce sparser representations. More answer choices: Adding plausible incorrect answers increases representational sparsity. Conflicting information: When provided context contradicts the model's learned knowledge, activations become more concentrated. Longer contexts: Increasing the amount of information the model must process produces similar effects. Importantly, this is a statistical pattern across groups of examples, not a guaranteed rule for every individual input. The researchers summarize their discovery with a simple principle: The farther the distribution shift, the sparser the representation. 2. Why does this happen? One of the most interesting parts of the research is its explanation of how these patterns develop during training. The researchers studied a smaller Transformer trained from scratch in a controlled environment. They discovered two distinct phases of representation learning. Phase 1: Feature selection During the early stages of learning, the model increasingly concentrates its activations in a limited set of dimensions. It appears to emphasize features that are useful for learning the task. Representations become more concentrated while the model is still learning the underlying structure. Phase 2: Feature consolidation As training progresses and the model becomes more familiar with the data, the opposite pattern emerges. Representations become more distributed across multiple dimensions. The model develops broader internal representations for patterns it has successfully learned. This leads to an intriguing interpretation. Familiar problems can activate rich, distributed representations, while unfamiliar problems may rely on narrower activation patterns. In other words, representation density may reflect how well a model has internalized particular patterns during training. This is especially interesting because the model was not explicitly trained to produce the observed sparsity–difficulty relationship. The pattern emerged through learning. Final thoughts The most interesting implication of this research is not simply that difficult questions produce sparser activations. It is that the geometry of a model's internal representations may carry meaningful information about how familiar or challenging an input is. That changes how we might investigate AI reasoning. Rather than studying only the final answers generated by language models, we can also examine the internal structures that emerge while they process information. Understanding these structures could help researchers build systems that adapt more intelligently to unfamiliar problems. Perhaps one of the next breakthroughs in AI will come not only from making models larger, but from understanding their internal representations well enough to guide how they reason. The future of AI may depend as much on understanding how models represent difficulty as on improving how they answer questions. 📄 Research paper: Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs arxiv.org/abs/2603.03415 💻 Code: github.com/MingyuJ666/sparsi… #ArtificialIntelligence #LLM #AIResearch #MachineLearning #DeepLearning

We’re building a scientific computer-use benchmark for AI agents and welcome contributions from researchers across all scientific disciplines!

Excited to see Jev and RLCD bringing calibration into the spotlight. This connects closely to a question we’ve been exploring since last year: how can we build models whose confidence we can actually trust? In two EMNLP 2025 papers, we evaluated calibration in reasoning models and VLMs. We found that stronger capabilities don’t automatically translate into more reliable confidence. Task and modality matter. Reasoning models: aclanthology.org/2025.emnlp-… Vision-language models: aclanthology.org/2025.emnlp-… In our ACL 2026 work, we took this into agentic RL: explicitly optimizing calibration alongside task performance in tool-using agents. The Confidence Dichotomy: aclanthology.org/2026.acl-lo… For me, the excitement around Jev/RLCD highlights why this direction matters. As confidence starts driving real actions, calibration becomes a core capability to train and evaluate. Really looking forward to seeing more work on calibration. Excited to keep pushing this direction forward!

Excited to share what we built! Try it!

What if LLMs and agents could learn continuously from every interaction without retraining? Our new review on Test-Time Learning explores how deployed models can turn experience into adaptive state through a persistent write–read loop, improving future behavior over time. papers.ssrn.com/sol3/papers.…

When I first started doing research, I worked on a few projects around LLM confidence/reliability. The latter two were recently accepted to EMNLP, so I thought I’d wrap up this little line of work. 🔗 arxiv.org/abs/2505.24778 — Can LLMs reliably express confidence through linguistic markers? We found them surprisingly unstable under distribution shift. 🔗 arxiv.org/abs/2508.08992 — Can LLMs correctly interpret linguistic confidence and use it in decision making? Again, lots of instability. 🔗 arxiv.org/abs/2601.11004 — As factual tasks increasingly rely on retrieval, we studied how retrieval noise affects confidence calibration and how to improve it. Different settings, but one question: can models reliably express and perceive uncertainty, and reason and act accordingly? I think this is fundamental to reliable agents. Huge thanks to Prof. Song @yqsong and everyone I worked with! Confidence is truly a mysterious topic, with many nights spent staring at bizarre results 😂 Couldn’t have made it through all of this alone. This little line of work can finally take a pause :)

🚀 Towards an Agentic Era of AI 1/ We are excited to share our new review, “Towards an Agentic Era of AI,” which asks a fundamental question: How should we systematically design the next generation of AI agents?

Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus Introduces a benchmark that varies the language of supporting evidence while keeping English questions and answers. 📝 arxiv.org/abs/2606.15345

Excited to share our new paper 🧵MIXSD: Mixed Contextual Self-Distillation for Knowledge Injection Supervised fine-tuning is the common way to teach LLMs new knowledge, but it often catastrophically forgets existing capabilities. We introduce MixSD: a simple, external-teacher-free method to inject knowledge with far less forgetting. 📄arxiv.org/abs/2605.16865 Why does SFT forget? Targets written by humans or external systems diverge from the model's own autoregressive distribution, forcing the optimizer to imitate low-probability tokens. That's what drags pretrained capabilities down. MixSD: We hypothesize that keeping supervision close to the model's own distribution is key to avoiding forgetting. Instead of training on fixed, externally authored targets, at every token we mix between two conditionals of the base model itself: an expert conditional that sees the injected fact in context, and a naive conditional reflecting the model's prior. The result is supervision the model already finds high-probability, while still carrying the new factual signal. A Bernoulli rate λ controls the balance between memorization and retention. Findings: SFT only retains as little as 1% of held-out capability. MixSD retains far more, up to ~100% on larger models, with near-perfect training accuracy. It also beats on-policy self-distillation at a fraction of the compute, and holds across Qwen3 1.7B, 4B, 8B and Llama-3.2.

Does mechanistic interpretability really find the circuit? Our new paper, "All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMs," (Accepted by ICML 2026) suggests the answer may be: not always. A common implicit assumption in mechanistic interpretability is that a model's behavior is explained by the circuit — a sparse, canonical, almost-unique mechanism. Instead, for the same LLM task, we find multiple circuits/sheaves that are: ✅ faithful ✅ sparse ✅ structurally different ✅ low-overlap This means a discovered circuit may not be the unique mechanism behind a behavior, but one realization among many possible mechanisms. We call for rethinking how circuit/sheaf discovery results should be interpreted and evaluated. Huge thanks to my amazing collaborators: @frankniujc, @YutongYin774638, and @zhaoran_wang Paper: arxiv.org/abs/2605.12671 #MechanisticInterpretability #LLM #AI #MachineLearning

Just recognized as a gold reviewer at #ICML2026! However, I will not attend ICML this year. If doable, I really would like to transfer this free registration to a student from an underrepresented group. Is that possible? @icmlconf

I'm in Rio for #ICLR2026 this week! 🇧🇷 Excited to present our work in the very first poster session: ADAPT -- a rethink of data curation for LLM training. 🕥 Thu, Apr 23, 2026 • 10:30 AM–1:00 PM (-03) 📍 Pavilion 3, P3-#519 A short thread on what the paper is about 🧵👇 (1/6)

Dual-View Training for Instruction-Following Information Retrieval @SteveZeng7 et al. at Snowflake introduce a dual-view data synthesis strategy using polarity reversal to train retrievers that genuinely follow user instructions. 📝 arxiv.org/abs/2604.18845

Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers @SteveZeng7 et al. present benchmarks for evaluating retrievers on mixed-language queries. 📝 arxiv.org/abs/2604.17632 👨🏽‍💻 github.com/paddler2022/Code-…

Researchers found that when language models face harder questions, their internal brain activity literally shrinks into fewer paths. Language models actually compress their internal thinking when they get confused, and we can use that to help them. Standard AI models usually spread their thinking across many artificial neurons when they confidently recognize familiar information. The team discovered that if you confuse a model with tricky math or conflicting facts, this broad activation collapses into a highly concentrated signal in its final processing layer. This shrinking happens because the system drops its robust distributed memory and forces the computation into a tiny specialized space to survive the unfamiliar challenge. The big deal is that we usually have no idea when a language model is actually struggling with a weird prompt until it gives a wrong answer. This paper proves that the model actually broadcasts its confusion internally by abandoning its wide neural networks and falling back on a very tiny cluster of active neurons. Because we can measure this exact shrinking effect as a raw number, we do not have to guess if a question is too hard for the AI. We can just read that internal signal and automatically provide the system with the perfectly scaled stepping stones it needs to solve the problem. ---- Paper Link – arxiv. org/abs/2603.03415 Paper Title: "Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs"

In early Jan, I decided to write some blogs this year. Here comes the first one: qcznlp.github.io/blog/2026/i…. I played with PyLate on instruction-following IR tasks and got some interesting results. Would really love to get feedbacks, especially from @antoine_chaffin if possible!

🚀 Introducing NAACL (Noise-AwAre Confidence CaLibration): a scalable, self-bootstrapping framework that trains LLMs to provide reliable and grounded verbal confidence in RAG scenarios.  📄 Paper: arxiv.org/abs/2601.11004 💻 Github: github.com/HKUST-KnowComp/NA… #LLM #RAG #Alignment

This paper is accepted at #EACL2026 main conference! Read it and let me know how you think!