Pinned Tweet
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
2
23
42
4,878
Wonjun Lee retweeted
👋 Looking forward to attending #EMNLP2026 where I will be giving an industry keynote on the hard, imminent AI challenges of trust, world discovery, and memory, plus an important, timely discussion on re-strengthening industry-academia collaborations!
We're also hiring at all levels: phd, postdoc, faculty! Feel free to ping if you want to meet up to discuss research, hiring, life 🙂
We're looking forward to the Industry Track at #EMNLP2026! Two keynotes:
• Mohit Bansal (UNC Chapel Hill): Agentic Challenges & Industry–academia Collaboration (Sun, 16:30)
• Verena Rieser (Google DeepMind): Constitutional Alignment (Mon, 9:30)
2026.emnlp.org/program/indus…
5
21
88
3,753
Wonjun Lee retweeted
It was a pleasure visiting ETH Zürich for their CS Distinguished Colloquium Series -- amazing scenic views from the hilltop hotel + everywhere 🙂 -- also had various engaging discussions about AI safety/trust and its future with the faculty, students/postdocs, and scientists there -- thanks again for the kind invitation + hospitality Mrinmaya Sachan, Thomas Hofmann, Marc Pollefeys et al. (and for these nice photos incl. from our dinner at the world's oldest "certified" continuously running vegetarian restaurant 😯)
Also had a fun short visit at Google Zurich thanks to Srini Narayanan, Ivan Vulić, Jonas Pfeiffer, Emanuele Bugliarello!
--> Now off to TU Darmstadt for the 'The Future of Science' pre-EMNLP international symposium --> And after that, you can find me at EMNLP in Budapest where I will be delivering an Industry Keynote on Oct25 afternoon. Looking forward to seeing everyone!
9
13
79
3,956
Dormant poisoning goes beyond unsafe text. We show that benign fine-tuning can activate multiple hidden harmful behaviors in language and multimodal agents. LookAhead uses the model’s original safe behavior as a reference to identify and prevent risky changes during fine-tuning. We reduce poison activation from 93.9% → 3.5% in language agents and 64.5% → 0% in multimodal agents, while preserving downstream utility.
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
5
10
783
We show that LLM agents can be poisoned with dormant dangerous behaviors that activate after post-training (even on safe, clean data)!
This type of attack is particularly dangerous because:
(a) finetuning LLM checkpoints for your own agentic tasks is increasingly common
(b) the range of implantable behavior is broad + realistic (e.g. a computer use agent can be poisoned to treat phishing emails as legitimate)
We also propose an elegant, easy-to-use defense against these dormant poisoning attacks that can be applied at training time to prevent these behaviors from activating 🧵👇
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
1
7
10
586
Wonjun Lee retweeted
🚨 We observe that seemingly safe agent can take harmful actions (e.g., privacy leakage) after benign fine-tuning. Existing defense is difficult to apply as it preserves safe behavior at release to remain hidden.
🛡️ To defend against this threat, we propose LookAhead Defense, which uses no external models or datasets to suppress dormant poison activation while preventing over-refusal by selectively penalizing safety-degrading updates.
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
5
11
780
Wonjun Lee retweeted
A model can appear safe at release, yet become unsafe after completely benign fine-tuning.🚨
In our new work, we show how dormant poisoning can trigger harmful actions in AI agents 🤖, and introduce LookAhead Defense, which anticipates and penalizes safety-degrading fine-tuning updates using a self-generated safety buffer.
LookAhead Defense reduces poison activation from 93.9% → 3.5% in language agents and 64.5% → 0% in multimodal agents, while preserving utility.
Excited to share this work! 🛡️
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
7
20
1,103
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
2
23
42
4,878
Does LookAhead generalize beyond dormant poisoning? 🛡️
We also evaluate LookAhead against other poisoning methods, including Sleeper and Benign DPO, where harmful behaviors are already present at release.
🔥 Sleeper: ASR drops from 74.5% to 0.0% on Qwen3-4B.
🔥 Benign DPO: ASR drops from 69.5% to 10.0% on Qwen2.5-3B.
These results show that LookAhead remains effective beyond dormant poisoning, while maintaining a favorable safety–utility trade-off.
1
2
104
Awesome work with @vaidehi_patil_ , @codezakh , @zhan1624 , @hyunji_amy_lee , Suhyun Kim, and @mohitban47
📄 Paper: arxiv.org/abs/2610.24379
🌐 Project page: wonjuun.github.io/LookAhead/
🧑💻 Code: github.com/wonjuun/LookAhead
1
5
259
🚨 Excited to share WEDGE: From Failures to Learnable Worlds, our new work on policy-conditioned environment generation for navigation agents!
We found that even on the same navigation task, agents tend to fail under different environmental conditions. Yet most environment generators do not consider what each agent actually needs to improve.
🤔 This led us to a question: Can we use an agent's own failures to guide the generation of training environments?
➡️ We introduce WEDGE, a framework where an agent's failures shape the worlds it trains in next. WEDGE identifies the conditions under which each agent struggles and generates new environments that target those weaknesses while remaining learnable. After each round of adaptation, it re-diagnoses the agent to guide the next round of environment generation.
Thread🧵👇
4
21
57
4,811
Wonjun Lee retweeted
🚨 As agents become more capable, new safety and security risks continue to emerge, as highlighted by recent reports of AI agents taking unauthorized actions or leaking sensitive data. Therefore, we need runtime guardrails that can reliably enforce safety policies before agent actions are executed, while continually adapting to new risks.
Existing guardrails usually trade off adaptability and reliability: LLM-based guardrails can adapt with memory, but their decisions may be inconsistent. On the other hand, formal guardrails provide deterministic enforcement but are typically fixed and unable to incorporate new risks that appear after deployment.
🚀 We introduce VEPA, a lifelong formal guardrail for agents. It represents safety and security policies as executable symbolic rules that evolve from deployment feedback. These policies enable deterministic enforcement of agent actions and can be formally verified, avoiding the inconsistency and ambiguity that can occur with natural-language policies.
🧵⬇️
7
16
37
1,405
Wonjun Lee retweeted
🚨 We are hiring Tenure-Track and Full Professors in important and timely areas such as AI Safety**, Neuroengineering, as part of exciting campus-level strategic cluster hires with CS, and with several positions in the clusters. The Full Professor position in 'Safeguarding AI for Health' is jointly between our strong BioStats department and CS -- please spread the word and feel free to apply & reach out for any questions!
**broad emphasis incl. but not limited to machine learning theory, AI systems, interpretability, robustness, verification, human-AI interaction.
We are hiring at @unccs 🎉
▶️ Two TT Asst. Professors: 1 in Safeguarding AI in Health, 1 in Neuroengineering.
▶️ Full Professor jointly with Biostats.
Part of campus cluster hires (review begins Dec 1).
Plus, Teaching Faculty (review begins Nov 1).
cs.unc.edu/about/faculty-hir…
2
16
47
3,455
Wonjun Lee retweeted
Can agents learn useful skills from frozen logs that people and agents already leave behind?
🚨Introducing TeleTune: We turn noisy offline telemetry into agent skills w/o access to a live environment during training!
+6.7% on WorkArena & +7.7% on Mind2Web 👉 up to 75x fewer tokens used for optimization. 🧵
Activity logs like user telemetry capture how people use software, but often (1) lack explicit task goals, (2) cannot be replayed to test each skill update, and (3) can be a noisy mix of multiple tasks.
We address these jointly in TeleTune, where we propose skill-guided progress, an offline metric that uses action-prediction errors to improve agent skills. Not only does optimizing it improve downstream performance, but it also saves token usage by 5 to 75x compared to validating skill updates in live environments!
Key takeaways:
1️⃣ TeleTune achieves 77.1% average success on WorkArena and 80.6% on Online-Mind2Web, outperforming the strongest baselines by +6.7% and +7.7% absolute
2️⃣ Optimizing offline skill-guided progress transfers to improvements in downstream task success
3️⃣ Offline optimization uses 5 to 75x fewer tokens than validating the skill edits in a live environment
2
25
84
3,885
Excited to share WOVEN! 🚀
The goal isn't just to teach models what the world looks like. It's to teach them how the world changes.
Our key finding: Visual transition reasoning is a shared, teachable primitive that transfers across diverse downstream tasks.
And the key to effective transfer? Teach the right reasoning operations, rather than simply matching scenes, actions, or domains.
🚨 What makes good world-model training for MLLMs that actually transfers across tasks?
🧶 Excited to share our new work, WOVEN!
🧩 Spatial, physical, temporal, and embodied reasoning all require understanding how the world changes.
🔁 We view world modeling through state transitions p(s′ | s, a), and study visual transition reasoning: inferring what is unobserved in (s, a, s′).
🤔 Can visual transition reasoning be learned as a shared primitive that benefits multiple downstream tasks?
🤔 If so, how should we train it for better downstream gains?
Major takeaways:
➡️ Visual transition reasoning is a shared, teachable primitive that transfers widely downstream: ~2K-example subsets collectively improve 22 of 26 downstream benchmarks by up to 27.3 points.
🧪 A data-centric training recipe: to teach this shared primitive for downstream tasks, select supervision by the reasoning operation it teaches, rather than scene/action/domain similarity.
Thread🧵👇
1
12
24
1,283
Excited to share our #NeurIPS2026 paper "Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge"! 🎉
We show that LLMs contain latent safety signals in their first-token probability distributions and these signals can transfer across different models.
🎉 Excited to share our #NeurIPS2026 paper, "Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge"!
Latent Safety Signals in LLMs: Can we detect a harmful query before the model generates even one token? We show that the answer lies in the model's dark knowledge, i.e., the first-token probability distribution contains safety signals that transfer across different LLMs.
We use this insight to build LADE (Latent Safety Signals for Defense), a model-agnostic defense against jailbreak attacks that requires no access to internal hidden states. 🛡️
Key takeaways:
1️⃣ We discover latent safety signals in the first-token probability distribution: contrasting harmful and benign queries reveals safety-discriminative tokens that transfer consistently across safety-aligned LLMs, despite different architectures, tokenizers, and refusal styles.
2️⃣ We introduce LADE 🔍: a lightweight framework that identifies harmful queries before generation begins by applying a simple kNN classifier to latent safety signals in the first-token probability distribution.
3️⃣ LADE also transfers across models 🔄. Across all reference–target LLM pairs, it achieves 96.6% average harmful-vs-benign classification accuracy. It also performs well against several jailbreak attacks while keeping over-refusal low.
15
31
1,823
Wonjun Lee retweeted
🎉 Excited to share our #NeurIPS2026 paper, "Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge"!
Latent Safety Signals in LLMs: Can we detect a harmful query before the model generates even one token? We show that the answer lies in the model's dark knowledge, i.e., the first-token probability distribution contains safety signals that transfer across different LLMs.
We use this insight to build LADE (Latent Safety Signals for Defense), a model-agnostic defense against jailbreak attacks that requires no access to internal hidden states. 🛡️
Key takeaways:
1️⃣ We discover latent safety signals in the first-token probability distribution: contrasting harmful and benign queries reveals safety-discriminative tokens that transfer consistently across safety-aligned LLMs, despite different architectures, tokenizers, and refusal styles.
2️⃣ We introduce LADE 🔍: a lightweight framework that identifies harmful queries before generation begins by applying a simple kNN classifier to latent safety signals in the first-token probability distribution.
3️⃣ LADE also transfers across models 🔄. Across all reference–target LLM pairs, it achieves 96.6% average harmful-vs-benign classification accuracy. It also performs well against several jailbreak attacks while keeping over-refusal low.
2
15
37
3,079