Building AI that learns by interacting with the world. Associate Professor @ MIT, leading the Scene Representation Group (scenerepresentations.org).
Cambridge, Massachusetts
- Tweets 1,080
- Following 344
- Followers 20,783
- Likes 3,517
Pinned Tweet
Thrilled to introduce Mirror, our first "learning-from-experience" project! Assume you had a video generative model that, no matter your situation, could always generate video of what it would look like to solve a task. How can you put that plan into action? Clearly, you need to learn some video-to-action mapping - but where do you get the data for that? It turns out you can collect that data in an amazingly sample-efficient way by *incrementally following the video planner*! (1/n)
If video models scale like LLMs, they'll soon generate a perfect video plan for any embodied task. Do we need anything else? Introducing Mirror, which learns to follow these plans just by practicing, and even crafts an iron pickaxe in Minecraft w/o rewards or action labels! (1/n)
11
40
316
22,291
I have talked much about the "bitter lesson" of computer vision, arguing that many of the intermediate tasks that vision has solved traditionally - 3D reconstruction, segmentation, tracking, etc - are not necessary to build AI that understands the world through vision... (1/n)
In my recent blog post, I argue that "vision" is only well-defined as part of perception-action loops, and that the conventional view of computer vision - mapping imagery to intermediate representations (3D, flow, segmentation...) is about to go away.
vincentsitzmann.com/blog/bit…
3
8
148
12,765
our recent "Mirror" result is another example. This is an agent that learns to play MineCraft without any inductive bias or any classic computer vision. There is no map, no 3D, no segmentation, no optical flow, no point tracking. The only inductive bias here is that we factorize the policy into a video planner and a video-to-action model - everything else is just learning. (2/n)
nitter.siddz.com/vincesitzmann/status/2…
Thrilled to introduce Mirror, our first "learning-from-experience" project! Assume you had a video generative model that, no matter your situation, could always generate video of what it would look like to solve a task. How can you put that plan into action? Clearly, you need to learn some video-to-action mapping - but where do you get the data for that? It turns out you can collect that data in an amazingly sample-efficient way by *incrementally following the video planner*! (1/n)
1
13
2,674
I am confident that this same algorithm will work in the real world - all that is needed is to scale up video models! And, while expensive, we *know that this will work*, b/c it's just generative modeling. Of course, other formidable challenges still remain, but they are really about *learning* - not about representations in the classic computer vision sense. (n/n)
2
21
2,139
This is a really impressive result: only with self-play, without any hand-crafted rewards or any expert actions, Mirror achieves MineCraft milestones that outperform Dreamer v3!! Notably, Dreamer v3 used simplified mining & crafting actions, since random exploration is too unlikely to ever mine a block! This is exactly the problem that Mirror manages to solve: our agent tirelessly works on the milestone of mining a block until it figures it out :)
In Minecraft survival, Mirror reaches 9x the iron pickaxe success rate of DreamerV3 (5x w/ DreamerV3's own eval protocol) without using any rewards! With an oracle video planner, it even finds diamonds, unlike all online baselines with matched environment interaction time. (6/10)
3
6
28
2,595
Another question that I think is worth addressing: why do we lean so heavily on MineCraft and not use, e.g., real robots? The answer is two-fold. First, we are compute limited: It is expensive to train a video generative model. On real-world data, this is doubly true: Minecraft is an easy enough distribution such that we can iterate on the video planner even with our compute budget.
However, second, MineCraft is actually very, very difficult: it has tasks with extremely long time horizons and extensive tool use.
Hence, MineCraft is sort of the sweet spot for now :) That being said, as compute gets cheaper and video models get bigger, I'm sure it will become more tractable to do real robots soon as well.
1
2
509
Another explanation that Xingjian came up with that I liked: while both the agent and the training cost FLOPS, they are not actually fungible. I.e., there isn't actually an effective way of turning the FLOPS that your Claude Code subscription gives you into more training flops for whatever model you are cooking on your four 5090 GPUs ;) So, while in theory, for small models, it might be better to spend the FLOPs on training compute, in practice, you should spend a few bucks on tokens for OPAL :D
Say you're a PhD student, broke on compute, with a half-working new model and an advisor meeting tomorrow. Or you run a factory line, have a few thousand photos of faulty parts, and want an open multimodal LLM to spot them.
How would you train the model? All you can do is pick the most standard recipe, or one from a recent paper, and live with the chance that it doesn't suit your setting.
A reactive optimizer takes that burden off you: it watches the training, analyzes the signals, and makes optimization decisions as training progresses.
2/9
2
36
3,947
Vincent Sitzmann retweeted
We are in an era where you can agentify almost anything, and often you get a more powerful system.
This work provides a neat example of that approach applied to optimizers:
1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
8
15
244
15,607
Vincent Sitzmann retweeted
Hot take: Most autoresearch today is just glorified overfitting.
Current autoresearch pipelines are essentially outer-loop searches for recipes that work in ONE specific setting. Try transferring a modded-nanogpt recipe to a different language or dataset. How much of that "optimal" recipe still holds up?
We need to move beyond static recipes toward inner-loop autoresearch: optimizers, architectures, hyperparameters, etc., continuously adapting DURING training based on the training trajectory itself.
Check out OPAL that enables this! Cool work with @SimulatedAnneal, Jia-Hao Liang, and @vincesitzmann.
1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
6
15
209
9,524
Excited to introduce reactive optimizers: We let an agent reactively adapt the optimizer code during the training run! The agent acts like a team of scientists constantly choosing the best algorithm, learning rate, momentum, etc, based on its own analysis of step-to-step training dynamics! The results are really exciting. Besides out-performing standard "best-practice" recipes across everything we threw at it, the agent gains a few "superpowers" that no human could have designed into the optimizer... (1/n)
1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
14
31
332
18,672
Opal also designs highly sophisticated learning-rate schedules, something that we can usually only afford for the highest-impact, super-late training runs like DeepSeek - now anyone can get the same (or more?) sophistication :) (4/n)
2
7
1,051
Also, I'm sure other folks have come up with this as well! But we were surprised that we couldn't find a prior record of this concept, so @SimulatedAnneal, @jayden_teoh_, and Jia-Hao Liang took it on. I think it's valuable to have a clear package and a writeup to reference, as well as broadly disseminating the idea! Excited to discuss with everyone :)
1
11
643
Thrilled to introduce Mirror, our first "learning-from-experience" project! Assume you had a video generative model that, no matter your situation, could always generate video of what it would look like to solve a task. How can you put that plan into action? Clearly, you need to learn some video-to-action mapping - but where do you get the data for that? It turns out you can collect that data in an amazingly sample-efficient way by *incrementally following the video planner*! (1/n)
If video models scale like LLMs, they'll soon generate a perfect video plan for any embodied task. Do we need anything else? Introducing Mirror, which learns to follow these plans just by practicing, and even crafts an iron pickaxe in Minecraft w/o rewards or action labels! (1/n)
11
40
316
22,291
There are also first glimpses of emergent behavior: The video model remembers whether a particular strategy failed before and biases future sampling to successful strategies!
We also find early signs of emergent behavior in the video planner! Mirror can learn from its own history in-context, w/o any finetuning. Dropped in a lava lake, it reaches a cumulative escape rate of 76% over 30 attempts, vs. 46% when context is reset after every try. (8/10)
2
6
926
Paper: arxiv.org/abs/2610.24177
Website: wilcoxsonm21.github.io/mirro…
1
11
648
The old wisdom strikes again: go where the data is. Turns out there is a lot of video, so training a video planner is actually the *easy* path, albeit expensive. The truth of the matter is that the only way to get generality today is by training on diverse data at scale. Hence, any policy that cannot - in one way or another - scale with data is hopeless.
The interesting corollary that Hyunwoo points out here is that if you have expert demonstrations, even if they have action labels, training a video planning (like our Large Video Planner, Vera, and now mirror) is still preferable over training a world model (like Dreamer v4) b/c overall, action labels are scarce.
nitter.siddz.com/RyuHyunwoooo/status/21…
First: with expert data, do you need an action-conditioned world model? Not at all. Video planning is far better. In offline Minecraft, our video planner + IDM significantly outperforms Dreamer 4, the SOTA world-model agent: ~2x success on iron pickaxes, ~8x on diamonds. (2/15)
4
5
107
7,595
Vincent Sitzmann retweeted
Video models are getting really good at imagining how to solve a task. With Mirror, an agent learns to follow these videos by practicing, so as video models improve, agents should too, without extra action data. Fun working with @ericmchen1, @RyuHyunwoooo and @vincesitzmann!
If video models scale like LLMs, they'll soon generate a perfect video plan for any embodied task. Do we need anything else? Introducing Mirror, which learns to follow these plans just by practicing, and even crafts an iron pickaxe in Minecraft w/o rewards or action labels! (1/n)
4
12
939