Sr. Director, Robotics @Nvidia, Isaac - building across full robotics stack enabling ecosystem - Past: Postdoc @Stanford, PhD @EPFL, Robotics Lead @Postmates-X
San Francisco, CA
- Tweets 142
- Following 295
- Followers 221
- Likes 259
Exciting to see what happens when AI agents meet the physical AI development loop! 🚀
Our team has used OpenAI Astra with NVIDIA Omniverse and Isaac libraries and tools for 3D Reconstruction, OVRTX, USD Content Agents, and Isaac Sim to turn real-world captures into editable, interactive simulation scenes - connecting reconstruction all the way to interaction testing.
Check out the tech blog and explore what’s possible!
What happens when you put Frontier AI agents to work with NVIDIA Omniverse libraries?
Developers are using @OpenAI GPT-6 Astra to turn existing assets, captured spaces, and simple prompts into interactive, simulation-ready 3D experiences.
🔗 nvda.ws/4AE03r1
Here’s what they built. 👇
3
190
CoRL 2026 paper decisions are out — check them out on OpenReview!
As General Chair, I'm delighted to welcome you to #CoRL2026 in Austin, TX, this November. We expect tickets to sell out quickly, so please register early!
corl.org/attending/registrat…
6
12
175
17,569
Soha Pouya retweeted
Exciting day for NVIDIA and @huggingface.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. They allow every developer, startup, university, industry and country to build with, customize and benefit from AI.
Thank you @ClementDelangue for coming to me.
NVIDIA is going to be a great home for Hugging Face, its community and the future of open models. 🤗
blogs.nvidia.com/blog/nvidia…
1,710
3,296
27,320
6,179,707
Soha Pouya retweeted
Super excited about Hydra-0 from @Hongyu_Lii and team! The key idea is to use flow as a shared visual interface across embodiments/objects for controllable video generation, allowing a single generalist world model to learn from human, handheld-gripper, and robot interaction data.
My favorite result is the video below: start from a real video of a human doing the task (left), extract the desired object flow, and condition the model on that flow (right). The model then hallucinates a plausible robot motion that could produce the same object motion. Very cool glimpse of how a generalist world model can bridge human demonstrations and robot control.
What if robots, humans, and video models could all describe action in the same language? Introducing Hydra-0🐍: a generalist world model that represents robot actions as action flow. One interface. Many embodiments, tasks, and environments.
🧵1/7
9
71
11,097
Introducing ActionFlow, a very exciting research work led by @Hongyu_Lii and @Dr_YanChang from our team at @NVIDIARobotics, Isaac.
AI has repeatedly scaled by finding shared representations across diverse data, from language to vision.
Physical AI faces a similar challenge: humans and different robots may perform the same task, but their actions live in very different spaces.
Action Flow offers a powerful idea: represent action through a shared visual language of motion: connecting human + robot data, different embodiments, and world models.
Breaking these data and embodiment silos could be a key unlock for scaling Physical AI.
📄 arXiv: arxiv.org/abs/2608.18077
📷 Project/code: nvidia-isaac.github.io/video…
Kudos to contributors across our team and collaborators from Nvidia Isaac, Nvidia Research, Brown University, Columbia University and Harvard University.
@Hongyu_Lii @bowenwen_me @zhu_xinghao @YXWangBot @du_yilun @YunzhuLiYZ George Konidaris
@BirchfieldStan @80gg_overmind @Dr_YanChang
What if robots, humans, and video models could all describe action in the same language? Introducing Hydra-0🐍: a generalist world model that represents robot actions as action flow. One interface. Many embodiments, tasks, and environments.
🧵1/7
3
4
23
2,573
Soha Pouya retweeted
What if one action language could connect physics simulation, learned world models, humans, and robots?
I’m excited to share Hydra-0: Action Flow for Generalist World Modeling and Control. 🐍🤖
At its core, Hydra-0 bridges physics-based and world-model-based simulation. A physics engine executes the robot’s motion and translates it into action flow, while a learned world model predicts how the scene and objects respond—combining precise robot dynamics with flexible modeling of complex environments.
Instead of embodiment-specific joint or end-effector commands, action flow represents actions as pixel-space trajectories. This shared visual interface works across human hands, handheld grippers, single-arm and bimanual robots, and multiple video-model backbones.
Trained on 7 datasets with 178K episodes, 1.56M video windows, and 2,200+ hours, Hydra-0 achieves:
→ 90.4% lower robot-motion error
→ 60.2% lower object-motion error
→ Strong zero-shot and data-efficient transfer
The interface can also run in reverse: given desired object flow from a human demonstration, Hydra-0 predicts compatible robot motion and converts it into executable actions—without task-specific expert robot demonstrations.
Action flow provides a shared interface for heterogeneous data, hybrid simulation, world modeling, policy evaluation, and real-robot control.
📄 Paper: lnkd.in/gTjAw5kJ
🌐 Project: lnkd.in/gxN9J3xE
Huge thanks to the amazing collaborators
@Hongyu_Lii
@bowenwen_me
@zhu_xinghao
@YXWangBot
@du_yilun
@YunzhuLiYZ
George Konidaris
@BirchfieldStan
@SohaPouya
@80gg_overmind
@Dr_YanChang
7
19
2,045
SONIC is officially published in Science Robotics today and it made the Science front page.
We show the promise of scaling motion tracking toward natural, robust whole-body control for humanoid robots.
Huge thanks to the team, and more exciting work is on the way.
Paper: science.org/doi/10.1126/scir…
Code: nvlabs.github.io/GEAR-SONIC
3
21
195
19,822
Soha Pouya retweeted
Closing the sim-to-real gap for humanoid robotics starts with better simulation environments.
Join us, @NianticSpatial, and @FlexionAI on Aug. 12 at 11 AM PT to see a real-to-sim workflow using 3D Gaussian splatting, Omniverse NuRec, Isaac, and OpenUSD.
📅 Add to calendar: nvda.ws/3SmCmSu
8
32
178
8,883
Soha Pouya retweeted
Announcing Modeling Uncertainty in Robotic World Models Workshop at CoRL 2026.
📍 Austin, TX · Nov 12, 2026 📄 Submissions due Sept 23 (AoE)
Learned world models are getting powerful. They're still bad at knowing what they don't know. 🧵
2
20
88
29,124
#PhysicalAI is accelerating as advances in data, foundation models, evaluation, deployment, & agents reinforce one another across the #robotics stack.
Was great to be back at #Stanford sharing what we’re building at #NVIDIA to help accelerate the robotics development lifecycle.
1
1
12
595
At #NVIDIA, we’re bringing these capabilities together through the "Isaac GR00T Reference Architecture", helping researchers & developers spend less time rebuilding infrastructure & more time solving unique problems that move robotics fwd. More here: developer.nvidia.com/blog/de…
1
2
127
Soha Pouya retweeted
The room runs deep. @SohaPouya, Senior Director of Robotics Software Engineering at @nvidia, delivers the keynote. @SebastianThrun, founder of @Google X and @udacity, closes the day.
1
1
6
721
Soha Pouya retweeted
10 million downloads! 🎉
An incredible milestone for NVIDIA Cosmos—and a testament to the team’s vision, hard work, and commitment to advancing physical AI.
Proud of what we’ve built together, and even more excited for what developers and researchers will create next. 💚
10 million downloads and counting for NVIDIA Cosmos models on @huggingface. 🤗
This milestone belongs to the developers and researchers using open world foundation models to build robots, autonomous vehicles and other physical AI systems.
Thank you for downloading, experimenting, and building with us. 💚
ALT Robotic figures working around a large golden '10M' on a platform.
9
4
34
5,998
Soha Pouya retweeted
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
15,990
29,295
171,594
66,481,861
Soha Pouya retweeted
Dreamers deserve tools they can actually build with. An API gives you answers. Open weights give you a foundation — something you can fine-tune, tear apart, learn from, and make yours.
That's how one model becomes ten thousand things its creators never imagined: a medical assistant in a language the original team didn't speak, a robot policy trained in a garage, a research idea tested overnight instead of never. Closed models scale usage. Open models scale creation.
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
7
11
65
12,546
Soha Pouya retweeted
At 4:20pm today, I will give a talk on Cosmos3 in wangywust.github.io/cvpr-tut…
If you are at CVPR and interested about Cosmos3, please come.
2
6
66
4,708
NVIDIA Cosmos 3 is out - an OmniModel that brings together video, audio, action, & language understanding and generation into a single foundation for Physical AI.
Proud of our team's contribution to this milestone & incredible collaborations across NVIDIA that made this happen.
Introducing NVIDIA Cosmos 3
We released NVIDIA Cosmos 3 last night.
And today, seeing it take the top spots across 8+ open model leaderboards feels surreal. We spent months working towards this moment.
Here’s the breakdown:
The Leaderboard Wins
World Reasoning
🏆 #1 open model on VANTAGE-Bench for vision AI
🏆 #1 overall on Traffic Anomaly Reasoning (TAR)
World Generation
🏆 #1 open model on Artificial Analysis Image-to-Video leaderboard
🏆 #1 open model on Artificial Analysis Text-to-Image leaderboard
🏆 #1 open model on PAI-Bench for physical AI synthetic data generation
🏆 #1 open model on Physics-IQ, which measures accuracy on physical laws
🏆 #1 open model on R-Bench for world generation quality
World Action
🏆 #1 on RoboArena for specialized policy
🏆 #1 on RoboLab for action generation
But the leaderboards are only part of the story. The real story is why we built Cosmos 3 in the first place.
The Problem
Training robots and autonomous systems in the real world is painfully hard.
Robots need to try the same thing numerous times before they succeed reliably. Self-driving cars need rare edge cases that may never happen naturally. Smart machines need to understand physics, motion, contact, failure, and surprise.
And real-world data is slow, expensive, and sometimes dangerous to collect. At some point, the answer cannot just be “collect more data.”
You can’t collect your way out of an infinite physical world. You have to generate it.
That… was the question behind Cosmos: Can one model understand the physical world deeply enough to reason about it, simulate it, and generate actions inside it?
What We Built
Cosmos 3 is the first omni-model for physical AI. It can understand and generate across: language · images · video · audio · action sequences
It is not just a VLM.
Not just a video generator.
Not just a robot policy model.
It is all of them, in one single model.
That matters because physical AI has been fragmented for a long time. Cosmos 3 is our attempt to collapse that fragmentation.
Depending on how you configure the inputs and outputs, the same model can act as a vision-language model, a video/world generator, a world simulator, or a world-action model.
No separate architecture required.
The Architecture
Under the hood, Cosmos 3 uses a dual-tower Mixture-of-Transformers architecture.
One tower is autoregressive for reasoning. It handles next-token prediction for language and discrete understanding.
The other tower is diffusion-based- for generation. It denoises images, video, audio, and action trajectories.
Two towers. Dual-stream joint attention. One shared world representation.
Each modality gets its own tools: visual encoders, video VAEs, audio VAEs, and action projectors that can map different embodiments into a unified action space.
Action is a first-class modality in Cosmos 3.
That’s what makes it more than a video model. It doesn’t just predict and generate what the world might look like. It can connect reasoning and world modeling to physically grounded action.
Why This Matters
One of the most interesting findings from the ablation work is that training action domains together creates positive transfer.
That means adding more embodiments does not just add more use cases. It can actually make the model better.
This is the heart of why omnimodal training matters.
A shared world representation is not just convenient. It can make each individual task stronger. That’s the part that feels like the beginning of something much bigger.
The part I’m most excited about is that Cosmos 3 is fully open.
Developers get the models, scripts, optimization, inference endpoints, post-training recipes, datasets, and benchmarks.
Everything is available under the Linux Foundation’s OpenMDW 1.1 License.
You can use Cosmos 3 out of the box. You can use the VLM, world model, or world-action pieces separately.
You can post-train it for your own domain, embodiment, or accuracy target.
That’s what makes this feel different.
Cosmos 3 is not just a model release. It is the foundation for building intelligence for autonomous machines.
For me, Cosmos 3 feels like a step toward a world where physical AI development becomes much more scalable and accessible - to a new age of developers and agents.
That’s what we built Cosmos 3 for. I cannot wait to see what you build with it.
Download Models on Hugging Face
huggingface.co/collections/n…
Customize Models on GitHub
github.com/NVIDIA/cosmos
Read the Tech Blog to Learn More
developer.nvidia.com/blog/de…
1
177