Pierluca D'Oro retweeted
⚡️Few-step diffusion without a teacher, self-distillation or JVPs. iDDM: a single model trained from scratch in one stage. ImageNet-256: 4.48 FID at 4 steps and 2.38 at 50, from the same checkpoint 🧵 (1/6)
5
36
194
12,205
Our paper on computer use agents evaluation has been selected for an oral presentation (112 papers among 30709 submissions) at NeurIPS! We took a holistic approach to AI agents evaluation, and worked across the scientific stack (irreverent experiments, theory, statistical framework, conceptual design principles, complex engineering artifacts) to challenge common assumptions while simultaneously advancing fundamental understanding and providing new tools to rigorously benchmark agents. In summary: - Irreverent experiment: we show that in common CUA benchmarks a "replay agent" that just memorizes a frontier model's successful trial can reach state of the art performance when evaluated under common protocols. - Theory: we mathematically prove that the unsettling replay agent is formally the agentification of the pass@k metric that has been often used by the community, exposing the flaws of this metric. - General principles for environment design: we propose a set of general principles (the PRISM principles) to design benchmarks for AI agents, that allow to avoid easy replay and other common problems often found in existing environments. - Digiworld as a showcase of the PRISM principles: we instantiate these principles into a large benchmark for mobile computer use. Digiworld avoids replay agent attacks by automatically generating and checking millions of variations for each one of its apps and tasks. - Statistical framework: to handle this variability, we draw from work on rigorous evaluation for reinforcement learning agents (by Rishabh Agarwal Marc G. Bellemare and others) and build a framework able to accurately estimate an agent's performance by leveraging the structure of a benchmark. - Evaluation of existing models: armed with our new environment and framework, we show that existing frontier models used as CUA agents can be quite brittle to variations of apps and tasks that wouldn't make a big difference for humans (e.g., UI colors). This was extremely fun to work on together with Sneha Silwal, Allen Bolourchi, and other collaborators at Meta Superintellligence Labs. See you at NeurIPS! If you are interested in talking about rigorous evaluation of AI agents, reach out!
6
8
38
2,242
I have an answer for this, but you guys might not like it
To what degree would people say that so-called “reward hacking” by AI models is a conceptual descendant of what would have been called “overfitting” in the ML era?
2
6
1,026
Live now: our Computer Use Track from AI Engineer World's Fair 2026. Nobody is going to rebuild the web for your agent. piped.video/watch?v=Ki980nV0… - Computer Use models will agentify the web, not APIs: @DhruvBatra_, Yutori - Computer Use at the Edge of the Statistical Precipice: @proceduralia, Programma Labs - Bringing agents onto the world wide web: @pk_iv, Browserbase - The Dark Arts of Web Automation: @CoreyGallon, Rexmore - From RL to IRL: Gaurav Mishra, Amazon AGI Lab - The Rise of CaaS, context as a service: Omer Primor, Bright Data - Computer-Use 2.0, agents just got multi-cursor: @francedot + Dillon DuPont, Cua - Persona Engineering, a field guide to synthetic personas: @ianand, InsightSciences.ai - How Web Data Infrastructure Powers the Next Generation of AI: Patricija Žemaitytė, Oxylabs
4
6
27
4,205
Euro: Sicilian edition
1
5
510
I agree, and there's more to it. There is a entire class of problems that I call deployment-complete"reinforcement learning problems. Computer use belongs to this class. RL works well when one can easily reset the environment to diverse enough situations, when there is a clearly-defined reward signal, when one can easily generate trajectories in parallel, when the agent can generate actions without worrying about safety. Guess what? For most realistic computer use situations, you can't do any of this off-the-shelf. It's quite surprising, isn't it? It's stuff on a computer but it feels like blockers are all over the place not differently from what you would see in the physical world. To me, it almost feels like training to train a robot. And yes, as you can guess, also robotics is a deployment-complete reinforcement learning problem. So what do people do in robotics to overcome this problem? They build simulators, environments that mirror the one they ultimately care about but that satisfy the properties above. The advantage of computer use compared to robotics is that building environments is potentially easier, because you don't need to model the clumsy physical world. One could say you just gotta build a bunch of websites/apps, that are coincidentally the easiest thing to build with modern AI agents. But then why isn't CUA already solved? Because, in reality, to build an environment that is simultaneously useful and scalable for a deployment-complete RL problem, the environment shouldn't be just a bunch of websites, but it needs to be designed with extreme craft and effort. There are many things that could be said about what this extreme craft entails, but to me, it means that it should at minimum follow this set of design principles: - Privileged verification (P): Task completion must be verified as much as possible against the environment’s internal ground-truth state. - Realistic environments (R): Environment's apps must faithfully reproduce real-world apps. - Integrity-checked configurations (I): Every environment configuration must be automatically verified. - Sandboxed execution (S): The environment must be fully self-contained and must support deterministic reset to any specified initial state. - Multifactorial variability (M). The environment must vary independently along multiple axes. I call them the PRISM principles. In practice, they entail pretty hard engineering and research challenges. But they define what needs to happen, at minimum, to have a good RL environment for CUA. If you want to learn more about this, and how this is also the only way out of the related CUA evaluation crisis, come to my talk at the @aiDotEngineer conference next Wednesday in SF, or reach out!
Here's a question I find confusing and interesting and which actually tells us a lot about the nature of current AI progress: Why has progress on computer use been so slow? Computer use is so clearly verifiable. I think the answer is that it is not enough for a domain to be verifiable. It also has to be very grindable—in the sense that you can run lots of parallel rollouts against a deterministic and replayable simulator. If you’re trying to make a model better at coding, you can create an environment that has a software repo with some missing feature that you’ve tasked the AIs with creating, and then you have a thousand parallel agents just go at the problem, each with their identical copy of the container. But this doesn’t work with computer use—at least not trivially. You can’t have a thousand agents go try the same checkout flow on Amazon. Because Andy Jassy will find and detect your bots and shut your ass down. How would we train an AI to build a business? How would you make an AI that’s really good at winning court cases? Or having a profitable day trading in the markets? Or helping a candidate win an election? What is the RL environment to make an AI as good at politics as Lyndon Johnson, or as good at building a space launch business as Elon Musk? The rollout requires interacting with the world and cannot be recreated simply within the datacenter. And the outer loop verification may take months or years of real world actions to elicit, and cannot be re-observed by perturbing the model’s actions thousands of times in parallel so that you can isolate what exactly the model did that actually worked.
5
1
25
6,168
Pierluca D'Oro retweeted
Replying to @antirez
IMO in Europe and especially in Italy the overlooked bottleneck is the inability to create a community that simultaneously shares values and technical chops to move things forward. Such communities build hope, energy, and strong bottom-up leverage. In Silicon Valley, skilled and aligned people find each other, encourage each other, and sustain each other way more, with open mind and curiosity. This accelerates worthwhile change. If this dynamics is not there, the largest "community efforts" naturally remain top-down and regulatory, like many of the EU ones. I see things changing, but too slowly. Maybe this wasn't as important in the pre-AI tech world? But honestly now it feels to me like a hard prerequisite to effective use of capital.
1
3
672
I myself didn't fully understand what the deepseek r1 hype was about, but at some point I realized where the mismatch in hype was coming from. For many people with reinforcement learning knowledge, the o1 announcement was pretty clear. o1 was a model trained with reinforcement leaning to dynamically generate chain of thoughts, using verifiable rewards. I mean, I didn't know the details of their training recipe or anything like that. But the general strategy was pretty much implied by their blog post and release. To those people, r1 is an open source reproduction of the same, implied, strategy, or at least, the first one that got really good performance. That is extremely valuable as an engineering contribution, but far from being an actual scientific breakthrough. I came to learn that many other people, typically without a technical RL background, instead consider deepseek r1 the first model trained to reason with reinforcement learning and verifiable rewards, associating o1 to a possibly different paradigm (e.g., MCTS). It's honestly a good reminder of how history (e.g., of science) can feel very different depending on your perspective
Looking back, I never understood the hype of the "deepseek moment" from last year. Distilling others' models is possible and easier than pushing the frontier. Like, nobody remembers Alpaca?
2
10
4,330
I have decided to leave Meta Superintelligence Labs. I’ve been at Meta for more than three years, starting during my PhD. I’ve learned a lot during this time, and I’m grateful to have worked with many amazing collaborators. We advanced the frontier of LLMs, reinforcement learning, and computer use agents. I’m pretty proud of our latest work on rigorous agent evaluations and CUA envs (check it out: arxiv.org/abs/2605.08261). Looking forward to taking some time to explore what’s next!
16
12
319
31,169
now THIS is cooking hard
chagpt images 2.0 can cook hard
1
16
2,516
Pierluca D'Oro retweeted
You deserve more than a crowded Blue Bottle and a batch of 400 startups... I'm launching Kernel Grants, a pre-seed program that will invest $271,828 in 10 founders/year who are building tooling and infrastructure for the token factories of the future. We have an amazing set of speakers lined up for our first set of events: - @pirroh, President of Replit - @soumithchintala, CTO of Thinky - @jeremyphoward, Founder of Answer.ai - @NaderLikeLadder, Dir. of DevTech at NVIDIA - @OfficialLoganK, MOTS at DeepMind (and first ever Latent Space guest!) - @clattner_llvm, Founder of Modular - @dylan522p, Founder of SemiAnalysis - @swyx, Editor of Latent Space (+ AIE, Cognition, etc!) Batches are a relic of pre-AI acceleration. Any day is a great day to start building, so applications are open and we accept founders on a rolling basis. Let's build! kernellabs.ai/grants Enjoy an exclusive tour of our Kernel space 👀
18
19
182
24,819
enjoy!
1/ today we're releasing muse spark, the first model from MSL. nine months ago we rebuilt our ai stack from scratch. new infrastructure, new architecture, new data pipelines. muse spark is the result of that work, and now it powers meta ai. 🧵
2
714
this assumes that intelligence scales nicely and uniformly, that failing small pattern tasks means failing big science. but intelligence is asymmetric. humans who crack deep problems often stumble on simple ones. why should we aim for AI to be different?
Many people expect that current AI is ready to cure cancer and do breakthrough new science. ARC-AGI-3 envs are like a microcosm of the scientific method: you must observe a tiny world, form a theory of how it works, test it, iterate until correct. Over the course of a few minutes. If AI can't do it in an ultra-simple, ultra-small scale setting that is explicitly designed to be as accessible as possible, I expect there are a few steps missing until AI can crack the nature of reality.
1
1,074
Pierluca D'Oro retweeted
New Anthropic Fellows research: Abstractive red-teaming of language model character The worst way to find out about a character flaw in your language model is from a viral screenshot. How can we find these issues before deployment, rather than after? In this work, we introduce abstractive red-teaming, a new approach that searches over natural-language categories of queries, rather than individual prompts.
2
28
148
18,938
wait, if the test for AGI is hard games, why don't we just use existing hard games as the benchmark? I feel like arc-agi-n is supposed to be basically nethack or dark souls... but we already have those, no reason to wait to build them this was the original idea of the pioneers of modern reinforcement learning (e.g., @_rockt,). why should we go in circles?
Announcing ARC-AGI-3 The only unsaturated agentic intelligence benchmark in the world Humans score 100%, AI <1% This human-AI gap demonstrates we do not yet have AGI Most benchmarks test what models already know, ARC-AGI-3 tests how they learn
16
5
63
9,309
Pierluca D'Oro retweeted
In the limit, what's important is our ability to adapt.  What is a good recipe for teaching agents to adapt on-the-fly? We introduce two meta-learning for LLMs papers written with @JonnyCoook at @GoogleDeepMind. This is research from last year we can finally share 🧵👇
9
48
254
56,356
Pierluca D'Oro retweeted
Only words of appreciation for the first Sicilian AI Workshop organized by @proceduralia “et al”. This is an example of local AI community that benefits everyone! Plus fantastic food & you’re by the sea. I hope this will grow in the archipelago of Mediterranean schools/confs ☀️☀️
3
15
822
Pierluca D'Oro retweeted
Our paper Perception Encoder (PE) was accepted as an Oral at NeurIPS 2025! 🎉 We show that scaling simple contrastive vision–language pretraining, refined with a robust video data engine, is enough to build a universal vision encoder. 🔹 Strong embeddings for image & video tasks (cls, retrieval, VQA, detection) 🔹 Alignment methods unlock multimodal & spatial understanding 🔹 Open models, code & new datasets incl. human-annotated videos 📄 Paper: arxiv.org/pdf/2504.13181 💾 Data & Checkpoints: tinyurl.com/48y2h8k9 🔧 Code: tinyurl.com/sp8sdd9h #NeurIPS2025 #multimodalai
4
25
255
18,177