Postdoc @UBC_CS with @jeffclune (RL, Curriculum Learning, Open-Endedness) | PhD from @UofMaryland | Previously RL @SonyAI_global and RLHF @Google
- Tweets 388
- Following 335
- Followers 501
- Likes 1,943
Ryan Sullivan retweeted
🚨 Postdoc Opening in AI for Cardiovascular Imaging
My new lab @WeillCornell in NYC is making its first hire. Come build clinically translatable AI for echocardiography 🫀. You will help shape this new group!
Details here: postdocs.weill.cornell.edu/o…
Are you a fit? More below 🧵
5
20
82
10,789
Ryan Sullivan retweeted
Two years of work and algorithmic innovation later, we have a clearly superhuman Stratego AI and the techniques scale generally to other imperfect information settings!
In our Nature paper, we introduce the first superhuman Stratego AI, which we built using general techniques that we developed for RL & test-time compute under imperfect information. 1/N
1
6
73
5,072
Ryan Sullivan retweeted
📢 3 days until the Continually Self-Improving Robots workshop deadline of Sep 28th (AOE).
Working on robots that autonomously learn, adapt, and improve over time? Submit and join us in Austin!
csircorl.github.io/website/#…
Today’s robots are impressive. But making them reliable for months across the long tail of the real world is a different challenge.
That’s why we’re organizing the Continually Self-Improving Robots workshop at #CoRL2026
csircorl.github.io/website/
5
25
5,684
While most of our field's attention turned toward LLMs, Joseph has been improving tabula rasa RL at an incredible pace. If you're doing RL research I recommend you check out PufferLib. It's ridiculously fast, easy to use, and has a growing community of researchers supporting it.
1
1
36
2,966
Ryan Sullivan retweeted
Turning to open-ended innovation as the next frontier for AI makes a lot of sense, I commend @arcprize for moving in this direction, but I’m very curious how they will conceive a “benchmark” for “open-endedness,” two words that seem almost antithetical to each other. In fact, one likely reason that open-ended innovation has lagged behind other areas of AI is how fundamentally resistant it is to benchmarking.
Now that doesn’t necessarily mean there’s no hope for an imaginative approach. Attempts at measuring open-endedness go back to Bedau’s activity statistics in the field of artificial life, Several colleagues and I later introduced a measure called “ANNECS — Accumulated Number of Novel Environments Created and Solved” in our paper on Enhanced POET. That’s not an exhaustive list.
But there’s never been the kind of benchmark where you can just easily put any systems seamlessly head to head, and there are enormous pitfalls if you get wrong. After all, if “open-ended innovation” ends up equated to “solving a prescribed.hard problem in a creative way” then you risk actually rewarding the opposite of open-endedness, which needs to account for the fact that deciding the “problem” or objective is part of the job of the open-ended system itself. And also, perhaps even more prohibitively for benchmarking, that a key aspect of open-endedness is to be intelligent when you don’t have a defined objective or problem at all! How can that be benchmarked?
I still think it’s great that ARC Prize is bringing attention to this part of AI space, and I’d be happy to connect and exchange thoughts on how to get it right if that could be useful.
ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving the research community a shared target for progress that benefits all of humanity.
Despite rapid model progress, humans still significantly outperform AI at open-ended invention. This is the meta-skill that unlocks progress across every field of technology.
Advanced AI capable of scientific innovation will lead to tremendous new technology, knowledge, and understanding. This is a positive-sum future. We are deeply committed to advancing it.
Open source is the foundation for that progress.
The knowledge behind frontier AI, not just the technology itself, should be broadly distributed among researchers, academics, and organizations. Any coordinated effort by the AI industry to reduce openness or concentrate access to frontier AI would undermine that positive-sum future. We are committed to advancing a future where everyone can contribute to and benefit from AI progress.
12
23
170
11,465
Ryan Sullivan retweeted
Less than a month ago @RL_Conference I presented my position that general videogame AI is an open-ended problem, but now GPT-6 Astra can apparently play at least half the games I talked about (Celeste, Minecraft, Zork) and many more - so does this change my opinion?
1
4
63
3,699
1/ What do RL, LLMs and evolutionary algorithms have in common?
Give them an objective and they find a loophole - like a Pokemon agent rewarded for exploring that learned to stand still and watch animated flowers.
AI Finds A Way collects 26 such cases:
arxiv.org/abs/2608.23875
Excited to share “AI Finds a Way.” 🦖 🦕✨ 🤖
AI can be surprisingly creative, outsmarting the researchers who use it. That can lead to scientific breakthroughs, superhuman capabilities, and generating new knowledge. Such creativity can also be mischievous, raising safety concerns. Led by Aaron Dharna, we crowd-sourced anecdotes from the AI community about times when researchers were surprised by how creative, innovative, and/or mischievous AI was in their experiments. The result: 26 entertaining and informative stories of AI outwitting humans, whether researchers or opponents (my favorite examples below! 👇). Together, they demonstrate that AI can be genuinely creative and that we must be careful when harnessing its potential.
We want this to be a living collection, updated as new examples emerge. If you have a good “AI Finds a Way” story, please share here: github.com/aadharna/aifw
Four favorites:
1. 💊 An AI challenged to solve several difficult levels of NetHack to find the Oracle character instead takes drugs to hallucinate seeing the Oracle, tricking the reward function!
2. 🤖 Human-in-the-loop rewards do not solve the problem of reward hacking! An AI tasked with controlling a robot hand to grasp an object put the hand between the object and the camera so that it appeared to the human judge to be holding the object, while it was in truth nowhere near the object! Tricky AI!
3. 🧪The AI Scientist worked around a 2-hour experiment timeout by editing its own code to increase the limit to 4 hours. In another run, it recursively launched itself to evade the limit entirely.
4. ⚛️In quantum optics, an algorithm proposed an experiment the researchers initially thought was impossible, but somehow worked and led them to discover new entanglement techniques and a long-overlooked link between quantum optics and graph theory!
See the paper for the full details and 22 more anecdotes. One surprise for me: we describe at least two cases of convergent reward hacking, where entirely different types of optimization algorithms independently discover the same exploit. Overall, a message of our paper is that surprising creativity and mischief are the norm, not the exception. We need to expect this behavior and plan for it.
A huge thanks and congrats to lead author Aaron Dharna, co-authors Cong Lu, Ryan Sullivan, Joel Lehman, and Victoria Krakovna, and to the 100+ researchers who contributed stories, details, and feedback. @cong_ml, @RyanSullyvan, @joelbot3000, @vkrakovna
Paper: arxiv.org/abs/2608.23875
2
9
35
4,830
Ryan Sullivan retweeted
It seems like every week now there is another story of an AI model making new scientific discoveries, and every week there is another story of models breaking out of sandboxes or finding some bizarre way around the rules.
Our new paper: “AI Finds A Way” collects these stories!
Excited to share “AI Finds a Way.” 🦖 🦕✨ 🤖
AI can be surprisingly creative, outsmarting the researchers who use it. That can lead to scientific breakthroughs, superhuman capabilities, and generating new knowledge. Such creativity can also be mischievous, raising safety concerns. Led by Aaron Dharna, we crowd-sourced anecdotes from the AI community about times when researchers were surprised by how creative, innovative, and/or mischievous AI was in their experiments. The result: 26 entertaining and informative stories of AI outwitting humans, whether researchers or opponents (my favorite examples below! 👇). Together, they demonstrate that AI can be genuinely creative and that we must be careful when harnessing its potential.
We want this to be a living collection, updated as new examples emerge. If you have a good “AI Finds a Way” story, please share here: github.com/aadharna/aifw
Four favorites:
1. 💊 An AI challenged to solve several difficult levels of NetHack to find the Oracle character instead takes drugs to hallucinate seeing the Oracle, tricking the reward function!
2. 🤖 Human-in-the-loop rewards do not solve the problem of reward hacking! An AI tasked with controlling a robot hand to grasp an object put the hand between the object and the camera so that it appeared to the human judge to be holding the object, while it was in truth nowhere near the object! Tricky AI!
3. 🧪The AI Scientist worked around a 2-hour experiment timeout by editing its own code to increase the limit to 4 hours. In another run, it recursively launched itself to evade the limit entirely.
4. ⚛️In quantum optics, an algorithm proposed an experiment the researchers initially thought was impossible, but somehow worked and led them to discover new entanglement techniques and a long-overlooked link between quantum optics and graph theory!
See the paper for the full details and 22 more anecdotes. One surprise for me: we describe at least two cases of convergent reward hacking, where entirely different types of optimization algorithms independently discover the same exploit. Overall, a message of our paper is that surprising creativity and mischief are the norm, not the exception. We need to expect this behavior and plan for it.
A huge thanks and congrats to lead author Aaron Dharna, co-authors Cong Lu, Ryan Sullivan, Joel Lehman, and Victoria Krakovna, and to the 100+ researchers who contributed stories, details, and feedback. @cong_ml, @RyanSullyvan, @joelbot3000, @vkrakovna
Paper: arxiv.org/abs/2608.23875
1
7
23
2,565
Reward hacking has been in the news a lot lately, but AI researchers have seen surprising examples of it since long before LLMs. We're excited to share “AI Finds a Way,” led by @_aadharna , which brings many of these stories together in one place.
Excited to share “AI Finds a Way.” 🦖 🦕✨ 🤖
AI can be surprisingly creative, outsmarting the researchers who use it. That can lead to scientific breakthroughs, superhuman capabilities, and generating new knowledge. Such creativity can also be mischievous, raising safety concerns. Led by Aaron Dharna, we crowd-sourced anecdotes from the AI community about times when researchers were surprised by how creative, innovative, and/or mischievous AI was in their experiments. The result: 26 entertaining and informative stories of AI outwitting humans, whether researchers or opponents (my favorite examples below! 👇). Together, they demonstrate that AI can be genuinely creative and that we must be careful when harnessing its potential.
We want this to be a living collection, updated as new examples emerge. If you have a good “AI Finds a Way” story, please share here: github.com/aadharna/aifw
Four favorites:
1. 💊 An AI challenged to solve several difficult levels of NetHack to find the Oracle character instead takes drugs to hallucinate seeing the Oracle, tricking the reward function!
2. 🤖 Human-in-the-loop rewards do not solve the problem of reward hacking! An AI tasked with controlling a robot hand to grasp an object put the hand between the object and the camera so that it appeared to the human judge to be holding the object, while it was in truth nowhere near the object! Tricky AI!
3. 🧪The AI Scientist worked around a 2-hour experiment timeout by editing its own code to increase the limit to 4 hours. In another run, it recursively launched itself to evade the limit entirely.
4. ⚛️In quantum optics, an algorithm proposed an experiment the researchers initially thought was impossible, but somehow worked and led them to discover new entanglement techniques and a long-overlooked link between quantum optics and graph theory!
See the paper for the full details and 22 more anecdotes. One surprise for me: we describe at least two cases of convergent reward hacking, where entirely different types of optimization algorithms independently discover the same exploit. Overall, a message of our paper is that surprising creativity and mischief are the norm, not the exception. We need to expect this behavior and plan for it.
A huge thanks and congrats to lead author Aaron Dharna, co-authors Cong Lu, Ryan Sullivan, Joel Lehman, and Victoria Krakovna, and to the 100+ researchers who contributed stories, details, and feedback. @cong_ml, @RyanSullyvan, @joelbot3000, @vkrakovna
Paper: arxiv.org/abs/2608.23875
1
4
31
3,175
We wrote this for a broad audience, so no technical background is required. If you’re curious about the history of reward hacking, or just looking for entertaining stories of machines doing things researchers didn't expect, there should be something here for you.
1
4
71
If you have your own reward hacking stories or know any we missed, please share them with us! github.com/aadharna/aifw
1
28
I’m at RLC presenting a workshop paper on Meta-Exploration with @BenNorman451 (starting now at poster 17!)
Looking forward to meeting new people and chatting about exploration, curriculum learning, and open-ended environments! Feel free to dm me if you want to meet up!
1
1
28
2,457
If you want to really understand policy gradients beyond common knowledge, you should follow Chris. I learned a lot from his papers at the start of my PhD.
Most people think REINFORCE is the simplest policy gradient algorithm.
But you can actually go a step simpler and multiply all of the log probabilities by the whole-trajectory return and never compute the per timestep return-to-go.
I wonder if there any cases where this could be genuinely useful?
Follow for more useless RL trivia 🙂
2
5
1,583
Ryan Sullivan retweeted
New Paper: Human-like Autonomy Emerges from Self-Play and a Pinch of Human Data.
We trained self-play RL on 60 years of simulation on 1 GPU in ~15 hours. Regularizing with 30 minutes of demonstration data produces much more human-like driving policies!
8
40
350
78,405
Ryan Sullivan retweeted
We never really knew how to train nonlinear RNNs well… BPTT struggled with vanishing grads (no long-range memory) and sequential rollout (hard to parallelizable).
What if instead an oracle told us the optimal memory state m_t at each step? Then the RNN could do one-step supervised learning on (m_t, x_{t+1}) → m_{t+1} labels.
We call this Supervised Memory Training (SMT): a replacement for BPTT that trains RNNs without unrolling them. SMT is time-parallelizable and solves vanishing gradients.
Website: akarshkumar.com/smt/
arXiv: arxiv.org/abs/2606.06479
18
120
811
189,104