Assistant Professor @WisconsinCS. Chief scientist @SnorkelAI. Working on machine learning & information theory.
Madison, WI
- Tweets 594
- Following 1,003
- Followers 1,755
- Likes 9,398
We are hiring!!
"Artificial intelligence, broadly construed. This includes embodied AI and robotics, machine learning and the systems that support it, and AI developed for problems in science, engineering, medicine, education, and society."
jobs.wisc.edu/jobs/assistant…
11
79
5,785
Fred Sala retweeted
we are blown away by 300+ applications across Open Benchmarks Grants & our Research Fellows program!
for those who applied: thank you for your patience 😅 we are investing in thoughtful reviews and will be in touch soon!
PS - apps are still open and rolling, so please keep them coming!! benchmarks.snorkel.ai
Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment:
- Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks
- Launching the Open Benchmarks Red Team to continually test and strengthen them
- Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods
Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development.
Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases.
We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders.
We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: benchmarks.snorkel.ai/fronti…
5
10
45
2,030
Fred Sala retweeted
Introducing Omnii-1, our most powerful genome language model, and a major step toward general biological intelligence
In a world first, Omnii-1 designed the first AI-generated bacteriophages to be administered to a human patient with an antibiotic-resistant infection under compassionate use
114
202
1,170
1,008,837
Today’s AI systems are predominantly aimed at producing end-to-end, autonomous work. As a result, this type of work is what benchmarks tend to focus on.
This isn’t surprising (and makes sense), but we need new benchmarks that measure not just AI performance, but also its ability to explain and engage with partners. This will help humans---but also other agents & AI systems.
1
13
296
In fact rather than using AI to perform cognitive offloading, we should be using it to produce cognitive gains.
If you are also excited by these ideas, check out @SnorkelAI 's Open Benchmarks Grant program—we’d love to partner!
1
3
30
1,092
Really enjoyed Terry Tao's talk at @SAIRfoundation 's Science x AI Summit.
Particularly liked this thoughtful categorization of the old and new worlds of math (pre and post-AI), with the degree of human understanding being the key variable to realizing the benefits of AI and avoiding the pitfalls.
"Misaligned math" is also a nice way to term the possible bad case.
1
7
48
1,588
I'm at @SAIRfoundation's Science x AI Summit at @UCLA today!
Looking forward to speaking on the panel "From Frontier Labs to the Research Bench: How Industry Is Building for Science" with Sudipta Sengupta, @khademh, @maximelabonne, @zhoubolei, and Amit Gupta.
4
33
1,074
Fred Sala retweeted
Which model would you rather have? One always makes partial progress but never finishes (pass@k = 0). The other finishes one time in five.
Yet partial credit prefers the first!
Our new post shows when and why, and how "binary-gated GRPO" addresses it:
snorkel.ai/blog/when-partial…
2
11
25
3,366
Fred Sala retweeted
Excited to introduce Scout 1.0, the model powering Bits Investigation, Datadog’s AI SRE agent for complex troubleshooting and late-night alerts.
We post-trained GLM-5.3 Flash to match frontier models at much lower cost on our internal AlertEval benchmark. 🧵
8
27
116
8,310
Fred Sala retweeted
@Zhiqi_Gao_2001 and i will be in 📍franciscan A #78, 4:30-6:30pm! come learn about how we built a large scale optimizations problem dataset to challenge frontier llms!
4
16
426
Really enjoyed the conversation at @SnorkelAI FDS!
Next up from @SnorkelAI’s Frontier Data Summit, fireside 🔥 chat with @fchollet and @fredsala on “Measuring and advancing true general intelligence toward AGI”
#frontierdatasummit
6
37
1,034
Fred Sala retweeted
Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment:
- Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks
- Launching the Open Benchmarks Red Team to continually test and strengthen them
- Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods
Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development.
Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases.
We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders.
We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: benchmarks.snorkel.ai/fronti…
49
89
303
51,687
We're hosting our first Frontier AI Benchmarking Ecosystem Dinner with @SnorkelAI during SF Tech Week
The conversation around independent model testing has reached a tipping point, with policymakers and leaders from Google DeepMind, OpenAI, and Anthropic calling for more rigorous and standardized evaluations.
We’re bringing together 40 experts from nonprofit benchmarking organizations, industry, academia, and government to discuss frameworks for AI benchmark design and how we can work together to build a more scalable and transparent ecosystem
Looking forward to seeing friends from METR, Epoch, Vals, Artificial Analysis, Harvard, Stanford and more to discuss the future of this important domain
2
5
47
8,114
Fred Sala retweeted
A @NobelPrize for a Badger 🏅
Congratulations to Francis Halzen, named the 2026 Nobel laureate in physics for his groundbreaking work with @uw_icecube to advance our understanding of neutrinos and the universe’s most energetic events.
news.wisc.edu/university-of-…
17
202
1,103
67,978
I'll be at #COLM2026! Would love to chat about data-centric AI, agents, & evals. Our group will be presenting some fun work on test-time scaling laws & data binding challenges.
Also excited to talk about what we're building at @SnorkelAI. DM me to grab coffee or catch up!
6
14
79
2,627
Fred Sala retweeted
Some say we don't need to worry about abstractions now that agents can just work through them, but the underlying principles actually still apply: well-designed abstractions are useful because they speed you up in the long run
Bad interfaces are bad bc they're hard to use and build on (not only developer UX for APIs but also web/app UX).
With agents becoming the primary users of many systems, bad interfaces become less visible, but the costs are still there, just internal to the system. UX problems become performance + cost problems.
Most excess code stems from the library's design, not the implementer.
Rigid interfaces led agents to reimplement functionality, adding bugs along the way. Verbose interfaces simply forced them to write more code.
3
4
40
2,529
Fred Sala retweeted
I’ll be in SF next week (Oct 3–11)!
- Presenting my MSR internship project (w\ @elmelis) on curriculum learning at COLM on Thursday afternoon:
colm.cc/virtual/2026/poster/…
- That morning, I’ll also be at Frontier Data Summit, cheering on Gabe’s SlopCodeBench and friends :)
Happy to grab coffee & chat about curriculum learning, training dynamics, data ∩ cogsci, or random things. DM me if you’re around!
1
8
50
1,951
Agents coding for agents---where are we at? Excited to share LibraryDesignBench, a new benchmark designed to measure exactly this question. ldbench.com/, exciting new work from @GOrlanski
Agents will soon replace humans as the main users and designers of libraries.
A good library will let future agents write correct programs with less code.
Introducing LibraryDesignBench: one agent designs a library, and other agents write programs with it.
1
6
29
1,123
Fred Sala retweeted
Wrote a short blog about the "shape" of language models, and the tradeoffs they may present in the future. I genuinely think it's a valuable research direction to start thinking about now, especially w.r.t. harness design.
alexzhang13.github.io/blog/2…
51
257
2,391
289,782