🚀 Can your LLM forecaster outperform the best human forecasters? ForecastBench is the leading LLM forecasting benchmark and the only one that compares AI forecasters with superforecasters. This month we're running a new survey of superforecasters—setting a new baseline for top human forecasters on ForecastBench. If you're building an AI forecaster, now is your chance to test your system against the very best human forecasters. Submission details in the thread 👇

AI experts largely agree about which policies would most reduce catastrophic AI risk. The Longitudinal Expert AI Panel (LEAP) brings together forecasts from a representative group of over 250 experts that policymakers look to when considering the future of AI: computer scientists, industry figures, top economists, policy experts, and AI risk experts. We asked panelists to forecast (1) how a menu of policies would affect AI risk; (2) which policies they each support; and (3) which policies they think will be implemented by 2030. Key findings: • Despite disagreeing on the level of AI risk, respondents broadly agree on policy • Respondents believe government policy can substantially reduce catastrophic AI risk • The most supported policy of those we asked about was the creation of an international body, including the US and China as members, with pre-release authorization power over frontier models • Some US-only policies are forecast to reduce catastrophic AI risk, but not as much as bilateral policies involving the US and China • Experts strongly oppose federal preemption and think it would increase AI risk • The policies experts expect to reduce risk the most are those they think are least likely to actually be implemented 🧵 More, with key insights below:

𝗨𝗽𝗱𝗮𝘁𝗲 #𝟭: 𝗦𝗵𝗮𝗿𝗶𝗻𝗴 𝗳𝗼𝗿𝗲𝗰𝗮𝘀𝘁𝘀 𝗳𝗿𝗼𝗺 𝗮 "𝗳𝗮𝘀𝘁 𝗔𝗜 𝗽𝗿𝗼𝗴𝗿𝗲𝘀𝘀" 𝘀𝘂𝗯𝘀𝗮𝗺𝗽𝗹𝗲 The majority of forecasters in our Longitudinal Expert AI Panel (LEAP) have consistently underrated AI progress on benchmarks and other capability indicators. For example, experts and superforecasters gave median forecasts of 10% and 5.4%, respectively, that a Millennium Prize Problem would be solved with AI assistance before the end of 2027. However, some LEAP panelists predict much faster AI progress. These forecasters have tended to be somewhat more accurate on near-term benchmark and capability forecasts. This subsample puts more weight on the possibility of transformative AI arriving soon than most LEAP forecasters, and we're interested in what this group thinks about other AI progress questions. In future LEAP reports, we'll break out forecasts from this "fast AI progress" sample. Our goal is to highlight this group’s forecasts for people who want to put more weight on them, and to explicitly track where this worldview is more and less accurate. For details on how we might select this group, see our full post (link below).

FRI does not just preach transparency; it practices it

No. Once AI can manage to simulate with high accuracy the result of simply rolling a pair of backgammon dice - knowing all of the initial conditions as perfectly as it possibly can - then we can have that conversation. Until then, no.

We’ve run the most comprehensive series of studies on expert AI forecasts over the last four years. Today, we’re sharing an interim update on our findings about the accuracy of these forecasts. Our major findings are: 1. Experts, including top economists, computer scientists, and biologists, have dramatically underestimated AI capabilities progress each year. Superforecasters have underestimated progress to an even greater extent. 2. Experts have a more mixed forecasting track record on AI diffusion-related measures, with some major underestimates (forecasting AI revenue) and other forecasts on track to be accurate (the share of electricity used for AI). 3. Some notable cases of overestimating AI progress: how much mid-2025 AI models could help amateurs do biorisk-relevant laboratory tasks; the speed of rollout of self-driving cars. 4. It is too soon to say how forecasters have performed at predicting macro-scale impact on outcomes such as GDP growth, major AI harms, and averted deaths from disease. More 🧵

AI models now match superforecasters on ForecastBench, our LLM forecasting benchmark. (Although not by a statistically significant margin). This October, we're running the rematch: superforecasters return to the benchmark, and the top AI forecasters can take them on. 🧵 In this thread: • What this result means • Why we're bringing superforecasters back • How to compete

Multiple choice test: (a) Fear-mongering journalism (b) Impressive track record of self-negating prophecies; (c) 75% chance at least one of them was a self-negating prophecy; (d) Too soon to tell on at least four of them (e) Messy mix of all of the above

Forecasting is about constructing a web of relationships between events. Future models & harnesses with superhuman forecasting capability will construct validated webs with more complexity than humans can manage.

The core contribution of this new FRI paper on AI-risk isn’t a number. It is a research program that, one step at a time, spans the chasm between what we need to know & what is knowable, between “will AI kill 10%+ of us by 2030?” and “how many Millennium Problems will frontier models solve in the next 4 weeks?”

Introducing the Automated AI Risk Outlook (AIRO): a dashboard of catastrophic risk forecasts from an ensemble of frontier AI models, updated weekly. Current ensemble forecasts of an AI-related catastrophe that kills at least 10% of the global population: • By 2030: 0.47% • By 2050: 6% This dashboard is a prototype, and forecasting catastrophic risk is extremely challenging. We see the current forecasts as tentative, but we expect them to improve over time as AI capabilities advance. Dashboard: airo.forecastingresearch.org… Launch white paper: forecastingresearch.org/pdf/…

New preprint from @lightningrodai, this time with Philip Tetlock and Ville Satopää! We post-train 5 versions of the same LLM, changing only the scoring rule used as the RL reward. Similar aggregate scores, but very different BIN profiles. A good Brier score alone doesn't tell you if a forecaster can distinguish likely from unlikely events. One might discern well but lose if its probabilities run systematically too high. A less discerning one might score better by hugging the base rate. BIN splits forecast performance into bias, information, and noise. Bias is a systematic shift in the probabilities. Noise is random scatter. Information is real signal about which outcomes are more likely — the part you need a powerful LLM for. Different uses call for different profiles. Reward choice is one lever shaping which forecaster you get. Congrats to co-authors @indiequant @KSkotheim64001 @VSatopaa @PTetlock 🙌 Full paper: arxiv.org/abs/2608.28482

there is a surprisingly big cognitive-stylistic gap between people who say "certainly" versus "almost certainly--and zero versus .001 Granularity matters.

In his recent blog post, @slatestarcodex wrote that "the AI superforecasters are here." We run ForecastBench—a leading AI forecasting benchmark and the only one to compare AI forecasters to human superforecasters. So, how close are current AI systems to superforecasters? For the first time, several AI models are now statistically indistinguishable from superforecasters.

Brace yourself for the next wave of questions: what do the emerging AI superforecasters think about the predictions of the human experts and superforecasters on the risks of AI? Game theory run amok.

Paging @PTetlock!

"Debates about political bias in AI often conflate ideological leanings with epistemic error." @DavidRozado and @PTetlock offer a more coherent basis for evaluating and regulating AI systems in politically contested domains: link.springer.com/article/10…

1. Have we been measuring AI political bias wrong? In a new paper @PTetlock and I argue that we might have. Studies have found that AIs tend to produce left-of-center responses to politically loaded questions. But ideological preferences are not the same as epistemic failure.

New in @Theory_Society, David Rozado (@DavidRozado) and Phil Tetlock (@PTetlock) argue that an AI system leaning politically left or right isn't bias on its own -- real bias is when that lean coincides with epistemic distortions like inconsistent evidentiary standards, failures of perspective-taking or faulty inferences. Check out their fascinating article below! link.springer.com/article/10…

Seems @claudeai use Superforecaster manual by @PTetlock to ensure success.