Founder @alter_org_il, Methods @EvalConsensusAI, emeritus @superforecaster, PhD @PardeeRAND Optimistic on AI, pessimistic on humanity managing the risks well.
Rehovot, Israel/Washington DC
- Tweets 72,298
- Following 2,176
- Followers 11,674
- Likes 100,922
Great to attend the "AI Safety as a Global Security Imperative" workshop hosted by @CIGIonline / @thefuturesoc / @MIGSinstitute / @GladstoneAI this afternoon in Montreal.
Thanks to Duncan Cass-Beggs, @BalkanDevlen, @KathrinGar31229 and others for leading the discussions.
2
1
7
495
David Manheim ✈️ Montreal, Oct 26-28 retweeted
To riff on @ESYudkowsky's very different quote;
"Go three-quarters of the way from heuristic decision making to Bayesianism and then stop. You are now in the right place. Stay there at least until you're no longer computationally limited."
(And I suspect he'd mostly agree.)
1
2
19
652
It's kind of surreal to attend serious briefings for policy folks that are recapitulating what @allTheYud was arguing over a decade ago.
3
3
65
1,864
I'm kind of hoping for a proof that P=NP for polynomial-time decision algorithms, but with a construction lower bounded by some ridiculous 3- or 4-digit degree polynomial making it irrelevant, so theoretical CS people scream about it and it doesn't matter.
Updating my prior answer (15%) to 20% in light of ETH being false. It's elegant for P to not equal NP, but the world wouldn't need too look substantially different if they were equal.
3
1
29
1,972
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Could LLMs make chemical attacks easier to carry out? Ryan Stendall, Freddie Martin & @JonasSandbrink examine how AI may lower barriers to chemical weapons misuse & remaining obstacles.
Free access until the end of October in our AI-WMD special issue!
doi.org/10.1080/10736700.202…
1
1
2
237
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I'm not saying that the problems AI companies are having with RL environments and reward hacking are just evaluation design problems that they never bothered solving, but...
Made with AI
1
1
4
454
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I'd love for them to each suggest one policy that they think the others would all support.
I'll start with one too-easy example; funding NIST CAISSI appropriately to do its job.
Strongly agree; there is a lot of convergence on easy-win policy that all of @SnehaRevanur, @DKokotajlo, @tylercowen, @AlexBores, & @Kabakab would endorse, but newspapers focus on the "interesting" places of disagreement instead of highlighting those. (Unfortunately.)
1
1
1
201
David Manheim ✈️ Montreal, Oct 26-28 retweeted
We’re Fangcun AI (方寸跃迁), China’s first lab dedicated to independent third-party evaluation & technique research on frontier AI safety.
Today we’re publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
48
129
1,152
111,736
David Manheim ✈️ Montreal, Oct 26-28 retweeted
People say hosting a foreign open-weight model on sovereign compute is safe.
This is an example it's not.
Sovereign compute controls where a model runs, not what it does.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
21
14
108
18,884
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Based on my formal proof expertise, combined with my lab's work on specification synthesis for proof assistants (going on for a few years now but wildly hard to publish):
1. You should never trust the spec; you should vet the spec.
This seems like the fundamental problem with formal verification methods I'm seeing: if you trust an LLM to flawlessly convert your legacy codebase to a formal spec, wouldn't you just trust its initial code output anyways?
8
13
166
7,312
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Unrelated to everything: as we are now a bit more than ten years into the deep learning revolution, we should be incredibly conscious of the following dynamic:
Step 1: A technology prototype based on a simple principle is demonstrated.
Step 2: The strongest argument that it won't work amounts to "nuh-uh, the problem has to be harder than that."
Step 3: The consequences for the prototype becoming a mature technology are so extreme that many people reject either the conclusions or the argument that the principle really works.
Step 4: As the tech advances, disbelief becomes entrenched. This creates significant resistance to seriously preparing for the impacts of the technology.
Step 5: The technology becomes mature and the consequences are increasingly realized. Because of entrenched disbelief over time, people didn't prepare enough.
For whatever next technology might follow this pattern, we need to aggressively push back on the disbelief from earlier on in the cycle. We cannot afford to let such short-sighted viewpoints derail society-wide preparedness again. The stakes and consequences are only getting higher.
10
15
137
7,216
As @LongResilience pointed out, the short version is that these two papers say that we don't have good ways of safeguarding these models and preventing misuse, and we need to both develop better methods, and use all of the ones we have.
New research from AISI's Chem-Bio team: two studies on safeguarding biological AI tools - specialised models that are driving rapid advances in biology, from predicting protein structures to designing molecules. 🧵
3
393
David Manheim ✈️ Montreal, Oct 26-28 retweeted
WSJ op-ed:
"For investors mulling the Anthropic share offering, a 10% chance of extinction may be a 90% win for your portfolio"
2
4
41
2,059
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Look at all of these broken windows getting replaced, it's great for the economy!
i think the 'cyber as battle domain' framing is completely incorrect, where the correct framing is: look at all this infrastructure that we get to replace at a profit. i wouldnt call this defense at all, i'd call this the market moving according to need.
it's a good thing
4
2
17
1,605
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Our Frontier Strategy & Governance (FSG) team in the DeepMind Institute at @GoogleDeepMind is hiring a Senior Program Manager in London. Our North Star is stewarding AGI safely into the world for the good of humanity. 1/n google.com/about/careers/app…
1
6
39
5,463
Excited to be here in Montreal for the @isostandards plenary on Artificial Intelligence standards as an expert on behalf of @StandardsIsrael.
2
4
368