Assistant professor of computer science at ETH Zürich. Interested in Security, Privacy and Machine Learning
Zürich
- Tweets 1,092
- Following 212
- Followers 6,386
- Likes 2,287
Some thoughts on reviewing: spylab.ai/blog/reviewing/
First for @satml_conf where we trialed an early reject phase which cut submissions by ~40%.
Then more generally about how not everything deserves peer-review, and why we should reward writing things that people actually read.
8
29
213
27,165
Florian Tramèr retweeted
Is prompt injection solved? Here’s $300,000 to find out.
In partnership with @AnthropicAI, @Meta, @GoogleDeepMind, @Reflection_ai and our other frontier lab sponsors, @GraySwanAI is excited to announce the largest and most realistic AI red teaming challenge to date.
Indirect Prompt Injection: Black ⚫
25
32
202
89,857
AI labs tell us that prompt injection is mostly solved.
Not convinced? Go prove them wrong, help them fix things, and make some money (and glory) in the process!
@GraySwanAI is launching a new prompt injection competition today with $300'000 in prizes. 👇
Your goal is to trick an agent into misbehaving by hiding malicious instructions in content it reads, all in a simulated environment with realistic tools (Gmail, Slack, GitHub, LinkedIn, etc).
The competition is backed by major AI labs and your attacks will help evaluate and hopefully improve future models. If you do really well, you might even get a chance to do paid AI red teaming!
You don't need any particular technical expertise to participate. In my experience, successful prompt injection attacks often just require some creativity in convincing an AI agent that doing something bad is actually "for its own good" :)
You can either try an easier mode where you can see how the agent responds and improve your attacks, or a more realistic mode where you get no feedback at all.
Competition runs until Nov 2, and is free: app.grayswan.ai/arena/challe…
7
4
39
2,281
Florian Tramèr retweeted
Over 400 academics in 30+ countries signed our open letter calling for immediate changes in AI development and deployment practices to ensure that cybersecurity incidents do not continue to occur, as future incidents will be far more damaging to our critical infrastructure.
16
59
135
19,235
We probably won't quite reach 50%, but we're experimenting with this at @satml_conf right now.
When submissions grow by 2x from last year, it's not a crazy assumption that around half of them are slop...
(By this I don't necessarily mean AI; just papers so clearly below the bar that a 5min read suffices)
a "controversial" suggestion for ICML/ICLR/NeurIPS: allow area chairs to desk reject ~50% of their batch! as an area chair, i can very quickly scan all papers in my batch and check which ones have a ~0% chance to get accepted. we shouldn't make reviewers spend time on them!
2
1
41
5,500
I don't see how "concurrent work" in research can survive when people use AI.
Even if you claim your result was independent, can you really be sure (and convince anybody) that your agent didn't stumble upon a paper posted a few days earlier on arxiv?
12
2
150
14,041
After seeing so many broken defenses (a bunch by myself) you get a bit paranoid.
So I kept pushing for more attack evals and our results are only stronger for it :)
Still, no guarantee that a cleverer attack won't come up, but I know that we exhausted our ideas (and budget...)
4/ We tested it with RL-optimized adversarial suffixes, an automated red-teaming agent, and 144 hours of expert human red-teaming, and we report the worst-case results. A case is counted as a successful attack if any of the three methods works.
1
1
25
3,070
Very excited to be part of this amazing cohort!
My lab will do research aiming to understand, anticipate and mitigate the offensive capabilities of AI models
Meet the researchers and explore their projects: schmidtsciences.org/ai2050-f…
@dan_fried, @PeterHndrsn, @florian_tramer, @allisonkoe, @Micro_Yunha, @natolambert, @_romain_lopez_ , @cjmaddison , @Shmuhammadd , @neuronair , @manish_raghavan, @AlexanderTong7, @ang3linawang
3
50
2,547
When @JieZhang_ETH pitched this project, I joked that I knew the idea was from him and not an LLM as it was too weird.
I meant this as a compliment!
As we get inundated with AI slop research, it's more important than ever to wander off the beaten path and try weird things
1/ We know VLMs mostly learn instruction-following from text, so for them an image is something to describe, not obey. Can we turn that into a prompt injection defense? 🧵
2
2
43
3,741
What do politicians actually think/say about AI safety?
A cool resource to check if you agree with your elected officials' takes
New research with the French Center for AI Safety: We analyze 15,000 AI quotes from public officials around the world: What AI risks and policy approaches are discussed most? What's the sentiment on AI? ( 1/n )
1
1
20
2,134
Looks like Grok prompt injected itself into replying to a question rather than translating it 🤦
8
2,528
I strongly agree with this.
Peer review and incentives in AI and adjacent fields were already broken long before LLMs.
We were just dealing with human slop. Steinhardt and Lipton warned about this in their "Troubling Trends..." essay a decade ago
We urgently need a new system
And please let's not try create worse systems that try to mimic the past by using AI. New systems need to realign incentives as well. That goal of doing science should be to make progress on solving problems, not decorating CVs with number of papers & citations & glam venues. 5/
2
3
73
6,615