The effect isn’t that insiders hold crazy beliefs, but a combination of vigorous debate inside a shared attention field, and a portfolio of individually reasonable beliefs that, taken together, lean in a direction convenient for the organization. Epistemic advantages and disadvantages Many people argue that low doses of psychedelics let you see some things more clearly - and I tend to agree: psychedelics can often help some people see some things more clearly. At the same time, I think microdosing usually doesn’t improve people’s epistemics overall, even when it gives them otherwise hard to access insights. You are not completely sober. Being inside a lab has real advantages: you know more about the frontier of capabilities, may have access to the latest AIs before the public, read internal Slack and blogs, and sense the vibes. If you’re outside and an insider tells you what they observe, pay attention. These information and insight asymmetries are tricky. I don’t want to advocate “discount insiders,” any more than “ignore what people with psychedelic experience say.” What I want is …interpret insiders as microdosing. Be curious but cautious. And if you’re inside: notice that your mind may be somewhat altered. Not like tripping (people inside are highly functional and would pass any simple test), but in subtler ways. Mechanisms and conditions Stretching the metaphor a bit: what is the substance? Mostly the “spirit of the organization,” in the sense of the organizational egregore or superagent. A simple insight from thinking about hierarchical agency is that superagents can be understood as having goals and preferences, including convergent instrumental goals. One common instrumental goal is synchronizing beliefs and goals across subagents. In my estimate, Anthropic has the strongest “spirit” of the frontier labs, which is why it’s in the title: it makes the cleanest case study. But I could have written about other companies, Constellation, the MIRI cluster, some startups… I think the substance is mostly delivered through entirely ordinary channels. Selection effects: who applies, who gets an offer, who stays, who leaves or gets pushed out. Simple, but powerful. Social gradients: everyday subtle messaging in the social web: what gets warm engagement at lunch, what gets Slack reactions, what gives you access to resources or powerful people. People naturally synchronize with those around them. Internal information: internal Slack, memos, memes, and the sense of having important secret information. Daniel Ellsberg famously described this when warning Kissinger about what security clearances would do to him: first, the new access makes you feel foolish about everything you had confidently written before; then, within a couple of years, it makes you unable to learn from people who lack the clearances. You stop hearing them. Whatever they say, you’re thinking about what they would say if they knew what you know. Frontier labs run on a somewhat similar asymmetry: I’d expect insiders to feel they have privileged information about model capabilities, growth rates, and internal forecasts. "But we argue internally all the time" In my experience, insiders often object: “But we argue internally all the time.” Internal debates are healthy, people push back on leadership, everyone has their own opinions. This is true in some ways: it is instrumentally necessary for the organization to have a lot of exploration. On the other hand, I would expect a lot of synchronization one meta-level up: where is attention allocated, which questions are important, which framings are simply in the water, and so on. The default I'd expect is vigorous object-level debate inside a synchronized attention field. This type of attentional redistribution and frame shifts is fairly difficult to notice from inside. Collective epistemic shifts If this model is roughly right, you shouldn’t expect insiders to hold any individually crazy beliefs. You should expect a portfolio of individually defensible probability weights whose conjunction is suspiciously aligned with the organization’s instrumental goals. To continue with Anthropic as the example: alignment is important (so the org’s existence is justified), and hard (so you should defer to the people working on it), and somewhat on track (so it’s scary but not suicidal to keep racing), and the race with China is real and must be won by the good guys (so it’s necessary to keep racing). Each of these is a sensible position that many people outside also hold. But I’d expect the joint distribution inside to be shifted toward the part of the space where the beliefs align with what the superagent wants. What it isn’t People sometimes unhelpfully collapse this to different problems: “Following the leaders”. Organizational spirits often do get personified in the form of the organization leaders, but I don't think this is very accurate or helpful; I don't think the leader-organization alignment to be one-directional, leaders do often serve the organizations in a deep sense, rather than leading the orgs. Anything with the word “cult”. What's characteristic of cults is a decrease in agency of the members - the cultists become in some sense more of “tools” or “organs” of the egregore. But this is not the case - people inside the labs often become more powerful, gain status and wealth and attention. What to do about it Probably worth emphasizing: nothing I've described depends on anyone acting in bad faith, lying about their experience and similar. So, is this even a problem? In my view, yes. I'm scared of the epistemic situation, I'm worried about the increasing number of people being inside, and I'm sad when people join and I notice shifts in how they talk and think. So, what should happen? Can this be counteracted by introspection and self-audits? I doubt so: my guess is when honest insiders inspect individual beliefs, they consider the arguments, the counterarguments, and conclude they are being reasonable. My model of the dominant frame among Anthropic higher-ups is “it's scary, but we are the responsible adults in the room”. There are various types of evil people outside, confused people outside, people who like to write blog posts and chase LW karma, but it's kind of pointless and will not save us. And there is a lot of truth in the perspective. But also you are microdosing. So what should actually happen? First, on the margin, more people should quit. Fortunately, this started happening between when I began writing this in August and when I posted it in October. Too large a part of AI safety, alignment, and macrostrategy is inside, and too few people are quitting, in particular in Anthropic's case. Possibly you should quit: not because of arguments that working at labs is morally wrong, but because you’re in an altered state that makes the tradeoffs hard to see. If quitting feels too drastic, at least make the decision from a less dosed state: take a break from the organization and its social web, and see how the tradeoffs look from there. Second, not everyone who could join should join. Again, too much of AI safety is already inside the labs. Every marginal hire converts one more potentially sober observer into a dosed one. Third, for those who stay: discount outsiders less. I think you cannot undo the salience field by introspection, but you can partially counteract the Ellsberg effect. Is anyone “really sober”? Here comes another obvious objection: is anyone really sober? I don’t claim that. I’m obviously also influenced by various incentive fields, including LessWrong, academia, and the flavor of AI safety thinking usually done at nonprofits. My approach is closer to hedging incentives than to trying to live in an incentive-free landscape: no matter what I say, someone will be upset, so the easy option is to speak relatively freely and not worry about who. I’m not claiming you should aim for an incentive vacuum or a meme-free space, or move to a cave and study poetry. What I do claim is that the spirits inside are strong, the effects depend on dose, and some combination of sobriety, sanity, and situational awareness is becoming something like a scarce civilizational resource.

from lesswrong.com/posts/9GyniEBa… with @rosehadshar

Sorry forgot to mention @robinhanson