The "Pain Axis" paper has been getting a considerable amount of attention. I have reservations about several aspects of the paper and I don’t think we can take much away from it. In brief, since it was already essentially certain that some kind of “pain axis” exists, the key result from the paper is the claim that the identified pain axis “has some of the central functional properties of pain”. But I do not think the paper offers strong evidence for this. The first claim of the paper is the existence of a “pain axis”: LLM internal activations consistently differ when next-token-predicting painful experiences, as opposed to other negative experiences (like being afraid) or other sensations (like feeling the impulse to yawn). But we already know (e.g., from Anthropic’s “Emotion Concepts” paper) that it is possible to find directions in activation space that correspond to "current speaker is afraid", "current speaker is angry", etc., and contemporary LLMs are clearly sophisticated enough to also represent “current speaker is in pain”. As v2 of the paper acknowledges, nothing in these methods shows that there is any deeper sense in which the pain axis tracks “the model” being in pain, rather than a representation that the current speaker is in pain. The novel, stronger claim from the paper is that steering along the pain axis functions like pain in interesting ways. The first cited example is that steering along the pain axis causes expressions of worthlessness, which the paper describes as “consistent with the hypothesis that the models don’t treat pain paradigmatically as a physical state”. However, 80% of the test prompts used to generate the pain axis are cases of emotional pain (e.g., enduring repeated failure or social rejection) and only 20% are cases of physical pain. Accordingly, I think this can be viewed as a normal case of steering: If we shift model activations to be more like the activations associated with past expressions of emotional pain, this will lead to new expressions of emotional pain. The other case cited by the paper is that steering along the pain axis causes the assistant to “press” a button that will purportedly cause harm. However, it’s not clear to me why this is taken as evidence that the pain axis “has some of the central functional properties of pain”. The assistant does not seem to press the button in order to relieve its pain: As v2 of the paper reports, it is more likely to take harmful actions that it is told will increase its pain, rather than decrease its pain. And elsewhere the paper suggests that the opposite result (i.e., inaction) would also have been evidence for pain because becoming unmotivated is a signature of pain. So it seems like either outcome of this experiment could have been interpreted as evidence that steering along the pain axis functions like pain, making this particular outcome less compelling. Moreover, it seems like many things other than pain representations could produce these effects. In my own quick experiments, steering in random directions at the same strength causes just as much harmful button-pressing for around 25-30% of randomly selected directions. For these and other reasons, I don’t think the paper’s central findings give us much compelling new evidence about how current AI systems represent pain or whether current or future AI models experience pain.

i feel like at some point, the results get weird enough that I'm more like "how this vector is extracted is (imo) kinda weird [see pic], steering in general is messy, something's amiss" - than "this is a pain axis, but one with a very counterintuitive functional profile" eg...