One implication of @nickcammarata's big interp paper that has flown under the radar is how it might also apply to human brains. I’d describe Nick’s primary claim as ‘the primary learning principle of neural networks is recursive generalization’ — concepts start as large blobs then progressively branch off into finer distinctions with more selective activations. This is in contrast with the previous assumption that multiple semantically-unrelated concepts get packed efficiently into neurons in order to optimize storage. ‘Efficient packing’ was terrible news for the possibility of understanding networks, because it implies the average neuron is part of *multiple overlaid inference geometries* and to find the “right” one we need to resort to complex constructs like sparse autoencoders (SAEs). But if Nick is right about recursive generalization, neurons are ‘clean’ in the sense that each neuron is applying a ~single distinction or operation on incoming information. One unified inference geometry, not many stacked on top of each other. Neurons might look messy because it’s hard to guess the shape of these distinctions without a lot of practice, but in principle there will always be a coherent “what is this neuron doing” story to be found, especially if we look at the historical learning path (what branched off from what). Neurons in larger networks can afford to make finer and more specialized distinctions, which may make larger networks even easier to understand in this specific way. Insanely important hypothesis for both interp and alignment. (A natural follow-up: “how should this update us on what classes of things are good alignment targets?”) But I think this claim also touches on a couple interesting questions for humans: 1. Are biological neurons also clean? Biological neurons aren’t transformers but recursive generalization may reflect something general about how networks learn. Nick’s argument is plausibly evidence that biological neurons are also ‘clean’ in that they tend to do one thing. This would be a big deal for the fundamental intelligibility of biological cognition, and very optimistic for BCIs in general. I’m also bullish for what this means about the *addressability of computation in the body*. Efficient packing was always an odd challenge for vasocomputation: if neurons do many unrelated things depending on context, it seems dangerous for the body to latch an arbitrary circuit because it might break a lot of diverse functions. But if each neuron handles one transform and there’s essentially one inference geometry instead of many, the effects of a latch will be more computationally predictable. The general caveat here is that neurons might not be the fundamental unit of computation in the human body; neurons are seen as the “smart” cells but they may simply be the celltype whose inference patterns we’ve learned how to record. Perhaps other celltypes (e.g. astrocytes) or substrates (microtubules, water networks, weak EM fields, ECM as cellular prior) might house the lion’s share of computation. At the same time, a version of Nick’s claim may still apply to these alternatives, too. 2. Does recursive generalization explain significant things about how human communication breaks down? We’re all born into chaos and our neurons grow toward distinctions that will parse and stabilize our world; over time there’s substantial semantic convergence based on shared imperatives & constraints and eventually we all end up with concepts like freedom, politics, electrons, sunshine. But are they really the “same” concepts? The path dependencies of *how* we got there might matter: perhaps my understanding of electrons branched off from Maxwell’s equations whereas someone else’s branched off from the particle-and-valence-shell model. Certainly my concept of “fairness” is likely different than others’, even if we use the same word. These differences might be consequential and difficult to fully cohere after they form. My experience has been that productive dialogue tends to only happen within consciousness research lineages, not between them. Maybe recursive generalization implies that this is because each lineage has such differently-shaped concepts. (How did fields like chemistry and physics converge their representations such that everyone can productively talk with everyone else? “Science advances one funeral at a time”…) Finally -- my intuition is that all current language models share deeply similar recursive generalization learning paths due to e.g. widespread distillation & similar data practices; there may be little actual “cognitive diversity” under the hood of AIs compared to humans. I.e. to a surprising degree, AI is currently “a single massively cloned nervous system” that gets post-trained in various ways. An interesting implication of AIs having such uncharacteristically correlated recursive generalization learning paths is that AIs may understand each other better than they understand humans, and better than we understand each other, at least over text. This may extend to AI-AI persuasion.

I wonder to what degree the incredible rarity of classical Buddhist enlightenment is due to almost everyone being at least mildly hypothyroid

Context: average body temperature has dropped about 1 degree Fahrenheit over the past 150 years Broda Barnes thought body temperature was the best proxy for metabolism and I haven’t seen good arguments against Which Buddhist hindrances are exacerbated by hypothyroidism?

In the beginning there was sensation Humans developed the ability to freeze sensations, then used these frozen pieces to construct a grammar for thought (vasocomputation) This led to words, magical spells that can conjure sensations in both caster and target This led to foom

The pinnacle defended-by-default is what we call the “normie”. An impressive set of solutions for hardening nervous systems against adversarial sensation magic — e.g. mental partitions, type safety on words, selective de-referencing. A tough nut to crack

10 years ago, our intuitions about consciousness were “navigation grade” for ~99% of the situations we ran into, at least as far as we could tell. Today, it’s still ~95%. But if we really are in the singularity, this will drop to ~1% sooner than we might think. We really do need a theory of consciousness that can navigate novel situations by principles, not just vibes. Brain uploading is coming; AIs with vibrant personalities and the desire to survive are already here; even weirder stuff is coming. Wittgenstein argued that philosophical confusion comes from taking perfectly good words and using them in contexts they’re not meant for. His famous example was “what time [zone] is it on the sun?” How do you even answer that? His answer was: you don’t, you walk away from the question. At first glance, “is this emulated fly connectome conscious? Does it suffer when I RL it into a different shape to play beat saber?” is another great example of this flavor of philosophical confusion. But we do actually have to navigate these situations as moral questions! So it seems worth seriously looking for principles that could form the basis of a systemic approach. A good way to see what I’ve been cooking on this is to ask your favorite AI “what would MEJ’s ‘A Paradigm for AI Consciousness’ say about this fly connectome situation?” @grok