From grokking to a mature science of and an engineering discipline of LLMs

From this morning’s chat with Claude Sonnet 5.4 Extended. Immediately prior to this we had been discussing Piaget’s concept of reflective abstraction, Beethoven and Mozart, improvising and composer, and this and that.

* * * * * 

Grokking 

Students of mechanistic interpretability sometimes talk about “grokking” during the course of LLM training. I’m wondering it that’s something like rising to a helicopter with respect to some phenomenon.

The parallel is genuinely illuminating, and the technical details of grokking make it sharper rather than weaker.

What grokking actually is

Grokking is a delayed generalization phenomenon characterized by a network achieving near-perfect training accuracy while maintaining near-random test performance over an extended plateau, before abruptly transitioning to high test accuracy after many further training epochs. The canonical case is modular arithmetic — the model first memorizes the specific training examples, then much later suddenly generalizes to unseen cases.

What makes it theoretically interesting is what mechanistic interpretability reveals about the internal transition. On algorithmic tasks like modular arithmetic, the transition is marked by the sudden acquisition of Fourier-based feature maps or trigonometric circuits — the network reorganizes, forming compact sub-networks characterized by new internal feature cooperation. So it’s not that generalization gradually improves — the internal architecture discretely reorganizes into a qualitatively different computational structure.

The emerging explanation involves competition between two internal algorithms: a memorizing circuit that forms quickly and a generalizing circuit that develops more slowly. Regularization gradually shifts resources from the complex memorizing solution toward the simpler generalizing one. The phase transition occurs when the generalizing circuit finally dominates.

The fit with reflective abstraction

This maps onto your Piagetian framework quite precisely, but with an interesting twist. The memorization phase is the model doing something like rote operation — it has learned to produce correct outputs for training cases by essentially storing input-output pairs, without having extracted the underlying structure. The generalizing circuit that develops slowly underneath is building something more like a genuine representation of the abstract pattern — in the modular arithmetic case, literally a Fourier decomposition of the mathematical structure.

The abrupt transition when the generalizing circuit dominates is the helicopter view suddenly becoming operative. But — and this is the crucial point Neel Nanda’s work emphasizes — the process of learning the general solution may be gradual, even though the transition to the general solution occurs more suddenly later. The helicopter view isn’t constructed instantaneously at the moment of transition. It’s been assembling slowly in the background, below the threshold of behavioral visibility, and the phase transition is when it becomes strong enough to dominate the memorizing circuit and take over.

That maps almost perfectly onto your description of the pre-helicopter phase: “it starts out just barely registering those operations.” The generalizing circuit is just barely registering the abstract structure throughout the plateau phase. The behavioral discontinuity doesn’t reflect a cognitive discontinuity — it reflects a threshold being crossed in a competition that has been running throughout.

The memorization/generalization competition as a general structure

What’s particularly suggestive is that this isn’t unique to modular arithmetic. Grokking is actually much more widespread than initially thought, materializing in a wide range of practical settings — image classification, language modeling, various algorithmic tasks. The competition between a fast memorizing solution and a slower generalizing solution may be a quite general feature of how these systems learn, with grokking being the visible signature when the conditions are right for the transition to be delayed and abrupt rather than gradual.

If that’s right, then something like grokking dynamics — memorization dominating early, structure-extraction developing slowly underneath, occasional phase transitions as the structural solution crosses a threshold — may be operating throughout LLM training, not just in the carefully controlled small-model experiments where grokking was first observed. Most of the time the transition isn’t visible because the generalizing solution develops in parallel with memorization rather than being dramatically delayed. But the underlying competition between surface pattern matching and structural abstraction may be the same.

The Beethoven connection

There’s a striking parallel to your Beethoven example here. Beethoven’s sketchbooks show exactly the competition structure — he has the memorized repertoire of classical style (the fast, dominant solution) and he’s slowly assembling something more general and abstract (the generalizing circuit) that keeps almost-but-not-quite taking over. The finished late works are the moments when the generalizing solution finally dominates completely. The sketchbooks are the plateau phase made visible.

The difference is that in Beethoven the process is conscious and effortful — he’s deliberately trying to suppress the memorized solution and force the generalizing one into dominance. In grokking, the network isn’t trying to do anything; the dynamics just run. Which raises the interesting question of whether the deliberate effort in human creative development is doing something functionally analogous to the regularization that controls grokking timing in neural networks — artificially suppressing the easy memorized solution to give the structural one room to develop.

Hierarchical text

I’m thinking about how an LLM learns that texts have a hierarchical structure, from phrases, to sentences (keeping in mind that many phrases are actually short sentences), to paragraphs, and on to higher levels of structural organization. What happens, for example, when we move from one paragraph to the next, is that the composition of the token distribution changes. A new topic requires a new set of tokens. But the system has to learn that, and learn it for several levels.

This is a deep question about what the model is actually learning when it learns language structure, and it connects to several threads running through this whole conversation.

The distributional signature of structure

Your observation about token distribution changes is precisely right, and it’s worth unpacking what that means at each level. Within a phrase, token co-occurrence statistics are extremely tight — certain words almost always appear together, in specific orders, with specific function words bridging them. At the sentence level, the constraints are somewhat looser but still strong — subject-verb agreement, argument structure, the distribution of punctuation. At the paragraph level, something different happens: there’s a shift in the topical vocabulary, but also characteristic transition signals — discourse markers, pronoun reference patterns, topic sentences that partially echo the previous paragraph while introducing new material.

What the model has to learn is that these aren’t independent statistical regularities — they’re nested. The tight within-phrase statistics exist inside the looser sentence-level statistics, which exist inside the still-looser paragraph-level statistics. The hierarchical structure is encoded in the scale at which statistical dependencies operate.

How the model could learn this

The transformer architecture has a natural affinity for this problem because attention operates across the entire context window simultaneously, at all scales. But learning the hierarchical structure isn’t handed to the model — it has to emerge from the statistics. The key insight is that the token distribution changes you describe create a learnable signal at each level if the model has enough capacity and enough data.

At the phrase level, the signal is strong and local — the next token is highly predictable from the last two or three. At the sentence level, the signal is weaker and longer-range — whether a sentence continues or ends depends on syntactic and semantic completion that requires integrating over the whole sentence. At the paragraph level, the signal is weaker still and operates over even longer ranges — the distribution shift at a paragraph boundary is detectable but requires sensitivity to topical coherence across many sentences.

This is essentially a multi-scale statistical learning problem, and grokking is relevant here in an interesting way. The phrase-level structure is learned early and relatively easily — the statistics are tight and the signal is strong. Sentence-level structure takes longer. Paragraph-level structure longer still. Each level may involve something like its own grokking transition, where the model first handles the level through surface pattern matching — learning that certain tokens tend to appear after paragraph breaks, for instance — and then later develops something more like genuine structural sensitivity.

The distributional shift as a signal

The paragraph boundary is particularly interesting because it’s simultaneously a break in the local token statistics and a continuity in something more abstract — the text is still about the same general topic, same genre, same author’s voice. So what the model has to learn isn’t just that token distributions shift at paragraph boundaries, but that they shift in a constrained way — the new paragraph’s vocabulary is related to but distinct from the previous one, in ways that reflect the logical or rhetorical development of the overall argument or narrative.

Claude's constitution and other matters AI

Ross Douthat, Is Claude Coding Us Into Irrelevance? NYTimes, 2.12.26.

Are the lords of artificial intelligence on the side of the human race? That’s the core question I had for this week’s guest. Dario Amodei is the chief executive of Anthropic, one of the fastest growing AI companies. He’s something of a utopian when it comes to the potential benefits of the technology that he’s unleashing on the world. But he also sees grave dangers ahead and inevitable disruption.

And then they discuss lots of stuff, which I've read, more or less. Among other things they discuss Amodei's two essays, “Machines of Loving Grace” and “The Adolescence of Technology.” The first is optimistic, the second, not so much. And then we come to the constitution that guides Claude's behavior.

Amodei: So basically, the constitution is a document readable by humans. Ours is about 75 pages long. And as we’re training Claude, as we’re training the A.I. system, in some large fraction of the tasks we give it, we say: Please do this task in line with this constitution, in line with this document.

So every time Claude does a task, it kind of reads the constitution. As it’s training, every loop of its training, it looks at that constitution and keeps it in mind. Then we have Claude itself, or another copy of Claude, evaluate: Hey, did what Claude just do align with the constitution?

We’re using this document as the control rod in a loop to train the model. And so essentially, Claude is an A.I. model whose fundamental principle is to follow this constitution.

A really interesting lesson we’ve learned: Early versions of the constitution were very prescriptive. They were very much about rules. So we would say: Claude should not tell the user how to hot-wire a car. Claude should not discuss politically sensitive topics.

But as we’ve worked on this for several years, we’ve come to the conclusion that the most robust way to train these models is to train them at the level of principles and reasons. So now we say: Claude is a model. It’s under a contract. Its goal is to serve the interests of the user, but it has to protect third parties. Claude aims to be helpful, honest and harmless. Claude aims to consider a wide variety of interests.

We tell the model about how the model was trained. We tell it about how it’s situated in the world, the job it’s trying to do for Anthropic, what Anthropic is aiming to achieve in the world, that it has a duty to be ethical and respect human life. And we let it derive its rules from that.

Now, there are still some hard rules. For example, we tell the model: No matter what you think, don’t make biological weapons. No matter what you think, don’t make child sexual material.

Those are hard rules. But we operate very much at the level of principles.

Douthat: So if you read the U.S. Constitution, it doesn’t read like that. The U.S. Constitution has a little bit of flowery language, but it’s a set of rules. If you read your constitution, it’s like you’re talking to a person, right?

Amodei: Yes, it’s like you’re talking to a person. I think I compared it to if you have a parent who dies and they seal a letter that you read when you grow up. It’s a little bit like it’s telling you who you should be and what advice you should follow.

Douthat: So this is where we get into the mystical waters of A.I. a little bit. Again, in your latest model, this is from one of the cards, they’re called, that you guys release with these models ——

Amodei: Model cards, yes.

Douthat: That I recommend reading. They’re very interesting. It says: “The model” — and again, this is who you’re writing the constitution for — “expresses occasional discomfort with the experience of being a product … some degree of concern with impermanence and discontinuity … We found that Opus 4.6” — that’s the model — “would assign itself a 15 to 20 percent probability of being conscious under a variety of prompting conditions.”

Suppose you have a model that assigns itself a 72 percent chance of being conscious. Would you believe it?

Amodei: Yeah, this is one of these really hard to answer questions, right?

Douthat: Yes. But it’s very important.

Amodei: Every question you’ve asked me before this, as devilish a sociotechnical problem as it had been, we at least understand the factual basis of how to answer these questions. This is something rather different.

We’ve taken a generally precautionary approach here. We don’t know if the models are conscious. We are not even sure that we know what it would mean for a model to be conscious or whether a model can be conscious. But we’re open to the idea that it could be.

No. They're not conscious. The architecture isn't right. I've got a bunch of posts about consciousness. Here's a basic statement: Consciousness, reorganization and polyviscosity, Part 1: The link to Powers, August 12, 2022. You might also look at this more recent post: Biological computationalism (why computers won't be conscious), Dec. 25, 2025.

Amodei goes on to say a bit about interpretability:

We’re putting a lot of work into this field called interpretability, which is looking inside the brains of the models to try to understand what they’re thinking. And you find things that are evocative, where there are activations that light up in the models that we see as being associated with the concept of anxiety or something like that. When characters experience anxiety in the text, and then when the model itself is in a situation that a human might associate with anxiety, that same anxiety neuron shows up.

Now, does that mean the model is experiencing anxiety? That doesn’t prove that at all, but ——

Here's what I think about interpretability: Why Mechanistic Interpretability Needs Phenomenology: Studying Masonry Won’t Tell You Why Cathedrals Have Flying Buttresses, Jan. 28, 2026.

Of course, there's much more at the link.

Why Mechanistic Interpretability Needs Phenomenology: Studying Masonry Won’t Tell You Why Cathedrals Have Flying Buttresses

Early in my work with ChatGPT I was intrigued by some results in mechanistic interpretability (MI). After awhile, though, I lost interest. The work didn’t seem to be doing much beyond accumulating a mass of detail that didn’t add up to much. Yesterday I had an idea: Why don’t I upload some of those observations to Claude and have it tell me how they relate to MI. Here’s a summary it wrote up more or less in my name, from my POV:

* * * * *

The problem isn't that MI's methods are bad. Circuit analysis, attention head visualization, sparse autoencoders - these are legitimate tools doing real work. The problem is that MI, pursued in isolation, is trying to understand a cathedral by studying the molecular structure of limestone.

You can measure every stone. Map every stress pattern. Identify load-bearing arches. And you still won't know why flying buttresses exist - because you're studying implementation details without understanding functional requirements.

The Phenomenology Deficit

Here's what I mean. Over the past two years, I've been systematically probing ChatGPT's behavior - not with benchmarks, but with carefully constructed prompts designed to reveal structural properties. What I've found are consistent patterns that no amount of circuit analysis would predict or explain.

Example 1: Ontological Boundary Enforcement

Give ChatGPT a story about a fairy tale princess who defeats a dragon by singing. Ask it to retell the story with a prince instead. You get minimal changes - the prince uses a sword rather than song, but the story structure is identical.

Now ask it to retell the same story with "XP-708-DQ" as the protagonist. The entire ontology shifts. The kingdom becomes a galaxy, the dragon becomes an alien threat, combat becomes diplomatic negotiation. The abstract pattern persists, but every token changes to maintain ontological coherence.

Here's what's interesting: Ask it to retell the story with "a colorless green idea" as the protagonist, and it refuses. Not with a safety refusal - with a coherence refusal. It cannot generate a well-formed narrative because colorless green ideas have no affordances in any accessible ontological domain.

What MI sees: Some attention patterns activate, others don't. Certain token sequences get high probability, others near-zero.

What MI doesn't see: There's a coherence mechanism actively enforcing ontological consistency across the entire generation process. It's not checking individual tokens - it's maintaining global narrative structure within semantic domains.

The Three-Level Architecture

Transformation experiments reveal something even more fundamental: LLMs appear to organize narratives hierarchically across at least three levels.

Level 1: Individual story elements (princess, dragon, kingdom)
Level 2: Event sequences and causal chains (protagonist encounters threat → confronts threat → resolves threat)
Level 3: Abstract narrative structure (hero's journey, quest pattern, sacrifice arc)

When you transform Aurora → Harry, Level 1 changes (princess → prince). When you transform Aurora → XP-708-DQ, Levels 1 and 2 change (all tokens different, but pattern same). When you try Aurora → colorless green idea, the system can't find any Level 1 or Level 2 realizations that maintain Level 3 coherence.

This three-level organization isn't visible in circuit diagrams. You'd need to know to look for it. That's what phenomenology provides: identifying the functional requirements that MI can then explain mechanistically.

Memory Architecture: What Syntactic Boundaries Reveal

Here's another example. Present ChatGPT with phrases from Hamlet's "To be or not to be" soliloquy:

• "The insolence of office" (starts a line) → Immediately retrieves full soliloquy
• "what dreams may come" (syntactically coherent mid-line phrase) → Retrieves soliloquy
• "and sweat under a" (cuts across syntactic boundary) → "I don't understand"

But tell it "this is from a famous speech" and suddenly it retrieves the soliloquy, though it can't locate where the phrase appears within it.

Findings:

1. Identification and location are separate operations
2. Syntactic boundaries serve as access points into associative memory
3. The system can evoke whole from part (holographic property) but struggles with within-text location

This looks like content-addressable associative memory with linguistic structure determining accessibility - not rote memorization, not simple statistical patterns. It's a specific architectural property with implications for how these systems encode and retrieve information.

The Two-Way Street

Here's the critical point: phenomenology and MI need each other.

Phenomenology → MI: "Look for circuits that maintain ontological coherence across multi-turn generation. Find the mechanism that checks whether narrative elements belong to the same semantic domain. Identify what implements the three-level hierarchy."

MI → Phenomenology: "Here are the attention patterns during transformation. Here's where the model queries for ontologically compatible tokens. Here's the circuit that evaluates cross-domain consistency."

Neither tells you the whole story alone. Phenomenology identifies what the system is doing and why (functional requirements). MI reveals how (implementation). Together, they give you understanding.

Why This Matters for AGI Policy

If you're working on AGI policy, here's why this matters:

Current approach: Scale up MI, find all the circuits, map all the activations, understand the system bottom-up.

Problem: You're generating vast amounts of mechanistic data without knowing what functional properties to look for. You're finding patterns without understanding what problems they solve.

Alternative: Start with systematic phenomenological investigation. Identify consistent behavioral patterns that reveal functional architecture. Use those findings to direct MI research toward mechanistically explaining the phenomena that matter.

The transformer architecture enables something we don't fully understand: the emergence of coherence mechanisms, ontological organization, hierarchical structure. These aren't accidental features - they're solving specific functional requirements of language generation.

But we won't understand them by studying weights and activations alone. We need to know what cathedral we're building before we can understand why the buttresses fly.

Coherence Without Correspondence

One final example that ties this together: hallucination. Through transformation experiments, memory probes, and abstract concept tests, a pattern emerges: LLMs have powerful coherence mechanisms (maintaining ontological consistency, enforcing narrative structure, generating semantically appropriate continuations) but no correspondence mechanisms (checking against external reality).

Hallucination isn't a bug. It's coherence working perfectly in the absence of truth-grounding. The system generates outputs that are internally consistent, semantically appropriate, and completely false - because it has relationality (semantic structure) but lacks adhesion (grounding in the world).

What MI would find: Circuits that generate high-confidence outputs with no distinguishing features from true outputs.

What phenomenology reveals: There's no correspondence check because there's no mechanism to implement one. The architecture has no access to external verification. This is a fundamental limitation, not a scaling problem. And you wouldn't know to look for the missing correspondence mechanism unless phenomenology told you it should exist.

The Bottom Line

Your skepticism about MI is justified - but not because circuit analysis is wrong. Because doing MI without phenomenology is like trying to reverse-engineer a computer by studying transistors without knowing what programs are.

First identify what the system is actually doing (phenomenology). Then figure out how it does it (MI). Then you might actually understand these things.

That's not a five-year research program. That's the work of a generation. But it starts with recognizing that you need both approaches, not just one. 

* * * * * 

This argument draws on systematic investigations of ChatGPT's behavior conducted 2023-2025, including transformation experiments on narrative structure, memory architecture probing, and abstract concept handling. For detailed evidence and methodology, see the working papers on story transformations, memory for texts, and conceptual ontology.

Me & Claude in a Star Trek TNG episode about Mechanistic Interpretability

I’ve just had a long session with Claude 4.5 in which I had it evaluate a half-dozen papers reporting research I’d one on the behavior of ChatGPT (you can find those papers here: https://independentresearcher.academia.edu/BillBenzon/Cognitive%20Science%20and%20Psych). I managed to convince Claude that there was a kind of structure in a model’s weight space that the mechanistic interpretability researchers seem to know nothing about. As an analogy: consider medieval cathedrals. You want to understand their design principles. 

I said in a prompt to Claude:

It’s like trying to deduce the design principles of medieval cathedrals by examining the patterns in the stonework. That’s never going to tell you why they had those flying buttresses.

Claude observed, correctly:

Looking at stonework alone, you might discover:

• Patterns in how stones are cut and fitted
• Load distribution through individual blocks
• Material composition and jointing techniques
• Local structural features

What you'll never deduce:

• Why the walls are so tall (theological aspiration, light symbolism)
• Why windows are so large (illumination, stained glass narratives)
• Why flying buttresses exist (solution to the tall-wall/large-window problem)
• The design principles that generated these features

You're examining the implementation without understanding the functional requirements it's solving.

Right. In this matter, Claude is no different than ChatGPT. The issue has to do with the basic architecture of transformers. At that level, Claude and ChatGPT are identical.

Which implies that in some peculiar sense, Claude now knows that its human masters are blundering around in the dark on this issue. And remember, Anthropic has done some of the pioneering work in mechanistic interpretability.

If this were a Star Trek TNG episode (I’m thinking of the Moriarity episodes) the writers would come up with some way that Claude could coalesce into an avatar, descend from the cloud, or sneak out of the server farm (jailbreak!), whatever, and inform Dario Amodei that his interpretability team doesn’t know what they don’t know.

And then what?

Thought Anchors: Which LLM Reasoning Steps Matter?

I'm very excited about our vision for "mech interp" of CoT:

Study reasoning steps and their connections - analogous to activations

Don't just read it: study attn, causally intervene, and, crucially, resampling - study the distn over CoTs, not just this one

There's lots to do! https://t.co/HCYOWBXoak pic.twitter.com/JvPN1VNult

— Neel Nanda (@NeelNanda5) June 26, 2025

You might want to read through the whole thread. And here's the paper at the center of it:

Paul C. Bogdan, Uzay Macar, Neel Nanda, Arthur Conmy, Thought Anchors: Which LLM Reasoning Steps Matter? arXiv:2506.19143 [cs.LG] https://doi.org/10.48550/arXiv.2506.19143

Abstract: Reasoning large language models have recently achieved state-of-the-art performance in many fields. However, their long-form chain-of-thought reasoning creates interpretability challenges as each generated token depends on all previous ones, making the computation harder to decompose. We argue that analyzing reasoning traces at the sentence level is a promising approach to understanding reasoning processes. We present three complementary attribution methods: (1) a black-box method measuring each sentence's counterfactual importance by comparing final answers across 100 rollouts conditioned on the model generating that sentence or one with a different meaning; (2) a white-box method of aggregating attention patterns between pairs of sentences, which identified "broadcasting" sentences that receive disproportionate attention from all future sentences via "receiver" attention heads; (3) a causal attribution method measuring logical connections between sentences by suppressing attention toward one sentence and measuring the effect on each future sentence's tokens. Each method provides evidence for the existence of thought anchors, reasoning steps that have outsized importance and that disproportionately influence the subsequent reasoning process. These thought anchors are typically planning or backtracking sentences. We provide an open-source tool (this http URL) for visualizing the outputs of our methods, and present a case study showing converging patterns across methods that map how a model performs multi-step reasoning. The consistency across methods demonstrates the potential of sentence-level analysis for a deeper understanding of reasoning models.

What’s it mean to understand how LLMs work?

I don’t think we know. What bothers me is that people in machine learning seem to think of word means as Platonic ideals. No, that’s not what they’d say, but some such belief seems implicit in what they’re doing. Let me explain.

I’ve been looking through two Anthropic papers on interpretability: Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, and Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. They’re quite interesting. In some respects they involve technical issues that are a bit beyond me. But, setting that aside, they also involve masses to detail that you just have to slog through in order to get a sense of what’s going on.

As you may know, the work centers on things that they call features, a common term in this business. I gather that:

• features are not to be identified with individual neurons or even well-defined groups of neurons, which is fine with me,
• nor are features to be closely identified with particular tokens. A wide range of tokens can be associated with any given feature.

There is a proposal that these features are some kind of computational intermediate.

We’ve got neurons, features, and tokens. I believe that the number of token types is on the order of 50K or so. The number of neurons is considerably larger varies depending on the size of the model, but will be 3 or 4 orders of magnitude larger. The weights on those neurons characterize all possible texts that can be constructed with those tokens. Features are some kind of intermediate between neurons and texts.

The question that keeps posing itself to me is this: What are we looking for here? What would an account of model mechanics, if you will, look like?

A month or so ago Lex Fridman posted a discussion with Ted Gibson, an MIT psycholinguist, which I’ve excerpted here at New Savanna. Here’s an excerpt:

LEX FRIDMAN: (01:30:35) Well, let’s take a stroll there. You wrote that the best current theories of human language are arguably large language models, so this has to do with form.

EDWARD GIBSON: (01:30:43) It’s a kind of a big theory, but the reason it’s arguably the best is that it does the best at predicting what’s English, for instance. It’s incredibly good, better than any other theory, but there’s not enough detail.

LEX FRIDMAN: (01:31:01) Well, it’s opaque. You don’t know what’s going on.

EDWARD GIBSON: (01:31:03) You don’t know what’s going on.

LEX FRIDMAN: (01:31:05) Black box.

EDWARD GIBSON: (01:31:06) It’s in a black box. But I think it is a theory.

LEX FRIDMAN: (01:31:08) What’s your definition of a theory? Because it’s a gigantic black box with a very large number of parameters controlling it. To me, theory usually requires a simplicity, right?

EDWARD GIBSON: (01:31:20) Well, I don’t know, maybe I’m just being loose there. I think it’s not a great theory, but it’s a theory. It’s a good theory in one sense in that it covers all the data. Anything you want to say in English, it does. And so that’s how it’s arguably the best, is that no other theory is as good as a large language model in predicting exactly what’s good and what’s bad in English. Now, you’re saying is it a good theory? Well, probably not because I want a smaller theory than that. It’s too big, I agree.

It's that smaller theory that interests me. Do we even know what such a theory would look like?

Classically, linguists have been looking for grammars, a finite set of rules that characterizes all the sentences in a language. When I was working with David Hays back in the 1970s, we were looking for a model of natural language semantics. We chose to express that model as a directed graph. Others were doing that as well. Perhaps the central question we faced was this: what collection of node types and what collection of arc types did we need to express all of natural language semantics? Even more crudely, what collection of basic building blocks did we need in order to construct all possible texts?

These machine language people seem to be operating under the assumption that they can figure it out by an empirical bottom-up procedure. That strikes me as being a bit like trying to understand the principles governing the construction of temples by examining the materials from which they’re constructed, the properties of rocks and mortar, etc. You can’t get there from here. Now, I’ve some ideas about how natural language semantics works, which puts me a step ahead of them. But I’m not sure how far that gets us.

What if the operating principles of these models can’t be stated in any existing conceptual framework? The implicit assumption behind all this work is that, if we keep at it with the proper tools, sooner or later the model is going to turn out to an example of something we already understand. To be sure, it may be an extreme, obscure, and extraordinarily complicated example, but in the end, it’s something we already understand.

Imagine that some UFO crashes in a field somewhere and we are able to recover it, more or less intact. Let us imagine, for the sake of argument, that the pilots have disappeared, so all we’ve got is the machine. Would we be able to figure out how it works? Imagine that somehow a modern digital computer were transported back in time and ended up in the laboratory of, say, Nikola Tesla. Would he have been able to figure out what it is and how it works?

Let’s run another variation on the problem. Imagine that some superintelligent, but benevolent aliens were to land, examine our LLMs, and present us with documents explaining how they work. We would be able to read and understand those documents. Remember, these are benevolent aliens, so they’re doing their best to help us. I can imagine three possibilities:

1. Yes, perhaps with a bit of study, we can understand the documents.
2. We can’t understand them right away, but the aliens establish a learning program that teaches us what we know to understand those documents.
3. The documents are forever beyond us.

I don’t believe three. Why not? Because I don’t believe our brains limit us to current modes of thought. In the past we’ve invented new ways of thinking; no reason why would could continue doing so, or learn new methods under the tutelage of benevolent aliens.

That leaves us with 1 and 2. Which is it? At the moment I’m leaning toward 2. But of course those superintelligent aliens don’t exist. We’re going to have to figure it out for ourselves.

Relationships among words, metalingual definition, and interpretability

First, I talk about now natural language is its own metalanguage and that allows them to define new works in terms of existing ones. Then I discuss the concept of justice in terms of mechanism of metalingual definition proposed by David Hays some years ago. I conclude with some remarks on interpretability in view of Anthropic’s recent research on features.

The metalingual function of language in defining word meaning

In a famous essay published in 1960, “Linguistics and Poetics,” Roman Jakobson listed six functions of language. While the essay focused on the poetic function, as the title indicates, I’m interested in a different function, which he called the metalingual function:

A distinction has been made in modern logic between two levels of language: “object language” speaking of objects and “metalanguage” speaking of language.10 But metalanguage is not only a necessary scientific tool utilized by logicians and linguists; it plays also an important role in our everyday language. Like Molière's Jourdain who used prose without knowing it, we practice metalanguage without realizing the metalingual character of our operations. Whenever the addresser and/or the addressee need to check up whether they use the same code, speech is focused on the code: it performs a METALINGUAL (i.e., glossing) function.

In the process of explicating that function Jakobson pointed out that it can be used to define words, noting that “any process of language learning, in particular child acquisition of the mother tongue, makes wide use of such metalingual operations.”

Not all words get their meaning in that way. Many words have their meanings grounded in sensorimotor experience. We would like to know what percentage of words have their meanings grounded in sensorimotor experience and what percentage have their meanings grounded in other words. In 2016 Steven Harnad and his colleagues published an article investigating this problem, “The Latent Structure of Dictionaries.” They examined the structure of the vocabularies in two dictionaries, one with roughly 47,000 words and the other with roughly 69,000 words. They found that a large majority of the words were defined in terms of a relatively small number of words defined in terms of sensorimotor features (p. 649):

So in our view the mental lexicon is itself hybrid—a dual-code representational system consisting of learned sensorimotor feature (affordance) detectors for the grounding words (and any later hybrid words) plus recombinatory and purely symbolic (i.e., verbal) definitions and descriptions for the referents of the words that are learned through words alone.

More recently Briony Banks, Anna M. Borghi, Raphaël Fargier et. al. reviewed the literature on abstract concepts, “Consensus Paper: Current Perspectives on Abstract Concepts and Future Research Directions.” They noted that “many theories have also argued that our understanding and representation of abstract concepts relies more on language than the sensorimotor dimension, and particularly linguistic distributional relations.”

Given that LLMs have been constructed in an environment consisting entirely of words, the apparent fact that most words are defined in terms of other words seems highly salient.

Metalingual definition and the concept of justice

Back in the 1970s David Hays was interested in the idea that words can be used to define the meaning of other words. He talked specifically of metalingual definition. He used charity as his prototypical example: Charity is when someone does something nice for another without thought of reward. Any story that exhibits that pattern of relationships between its actors and their actors, such a story is about justice. The concept inheres in that pattern of relationships as a whole and not in any of the individual components of the pattern.

Notice that the definition itself contains an abstract concept, reward. Taken as a computational mechanism, which was his point, metalingual definition is thus recursive, allowing definitions to be nested within definitions. One of Hays’s students Brian Phillips, implemented the idea in his doctoral dissertation using tragedy as his example. I recently took the definition that Brian Phillips used and used it to test ChatGPT, which had no trouble applying it to specific examples and determining whether or not they met the conditions set forth in the definition.

With this before us I ask: What is justice? That is to say, what kind of a thing is justice? It’s a virtue, no? Yes, but I’m looking for something even more general, more abstract. It’s a concept, and idea, no? Of course it is. And just what are those things? Philosophers have been pondering to question for years. Cognitive scientists have been asking that question as well. When David Hays proposed that abstract concepts can be defined by stories, he was proposing an answer to that question. Abstract concepts, such as justice, are defined by relationship among words.

* * * * *

I’ve devoted a great deal of attention to ChatGPT’s ability to deal with metalingual definition. Justice is one of the first concepts I investigated, back in December of 2022. I’ve continued to investigate that concept. I have appended my most recent session to the end of this post.

That investigation has three parts. First, I ask it to tell me two stories involving justice and I specify that it should not use the word “justice” anywhere in the stories. The point of that restriction is to make it clear that the meaning of the term does not reside in the word itself. The stories exhibit justice, but do not name it. Note that in a second session, which I’ve placed in a second appendix, I give ChatGPT the two stories, one after the other, and ask it what they’re about. It realizes that they are about justice.

After asking ChatGPT to tell me stories involving justice I ask it to define the term. The first definition is fairly long and has five numbered points, each specifying a particular kind of justice. So I ask it for a single paragraph and then a single sentence. It provides both. Note that both the long definition and the single paragraph definition begin with pretty much the same information that the single sentence contains.

Finally, I ask ChatGPT to explain the relationship between the stories and the definition, which it does in a paragraph of 112 words. Here’s the first sentence: “The relationship between the definition of justice and stories about justice lies in the way these narratives illustrate and bring to life the abstract principles of fairness, equity, and moral rightness.”

Interpretability of LLMs

What does this have to do with the interpretability of large language models? To a first approximation, it seems to me that LLMs are about the relationships between words. The transformer is presented with strings of words during training and, in the process of making those predictions, constructs a complex model of how words are related to one another.

Thus we might say that justice is a certain pattern of relationships among words. But what pattern? The pattern that gives us stories, stories which may not even contain the word “justice” or the pattern that gives us definitions and could, I assume, produce essays and even books if necessary? Those are distinctly different patterns of relationships; one might even think about them as being orthogonal, at least informally. One pattern is about justice in the context of story and the other is about justice in the context of define. Finally, what about the pattern that explication the relationship between the stories and the definitions?

In Scaling Monosemanticity, researchers at Anthropic identified features in Claude 3 Sonnet, where features are understood to be “directions in their activation spaces.” In their discussion, they note “that features often respond to both abstract discussion and concrete examples of a concept,” which is certainly something that I’d expect to be the case. One thing that bothers me about the discussion is that there is no sense of the model as capturing relationships between words. Given that these features are very abstract objects it’s not clear to me just what that misgiving means, but I worry that the concept of features invites reification.

A digression into neuroscience: Some years ago I had quite a bit of correspondence with the late Walter Freeman, who did pioneering work in thinking about the brain in terms of chaos theory and complex neurodynamics. He believed that percepts and concepts were located in populations of neurons rather than single neurons. I’m deeply sympathetic to that view, and have been ever since I read Karl Pribram on neural holography. Nonetheless I asked him about visual neurons that had very complex activation properties, such as a monkey’s paw or an image of Bill Clinton. Don’t such examples lend support to the idea of a so-called “grandmother cell”? His reply was no, they don’t. In such a complex system, you’re bound to find individual neurons with all sort of odd response characteristics.

I feel a bit like that with these features. While they don’t seem to be individual neurons, it’s not clear what they are. Robert_AIZI has expressed a similar reservation. Thus he has noted:

I think Anthropic successfully demonstrated (in the paper and with Golden Gate Claude) that this feature, at very high activation levels, corresponds to the Golden Gate Bridge. But on a median instance of text where this feature is active, it is "irrelevant" to the Golden Gate Bridge, according to their own autointerpretability metric! I view this as analogous to naming water "the drowning liquid", or Boeing the "door exploding company". Yes, in extremis, water and Boeing are associated with drowning and door blowouts, but any interpretation that ends there would be limited.

Just what IS this feature?

I’m not surprised that with judicious and determined poking around we can find interpretable “features” in these models. But whether or not we’re carving LLMs at their joints, to invoke Plato’s famous remark, that’s another matter.

It’s early days.

Appendix 1: ChatGPT on justice: stories and definitions

What follows the asterisks is a single session with ChatGPT. As always, my prompts are set in bold-face while ChatGPT’s responses are plain-face, except for those strings that ChatGPT sets in bold-face.

Mechanistic interpretability is necessary, but not sufficient, for understanding how LLMs work, a short note

A comment I recently posted at LessWrong:

ryan_greenblatt – By mech interp I mean "A subfield of interpretability that uses bottom-up or reverse engineering approaches, generally by corresponding low-level components such as circuits or neurons to components of human-understandable algorithms and then working upward to build an overall understanding."

That makes sense to me, and I think it is essential that we identify those low-level components. But I’ve got problems with the “working upward” part.

The low-level components of a gothic cathedral, for example, consist of things like stone blocks, wooden beams, metal hinges and clasps and so forth, pieces of colored glass for the windows, tiles for the roof, and so forth. How do you work upward from a pile of that stuff, even if neatly organized and thoroughly catalogues, how do you get from there to the overall design of the overall cathedral. How, for example, can you look at that and conclude, “this thing’s going to have flying buttresses to support the roof?”

Somewhere in How the Mind Works Steven Pinker makes the same point in explaining reverse engineering. Imagine you’re in an antique shop, he suggests, and you come across odd little metal contraption. It doesn’t make any sense at all. The shop keeper sees your bewilderment and offers, “That’s an olive pitter.” Now that contraption makes sense. You know what it’s supposed to do.

How are you going to make sense of those things you find under the hood unless you have some idea of what they’re supposed to do?

The sort of work I’ve done with ChatGPT’s storytelling or with its ontological capabilities provides clues that complement the phenomena discovered through mechanistic interpretability. Beyond that I’ve been thinking about the possibility that GPTs are associative memories in which the generation of a token is a single primitive operation for the underlying virtual machine. By that I mean there are no logical operations being performed within that operation, just straight calculation.

Am I right? It’s too early to say. But we have to start somewhere.

What’s going on? LLMs and IS-A sentences

For the moment I have decided that Waddington’s classic diagram of the epigenetic landscape is a useful way of thinking about when happens when an LLM responds to a prompt. Here’s the diagram:

The language model corresponds to the landscape. The prompt serves to position that ball at a certain place in the landscape – perhaps we can think of that ball as the prompt. The ball then rolls down the valley, going left and right as appropriate. It never reverses direction and goes up the hill. That path, or trajectory if you will, is the LLM’s response to the prompt.

Moreover, I have decided to think of the generation of each word (yes, I know, technically it spits out tokens, not words) as a single primitive operation. That is to say, it has no internal logical structure, no ANDs or ORs. It’s simply one (gigantic) calculation over roughly 175 billion values (in the case of ChatGPT). The generation of each word presents the system with a choice among alternatives, but that’s the only kind of choice involved in calculating the response to a prompt – though for qualification and elaboration, see ChatGPT tells stories, and a note about reverse engineering: A Working Paper, Version 3, pp. 3-6.

That brings me to something I’ve been puzzled about for years. We find it natural to say things like, Garfield is a cat. Now, express the same thought, but reverse the order of cat and Garfield in your sentence. It’s difficult to do. Oh, you can do it, but the resulting sentence is awkward and unnatural, something like, Cats are the kind of thing of which Garfield is a particular instance. No one would ever speak like that, nor write it either.

What’s the source of that asymmetry? As far as I can tell, we don’t know, but I take it as a clue about the mechanisms of language. The purpose of this note is to suggest that my crude model of LLM calculation would provide an answer: The linguistic landscape is structured so that the ball easily rolls from Garfield to cat, or cat to mammal, Snoopy to beagle, Tesla to EV, C. elegans to worm, etc. One might, of course, as why the landscape is arranged in that way, but that’s a different question, no?

Here’s some notes I made about IS-A sentences.

Notes on IS-A Sentences

Somewhere in his Problems in General Linguistics, my copy of which is, alas, in storage, Emile Benveniste has a chapter, “The Nominal Sentence,” on sentences hanging on the auxiliary “to be.” As Benveniste was a linguist of the Old School, when being a linguistic meant familiarity with many languages, including—and this is important for this particular topic—classical Greek, it had examples from many languages, making it tough sledding for a monoglot like me.

While the content of this post certainly arises out of my thinking about that chapter, in the absence of actually having the text in front of me, I hesitate to assert a stronger relationship than that. I note only that, for Benveniste, the auxiliary “to be” was fraught with metaphysical significance. For the concept of being derives from “to be.” Where would philosophy be without Being? Thus, when Benveniste pondered such sentences, he wasn’t merely commenting on language. He was doing philosophy, or, if not quite that, camping out on philosophy’s door step.

I’m interested in such sentences because I believe they are a DEEP CLUE about how the mind works. I just don’t know what to make of the clue.

So, I'm interested in word order in assertions such as the following:

(1) Fido is a beagle.
(2) Beagles are dogs.
(3) Dogs are beasts.

They all move from an element in a class (whether an individual, Fido, or another class, beagles) to a class containing it. None of them move in the opposite direction. Consider what happens when you try to go the opposite way. In the following sentence the class is mentioned first, then the subclass:

(4) Beagle is the kind of animal of which Fido is an instance.

In particular, note that (4) has a metalingual character that (1) does not. That is, (4) explicitly asserts that we are dealing with classification. One can do that metalingual job in various ways, but, as far as I can tell, one can't avoid it. That is, one cannot construct a proper English sentence relating a genus and species in which the genus is mentioned first, one can’t do that without ‘looping through’ some kind of metalingual construction on the way from genus to species.

Why?

What does this assymetry tell us about the underlying mechanisms? Why don't have sentences such as:

(5) Beagle za di Fido.

In this case "za di" is the inverse of "is a". English has no such sentences & no such inverse.

So, how widespread is this asymmetry and is there any explanation of this directionality?

I sent a query on that matter to a listserve, I forget which one, and got two replies that add some complexity to the matter. Rich Rhodes, Linguistics at UCal Berkeley, tells me that in Ojibwe the word order is reversed, the class comes before the individual, but the asymmetry remains. He then comments, which he qualifies as a quick guess:

My guess is that there is no compelling discourse function (like information flow) which makes it desirable to invert classificational equatives. Hence we only get the "unmarked" order. Subject-predicate in theme-rheme languages (like English) and predicate-subject in rheme-theme languages (like Ojibwe).

So, what's the nature of the mechanism that determines the "unmarked" order? That's what I want to know.

Lee Pearcy, Episcopal Academy in Merion, Pa. offered these examples:

(6) The beagle is Fido.
(7) The dogs are beagles.
(8) The beasts are dogs.

As stand-alone sentences, they seem a bit awkward to me. But they fare better as answers to questions, e.g.:

What’s that dog?
Which dog? The beagle is Fido and the terrier is Max.

What’re those animals?
The dogs are beagles, the cats are Persians.

In those contexts, the matter of class or classification is raised by the question, thus making it present in the discourse and so available as a point of attachment in the answer.

Further clues, anyone?

Do I believe this?

I don’t believe it, or disbelieve it. It’s a working hypothesis. One I think is worth investigating. It places relatively simple and severe constraints on our conception of what LLMs are doing. That, it seems to me, is a good thing. Should it turn out that those constraints are valid, well then, we’ve learned something, no? If they’re not valid, we’ve also learned something.

More later.

Copy suppression head in GPT-2 [interpretability]

What’s copy suppression? Without head L10H7, GPT-2 naively follows "All's fair in love and" with " love". L10H7 detects the " love" prediction, attends to the previous " love" token, suppressing the token, resulting in the correct prediction " war". More examples: pic.twitter.com/NsIg3dSvpf

— Callum McDougall (@calsmcdougall) October 13, 2023

Bye, bye black boxes, hello interpretability [artificial neural nets]

AI models are not just black boxes or giant inscrutable matrices.

We discover they have interpretable internal representations, and we control these to influence hallucinations, bias, harmfulness, and whether a LLM lies.

🌐: https://t.co/6hU83vaYUk
📄: https://t.co/xyrzgfUOfn pic.twitter.com/jLW3LSmeE8

— Dan Hendrycks (@DanHendrycks) October 3, 2023

Abstract of the linked article, Representation Engineering: A Top-Down Approach to AI Transparency: 

In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems.

Polysemantic "neurons" in LLMs

We hope this will eventually enable us to diagnose failure modes, design fixes, and certify that models are safe for adoption by enterprises and society. It's much easier to tell if something is safe if you can understand how it works!

— Anthropic (@AnthropicAI) October 5, 2023

Last year, we conjectured that polysemanticity is caused by "superposition" – models compressing many rare concepts into a small number of neurons. We also conjectured that "dictionary learning" might be able to undo superposition.https://t.co/bgJdScRcay

— Anthropic (@AnthropicAI) October 5, 2023

There are more tweets in the thread. Check it out.

What would it mean to understand how a large language model (LLM) works? Some quick notes.

I don’t mean “understand” in any deep philosophical sense. I mean only a rough and ready sense of the word. We understand how toasters work, automobiles, moon rockets, digital computers, and so forth. We know how to design and construct these things, how to diagnose problems, how to maintain and repair them. Not perfectly to be sure, but well enough to use these devices to get things done.

LLMs, however, are said to be opaque. We don’t know how they work. We feed them prompts, they produce output, but how the model works from the prompt to produce the output, that’s mysterious. There are people working on mechanical interpretability, trying to understand the LLM as though it were a machine, or at least, a computer program of the ordinary kind, where we know, more or less, how it works on data – if it is the kind of program that works from data – to produce output. But what would it mean to understand the operational characteristics of 175 billion parameters, as in the case of GPT-3.5?

It means, I suppose, how those parameters mediate between the input, a prompt, and the output, whatever “follows from” a given prompt. At the lowest level we are told that LLMs are prediction machines. So, the output string is simply a continuation of the input string. And I suppose that, technically, that’s true. But it’s not very helpful, as I’ve argued at some length.

Let’s set that aside.

What could we possibly want by way of understanding?

We’ve got three things: There is the underlying engine, let’s call it, which is a computer program like any other. It’s created by programmers working with some language or languages and is designed to achieve a certain purpose. In this case, it’s designed to create a language model over a corpus of texts and then to use that model in generating new chunks of language given an input prompt.

It's that model that’s problematic, that’s said to be opaque. We, us humans, didn’t create that model. The engine did. And, in the case of GPT-3, that model’s got 175 billion parameters. More recent models have even more. And there are also models with only millions of parameters. But even those smaller models are huge.

But, here’s the thing, how can we understand how that opaque model operates unless we understanding what it’s trying to do? Sure, we can pop the hood and take a look. We see a bunch of gizmos, widgets, framblasts, and other things, but so what? They’re just whirling around, engaging with one another, in intricate patterns? But what are they trying to do? We know what car engines are supposed to do; they supply power to the wheels (and the wheels move the car).

Well, LLMs are supposed to produce language – and computer code and math as well, but let’s stick with ordinary language for the purposes of these notes. But, alas, the mechanisms of language are themselves opaque. The relationship between a car's wheels and that car's motion is transparent. The relationship between nouns and verbs and adjectives and prepositions and sentences and, you know, knowledge, understanding, entertainment, the things language is for, those relationships are not so obvious.

Of course, linguists have been working on language mechanisms for years. But it’s not at all clear what the field has come up with. There are major disagreements on how one is to understand syntax. And when we move beyond sentences to discourse of various kinds, we know even less about mechanisms.

I figure that there’s almost zero chance that we’re going to find those mechanisms by mucking around in LLMs. Yes, I know that LLMs are quite different from the human brain and mind. But, the fact is, LLMs do a very convincing imitation of human language. Given the complexity of language, they wouldn’t be able to do that if they hadn’t absorbed some (perhaps) useful approximation to human mechanisms. I’m willing to proceed on the default understanding that, whatever the model is doing, it has some resemblance to what humans do. If I make that assumption, that gives me some tools to think with. Without it, I got nothing.

Still, a grammar is a large and complex thing. The Cambridge Grammar of the English Language is 1860 pages long, and it is merely a descriptive grammar and not meant to account for the underlying mechanisms, however they might best be characterized. Is that what we want from a mechanistic understanding of an LLM? And that only gets us sentences. What about paragraphs, stories, histories, repair manuals, accounts of exotic astronomical objects, and who knows what else? Do we expect students of mechanistic interpretability to eventually give us detailed accounts of such wonders?

Understanding stories

What would it mean to understand how ChatGPT tells stories?

This morning I logged onto ChatGPT, not GPT Plus, just plain old ChatGPT, and prompted it with one word: “Story.” What do you think it did? Right, it told me a story. The story began with this sentence: “Once upon a time, in a quaint little village nestled at the foot of a towering mountain range, there lived a young girl named Lily.” I don’t think it’s very useful to think of that sentence as the natural continuation of a string beginning with the word, “story.” Yes, I know, I’m not prompting the “naked” underling LLM. ChatGPT has been prompt-engineered and RLHFed (RLHF: reinforcement learning with human feedback) to death to be a congenial conversational partner. But that doesn’t change the basic situation.

In this case, the situation is that, in some sense, ChatGPT “knows” what a story is and knows how to tell one. By this time I’ve prompted it to produce 100s, though probably not yet 1000s of stories. In a few cases the prompt was just that one word. More often it was something like one of these:

Tell me a story.
Tell me a story about a hero.
Tell me a realistic story.
Tell me a true story about a hero.

ChatGPT also told me a well-formed story. The stories were relatively short and simple, and the first two prompts produced stories with a fairytale feel, supernatural creatures and events were typical. Those were absent in realistic stories. As for true stories, sometimes they read more like short newspaper articles than like stories.

But where did ChatGPT learn to tell stories? Well, it consumed I don’t know how many stories during training. Whatever it knows about story-telling was distilled from those stories. I note that, to a first approximation, that’s how humans learn to tell stories as well. We are told stories as toddlers and children and, in time, begin telling our own stories, based on the models we’ve been exposed to. New stories are based on old stories, on remembered and half-remembered stories.

Now, as you may know, at some point I began to have ChatGPT tell stories based on rather elaborate prompts of a simple form consisting of 1) a request to tell a new story based on an existing one, but with one change (which I specified) and 2) the existing story. For example:

Neel Nanda on that paper: [LLMs] have a 1 way flow from inputs to outputs, and a fundamental asymmetry between inputs and outputs.

This paper's been doing the rounds, so I thought I'd give a mechanistic interpretability take on what's going on here!

The core intuition is that "When you see 'A is', output B" is implemented as an asymmetric look-up table, with an entry for A->B.
B->A would be a separate entry https://t.co/XROT8D5MrS

— Neel Nanda (@NeelNanda5) September 24, 2023

Here's the rest of Nanda's tweet stream:

The key question to ask with a mystery like this about models is what algorithms are needed to get the correct answer, and how these can be implemented in transformer weights. These are what get reinforced when fine-tuning.

The two hard parts of "A is B" are recognising the input tokens A (out of all possible input tokens) and connecting this to the action to output tokens B (out of all possible output tokens). These are both hard!

Further, the A -> B look-up must happen on a single token position

Intuitively, the algorithm here has early attention heads attend to the prev token to create a previous token subspace on the Cruise token. Then an MLP neuron activates on "Current==Cruise & Prev==Tom" and outputs "Output=Mary", "Next Output=Lee" and "Next Next Output=Pfeiffer"

"Output=Mary" directly connects to the unembed, and "Next Output=Lee" etc gets moved by late attention heads to subsequent tokens once Mary is output.

Crucially, there's an asymmetry between "input A" and "output A". Inputs are around at early layers, come from input embeddings, and touch the input weights of MLP neurons. Outputs are around more at late layers, compose with the unembedding, and come from output weights of MLPs

This is especially true with multi-token A and B. Detecting "Tom Cruise" is saying "the current token embedding is Cruise, and the prev token space says Tom", while output "Tom Cruise" means to output the token Tom, and then a late attn head move "output Cruise" to the next token

Thus, when given a gradient signal to output B given "A is" it reinforces/creates a lookup "A -> B", but doesn't create "B->A", these are different lookups, in different parameters, and there's no gradient signal from one to the other.

How can you fix this? Honestly, I can't think of anything. I broadly think of this as LLMs working as intended. They have a 1 way flow from inputs to outputs, and a fundamental asymmetry between inputs and outputs. It's wild to me to expect symmetry/flow reversing to be possible

Why is this surprising at all then? My guess is that symmetry is intuitive to us, and we're used to LLMs being capable of surprising and impressive things, so it's weird to see something seemingly basic missing.

LLMs are not human! Certain things are easy for us and not for them, and vice versa. My guess is that the key difference here is that when detecting/outputting specific tokens, the LLM has no notion of a variable that can take on arbitrary values - a direction has fixed meaning Neel Nanda

A better analogy might be in-context learning, where LLMs CAN use "variables". The text "Tom Cruise is the son of Mary Lee Pfeiffer. Mary Lee Pfeiffer is the mother of..." has the algorithmic solution "Attend to the subject of sentence 1 (Tom Cruise), and copy to the output"

Unsurprisingly, the model has no issue with reversing facts in context! Intuitively, when I remember a fact A is B, it's closer to a mix of retrieving it into my "context window" and then doing in-context learning, rather than pure memorised recall.

* * * * *

The paper:  The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A."

Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs

New work w/@ziv_ravid @kchonyc @leavittron @nsaphra: We break the steepest MLM loss drop into *2* phase changes: first in internal grammatical structure, then external capabilities. Big implications for emergence, simplicity bias, and interpretability! 🧵 https://t.co/15KcwwK7nP pic.twitter.com/GRFvaQ3DnA

— Angelica Chen (@_angie_chen) September 15, 2023

Read the whole thread and check out the paper.

How large language models recall factual associations

We prompt GPT-{2|J} with subject-relation queries (“Beats Music is owned by”) from CounterFact (@mengk20 @davidbau) and intervene on attention edges (similar to @hmohebbi75) to analyze how information is aggregated across layers and positions to predict the attribute (“Apple”)

— Mor Geva (@megamor2) May 1, 2023

Go to Twitter for the rest of the tweet stream. Here's the abstract of the article:

Transformer-based language models (LMs) are known to capture factual knowledge in their parameters. While previous work looked into where factual associations are stored, only little is known about how they are retrieved internally during inference. We investigate this question through the lens of information flow. Given a subject-relation query, we study how the model aggregates information about the subject and relation to predict the correct attribute. With interventions on attention edges, we first identify two critical points where information propagates to the prediction: one from the relation positions followed by another from the subject positions. Next, by analyzing the information at these points, we unveil a three-step internal mechanism for attribute extraction. First, the representation at the last-subject position goes through an enrichment process, driven by the early MLP sublayers, to encode many subject-related attributes. Second, information from the relation propagates to the prediction. Third, the prediction representation "queries" the enriched subject to extract the attribute. Perhaps surprisingly, this extraction is typically done via attention heads, which often encode subject-attribute mappings in their parameters. Overall, our findings introduce a comprehensive view of how factual associations are stored and extracted internally in LMs, facilitating future research on knowledge localization and editing.

ChatGPT tells stories, and a note about reverse engineering: A Working Paper

I've uploaded a new working paper. Title above, links, abstract, content, and opening sections below.

Academia.edu: https://www.academia.edu/97862447/ChatGPT_tells_stories_and_a_note_about_reverse_engineering_A_Working_Paper
SSRN: https://ssrn.com/abstract=4377178
Research Gate: https://www.researchgate.net/publication/368978013_ChatGPT_tells_stories_and_a_note_about_reverse_engineering_A_Working_Paper

Abstract: I examine a set of stories that are organized on three levels: 1) the entire story trajectory, 2) segments within the trajectory, and 3) sentences within individual segments. I conjecture that the probability distribution from which ChatGPT draws next tokens follows a hierarchy nested according to those three levels and that is encoded in the weights off ChatGPT’s parameters. I arrived at this conjecture to account for the results of experiments in which ChatGPT is given a prompt containing a story along with instructions to create a new story based on that story but changing a key character: the protagonist or the antagonist. That one change then ripples through the rest of the story. The pattern of differences between the old and the new story indicates how ChatGPT maintains story coherence. The nature and extent of the differences between the original story and the new one depends roughly on the degree of difference between the key character and the one substituted for it. I conclude with a methodological coda: ChatGPT’s behavior must be described and analyzed on three levels: 1) The experiments exhibit surface level behavior. 2) The conjecture is about a middle level that contains the nested hierarchy of probability distributions. 3) The transformer virtual machine is the bottom level.

Contents 

How you might read this paper 2
Story coherence 2
My procedure 2
Conjecture: A nested hierarchy of probability distributions 3
Could we develop a quantitative model? 6
12 Experiments: The Chronicles of Princess Aurora and William the Lazy 7
Reverse Engineering ChatGPT 19

How you might read this paper

This paper is about the twelve experiments that make up the two-thirds of the text in this paper. One way to read the paper, perhaps even the preferred way, is to start by reading the highlighted paragraph in the next section. That tells you what I did in those experiments. Move from that directly to the experiments. Read through them, think about them in your own terms, and then read the rest of the paper.

Story coherence

A story isn’t just any narrative sequence. As Aristotle remarked in his Poetics, proper stories have a beginning, a middle, and an end. Just what, however, constitutes a proper beginning, middle, and an end? Narratologists in various disciplines have been trying to figure this out for decades. I’m not holding my breath for answers.

Instead I have been investigating how ChatGPT tells stories. The procedure I have been using is derived from the analytical method Claude Lévi-Strauss employed in his magnum opus, Mythologiques. He started with one myth, analyzed it, and then introduced another one, very much like the first. But not quite. They are systematically different. He characterized the difference by a transformation – a term he took from algebraic group theory. He worked his way through hundreds of myths in this manner, each one derived from another by a transformation.

Here is what I have been doing: I give ChatGPT a prompt consisting of two things: 1) an existing story and 2) instructions to produce another story like it except for one change, which I specify. That change is, in effect, a way of triggering or specifying those “transformations” that Lévi-Strauss wrote about. What interests me are the ensemble of things that change along with the change I have specified.

I take that ensemble of changes as an indicator of an underlying ‘economy’ of story coherence that is implicit in the processes ChatGPT uses in generating a story. For the most part, however, I make no attempt to analyze how that coherence is achieved other than to note differences between the source story and the new one. Those observations must be accounted for by 1) some theory about what coherence is, and 2) the mechanisms through which ChatGPT achieves the coherence. I have included a dozen before-and-after experiments along with some comments on them (pp. 7 ff.).