Engine Company No. 2

World "model" in Llama-2

Do language models have an internal world model? A sense of time? At multiple spatiotemporal scales?

In a new paper with @tegmark we provide evidence that they do by finding a literal map of the world inside the activations of Llama-2! pic.twitter.com/3kZmf3fa6q

— Wes Gurnee (@wesg52) October 4, 2023

For temporal representations, we run the models on the names of famous figures from the past 3000 years, the names of songs, movies and books from 1950 onward, and NYT headlines from the 2010s and train lin probes to predict the year of death, release date, and publication date. pic.twitter.com/KAajV03VKk

— Wes Gurnee (@wesg52) October 4, 2023

There are more tweets in the thread.

The paper, Language Models Represent Space and Time:

Abstract: The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a coherent model of the data generating process -- a world model. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual ``space neurons'' and ``time neurons'' that reliably encode spatial and temporal coordinates. Our analysis demonstrates that modern LLMs acquire structured knowledge about fundamental dimensions such as space and time, supporting the view that they learn not merely superficial statistics, but literal world models.

Note, however:

Self organizing maps (SOM) have been doing that since like forever.

Researchers just found SOM-like representations in layers of a trained deep neural net & jumping to premature conclusions that there must therefore exist a world model in those systems.

No https://t.co/OfHA0sLnxO pic.twitter.com/CJyvWiXBe1

— Chomba Bupe (@ChombaBupe) October 4, 2023

Midtown, Manhattan, Port Authority to the right

Entanglement and intuition about words and meaning

Two things have just occurred to me about my recent post, Word meaning and entanglement in LLMs:

1.) That the issue is one of intuition as well, and
2.) that we’re dealing with system 1 thinking, in the System 1/system 2 dichotomy popularized by Daniel Kahneman.

I’ve not yet read Kahneman’s book – Thinking Fast, Thinking Slow, though it’s on my “to be read someday” shelf – but I gather that System 1 is fast, intuitive and largely tacit (to use a word from Michael Polanyi) while System two is slow, deliberate, and logical.

My argument in that earlier post is that, in effect, our default notion of word meaning is that it is atomic and discrete. When words are linked in phrases, sentences, and paragraphs, it is liking beads on a thread, or freight cars in a train. The linkage is external and contingent. Without reflection, that’s just how we think about words (and meaning). That’s fine in informal discussions, but not so good in at least some technical contexts, such as large language models (LLMs).

Now, let’s take the idea that LLMs are “trained” by being asked to predict the next word. That is at least consistent with, if not actually reinforcing of, this default conceptualization of atoms-of-meaning. One can easily make predictions about the behavior of atoms. One simply observes them and notes down what they do from one moment to the next. There is no sense of “interiority.”

Whereas the idea that words are entangled with one another through their meanings, that’s all about “interiority.” Those vectors are “interior” to the token, and relate one token to another and, more generally, tokens among themselves. The idea of entanglement leads naturally to the idea of weaving, weaving a fabric of meaning. The so-called prediction procedure, then, is one of placing a word, with its 12K item vector, into the unfolding fabric of meaning. Backpropagation, in this view, is the act of fine-tuning the placement. Prediction is merely a means to an end, a device, not the point of the procedure.

To think in terms of atomic meaning is simply to gloss over all this. All of that may be implicit in the mathematics, but the atomic view of meaning stands in the way of allowing ones thought to be perspicuously guided by the mathematics. The mathematics becomes (and functions as) a secondary construction.

I note finally that traditional training in propositional and symbolic logic reinforces this atomic view of word meaning. Word meaning is reduced to variable names, Ps and Qs, having no intrinsic content whatsoever. That’s find for System 2 deliberative thinking, which is what logic was invented for. But it gets in the way of understanding how meaning works in collections of entangled, entangled what? What do we call them?

This leads to a final irony: The world of standard computer programming is close kin to that of symbolic and propositional logic, with their variables, bindings, and values. Thus the mode of thinking necessary for programming the computational engines that create LLMs, that mode of thought stands in the way of understanding how LLMs work. The AI/ML experts who create the models are thus crippled in understanding how they work. The intuitions that guide them in writing code render the operations of LLMs opaque and invisible when deployed in understanding them.

This opacity thus has two aspects:

1.) the sheer complexity of the models, and
2.) conceptual intractability.

I am suggesting, then, that thinking of meaning as entailing entanglement is a way to deal with the second issue (and this may also lead to a holographic account as well, but this is a secondary issue). On the first issue, complexity, that is there regardless of your conceptual instruments. Thinking in terms of entanglement will NOT eliminate the complexity, but it may well make it tractable. 

If your goal is mechanistic interpretability, then you need conceptual tools appropriate to the mechanisms you are trying to understand, no? You need to discard, or at least bracket, intuitions based on the idea of atomic-self-contained word meaning and develop intuitions that are consistent with the mathematics underlying the LLMs.

At last, someone has taken the proper measure of Dan Dennett, who comes up short.

Dan Dennett has written and autobiography, I've Been Thinking. I suppose he has. Stuart Jeffries reviews it in The Guardian. He opens with the story of how Dennett came up with the idea that his brain might very well have been in a vat, "which, he tells us proudly here, powerfully influenced the makers of The Matrix." Jeffries then goes on about some this and some that, including the fallout from Dennett's 1991 Consciousness Explained, and that leads to this wonderful paragraph:

These are deep matters and yet Dennett is an unreliable guide to them. His memoir involves more chippy score-settling with intellectual opponents, some of them long dead, than is dignified. But Dennett is a conceited fellow: it takes someone with an overdeveloped sense of self and an underdeveloped sense of tact to write the following: “The philosopher Don Ross once said of me: ‘Dan believes modesty is only a virtue to be reserved for special occasions.’” My copy is now dotted with marginal “ughs” to register my irritation at some piece of professorial preening. Even when Dennett tells us the truth about himself – namely, that he is one of the few philosophers to get out of the armchair to study neuroscience, artificial intelligence, computer science and psychology – the tone is boastful. He is unremittingly self-hagiographic, confirming the notion that autobiography can be the lowest of literary genres. He transforms an interesting thinker into a dull protagonist.

YES.

I've always thought Dennett coasted through his career on a vast sea of cleverness while avoiding the need to think deeply about anything. Well, yes, that's a bit of an exaggeration. He has written a thing or three that I've liked, something about patterns as I recall. But that's not enough to balance out the damage he's done, in particular, for all the nonsense about memes that he's fostered.

I've posted a couple of working papers in which I critique various of his ideas: Dennett on Memes, Neurons, and Software, An Inquiry into & a Critique of Dennett on Intentional Systems, and Cultural Evolution, Memes, and the Trouble with Dan Dennett.

From Zeitgeist to thesis, antithesis, and synthesis: a Hegelian breakfast in South Philly

What would it mean to understand how a large language model (LLM) works? Some quick notes.

I don’t mean “understand” in any deep philosophical sense. I mean only a rough and ready sense of the word. We understand how toasters work, automobiles, moon rockets, digital computers, and so forth. We know how to design and construct these things, how to diagnose problems, how to maintain and repair them. Not perfectly to be sure, but well enough to use these devices to get things done.

LLMs, however, are said to be opaque. We don’t know how they work. We feed them prompts, they produce output, but how the model works from the prompt to produce the output, that’s mysterious. There are people working on mechanical interpretability, trying to understand the LLM as though it were a machine, or at least, a computer program of the ordinary kind, where we know, more or less, how it works on data – if it is the kind of program that works from data – to produce output. But what would it mean to understand the operational characteristics of 175 billion parameters, as in the case of GPT-3.5?

It means, I suppose, how those parameters mediate between the input, a prompt, and the output, whatever “follows from” a given prompt. At the lowest level we are told that LLMs are prediction machines. So, the output string is simply a continuation of the input string. And I suppose that, technically, that’s true. But it’s not very helpful, as I’ve argued at some length.

Let’s set that aside.

What could we possibly want by way of understanding?

We’ve got three things: There is the underlying engine, let’s call it, which is a computer program like any other. It’s created by programmers working with some language or languages and is designed to achieve a certain purpose. In this case, it’s designed to create a language model over a corpus of texts and then to use that model in generating new chunks of language given an input prompt.

It's that model that’s problematic, that’s said to be opaque. We, us humans, didn’t create that model. The engine did. And, in the case of GPT-3, that model’s got 175 billion parameters. More recent models have even more. And there are also models with only millions of parameters. But even those smaller models are huge.

But, here’s the thing, how can we understand how that opaque model operates unless we understanding what it’s trying to do? Sure, we can pop the hood and take a look. We see a bunch of gizmos, widgets, framblasts, and other things, but so what? They’re just whirling around, engaging with one another, in intricate patterns? But what are they trying to do? We know what car engines are supposed to do; they supply power to the wheels (and the wheels move the car).

Well, LLMs are supposed to produce language – and computer code and math as well, but let’s stick with ordinary language for the purposes of these notes. But, alas, the mechanisms of language are themselves opaque. The relationship between a car's wheels and that car's motion is transparent. The relationship between nouns and verbs and adjectives and prepositions and sentences and, you know, knowledge, understanding, entertainment, the things language is for, those relationships are not so obvious.

Of course, linguists have been working on language mechanisms for years. But it’s not at all clear what the field has come up with. There are major disagreements on how one is to understand syntax. And when we move beyond sentences to discourse of various kinds, we know even less about mechanisms.

I figure that there’s almost zero chance that we’re going to find those mechanisms by mucking around in LLMs. Yes, I know that LLMs are quite different from the human brain and mind. But, the fact is, LLMs do a very convincing imitation of human language. Given the complexity of language, they wouldn’t be able to do that if they hadn’t absorbed some (perhaps) useful approximation to human mechanisms. I’m willing to proceed on the default understanding that, whatever the model is doing, it has some resemblance to what humans do. If I make that assumption, that gives me some tools to think with. Without it, I got nothing.

Still, a grammar is a large and complex thing. The Cambridge Grammar of the English Language is 1860 pages long, and it is merely a descriptive grammar and not meant to account for the underlying mechanisms, however they might best be characterized. Is that what we want from a mechanistic understanding of an LLM? And that only gets us sentences. What about paragraphs, stories, histories, repair manuals, accounts of exotic astronomical objects, and who knows what else? Do we expect students of mechanistic interpretability to eventually give us detailed accounts of such wonders?

Understanding stories

What would it mean to understand how ChatGPT tells stories?

This morning I logged onto ChatGPT, not GPT Plus, just plain old ChatGPT, and prompted it with one word: “Story.” What do you think it did? Right, it told me a story. The story began with this sentence: “Once upon a time, in a quaint little village nestled at the foot of a towering mountain range, there lived a young girl named Lily.” I don’t think it’s very useful to think of that sentence as the natural continuation of a string beginning with the word, “story.” Yes, I know, I’m not prompting the “naked” underling LLM. ChatGPT has been prompt-engineered and RLHFed (RLHF: reinforcement learning with human feedback) to death to be a congenial conversational partner. But that doesn’t change the basic situation.

In this case, the situation is that, in some sense, ChatGPT “knows” what a story is and knows how to tell one. By this time I’ve prompted it to produce 100s, though probably not yet 1000s of stories. In a few cases the prompt was just that one word. More often it was something like one of these:

Tell me a story.
Tell me a story about a hero.
Tell me a realistic story.
Tell me a true story about a hero.

ChatGPT also told me a well-formed story. The stories were relatively short and simple, and the first two prompts produced stories with a fairytale feel, supernatural creatures and events were typical. Those were absent in realistic stories. As for true stories, sometimes they read more like short newspaper articles than like stories.

But where did ChatGPT learn to tell stories? Well, it consumed I don’t know how many stories during training. Whatever it knows about story-telling was distilled from those stories. I note that, to a first approximation, that’s how humans learn to tell stories as well. We are told stories as toddlers and children and, in time, begin telling our own stories, based on the models we’ve been exposed to. New stories are based on old stories, on remembered and half-remembered stories.

Now, as you may know, at some point I began to have ChatGPT tell stories based on rather elaborate prompts of a simple form consisting of 1) a request to tell a new story based on an existing one, but with one change (which I specified) and 2) the existing story. For example:

The lion of South Philly don't get no respect.

The AI future isn’t going to be what you think, can’t be

Now that ChatGPT and other engines have put AI on the agenda, hopes and fears for the future are multiplying. But they are hopes and fears of people who have never lived a world where (powerful and flexible) AI is ubiquitous. Living in such a world will surely change our hopes and fears in ways we cannot now imagine, no matter how much science fiction we consume. After all, the creators of that science fiction haven’t actually lived in the worlds they are imagining.

This puts entrepreneurs in a tricky situation. What if the products they create change how people deal with the world in ways that make those products obsolete? The so-called Metaverse is one thing to those of us just imagining it – which is pretty much all of us. What will be to those who can enter it at will? And what, in consequence, will everything else become?

Word meaning and entanglement in LLMs

It is my impression that, unless someone has had experience with distributed accounts of word meaning, they’re likely to think of word meaning as an enclosed “atom” of meaning, distinct from other such atoms, but like word forms themselves. The meaning of a proposition or a sentence is just composed of a string of such atoms of meaning, as a freight train is composed of a string of cars. I like to oppose this with a different metaphor, dropping pebbles into a pond, one after the other. Each pebble sends ripples across the surface of the pond. The succession of ripples from each pebble interferes with the others. That growing interference pattern is the meaning of the string.

And that’s how we need to think about meaning in LLMs, sorta’. Each word consists of a token and the vector encoding its meaning as an embedding in a high-dimensional space – roughly 12K, I believe, for GPT-3. Given two words, we can compare their vectors, dimension by dimension. Where the words are closely related, they should have similar, perhaps even identical, values along some dimensions. Where the words are highly dissimilar, they will share few or no values.

I find the idea of entanglement useful here. Some words have meanings that are closely entangled, while others do not. We can think of an embedding model as an entanglement matrix. This matrix shows how the meaning of any one word is a function of it position in the matrix. When you present a prompt to, say, ChatGPT, it generates an output by calculating the entanglement of the prompt with the language model.

Contrast this way of thinking with the standard, “Generate the net token, and the one after that, and so on.” The standard way of thinking has you thinking in terms of atomic units, tokens, and obscures the nature of the process, making it seem deeply obscure, even magical. Just what’s going on when the underlying model is “calculating the entanglement” of the prompt with the model is not at all obvious – I can’t tell you what it is – but it has a different feel. Similarly, training by “predict the next word” is really a way of calculating the entanglement of the text with the whole model, for the whole text (in the context window) is involved in the calculation, not just the leading word.

More later.

From yesterday's walk in South Philly

Dynamics in the epigenetic landscape

The molecular and mathematical basis of Waddington's epigenetic landscape: a framework for post-Darwinian biology? https://t.co/pXUg3PVxkx - this (2012) paper gives a great dynamical systems view of gene regulatory networks pic.twitter.com/UnNcOtxb6D

— Kevin Mitchell @WiringtheBrain@ mstdn .social (@WiringTheBrain) October 1, 2023

The transformation of Arnold Schwarzenegger

David Marchese, Arnold Schwarzenegger Is Here to Pump You Up (Emotionally), NYTimes, Sept. 28, 2023. Schwarzenegger has a new coming book out on Oct. 10: Be Useful: Seven Tools for Life. Marchese interviews him. From the interview:

The last 30 pages or so of the book are about the idea of having a responsibility to others. I’m curious to know more about how you got to that understanding. Because you could be a bully, you were disrespectful with women, you were driven primarily by thoughts of your own greatness. How does that guy transition into a person who realizes the value of selflessness? That’s a good question. First, pull back and do the master shot. I always fell into things. If it is helping other people, if it is the environmental issue, if it is becoming governor of California, if it is fighting prejudice — any of those things, I fell into it. To go back to this particular subject, I noticed that in bodybuilding, I always felt the best when I could teach and help someone. But one day I got a phone call from Wisconsin. They invited me to come to the university where they have a program for the Special Olympics, and would I teach them about weight training? I only paid attention because it was Jackie Kennedy who called. I said, I cannot say no to Jackie Kennedy. So I go. Then, of course, when she picked me up from the airport, I realized that it was not that Jackie Kennedy.

Too late to turn around. I go, and we started teaching Special Olympians how to bench press — the most unbelievable experience. You could see the change in those kids. I was blown away. I went home at night and said to myself: “It’s funny that I feel so good about this. I didn’t make any money, and it was not a career move.” It started there. It has not stopped since.

And then there's the part where he got Milton Berle to write one-liners for him. 

* * * * *

Cf. Buckminster Fuller, The Critical Path, p. 125:

... in my first thirty-two years of life I had been positively effective in producing life- advantage wealth—which realistically protected, nurtured, and accommodated X numbers of human lives for Y numbers of forward days—only when I was doing so entirely for others and not for myself.

Internal feedback in the cortical perception-action loop enables fast and accurate behavior

Jing Shuang (Lisa) Lia, Anish A. Sarmaa, Terrence J. Sejnowskic, and John C. Doyle, Internal feedback in the cortical perception-action loop enables fast and accurate behavior, PNAS, September 22, 2023 120 (39) e2300445120 https://doi.org/10.1073/pnas.2300445120, asXiv: https://arxiv.org/abs/2211.05922

Significance

Internal feedback projections—signals flowing from motor areas or late sensory processing regions back to early sensory processing regions such as primary visual and auditory areas—are ubiquitous in the sensorimotor nervous system and are as or more numerous than feedforward projections. However, the function of internal feedback is poorly understood, particularly in the context of task performance. We leverage control theory and simple models to demonstrate that internal feedback facilitates good task performance when there are communication limitations such as internal time delays and speed–accuracy trade-offs, which motivate compensatory feedback signals to counter self-generated and predictable movements. Control theory explains why motor-related signals are found throughout the sensory cortex and why the motor cortex is dominated by internal dynamics.

Abstract

Animals move smoothly and reliably in unpredictable environments. Models of sensorimotor control, drawing on control theory, have assumed that sensory information from the environment leads to actions, which then act back on the environment, creating a single, unidirectional perception–action loop. However, the sensorimotor loop contains internal delays in sensory and motor pathways, which can lead to unstable control. We show here that these delays can be compensated by internal feedback signals that flow backward, from motor toward sensory areas. This internal feedback is ubiquitous in neural sensorimotor systems, and we show how internal feedback compensates internal delays. This is accomplished by filtering out self-generated and other predictable changes so that unpredicted, actionable information can be rapidly transmitted toward action by the fastest components, effectively compressing the sensory input to more efficiently use feedforward pathways: Tracts of fast, giant neurons necessarily convey less accurate signals than tracts with many smaller neurons, but they are crucial for fast and accurate behavior. We use a mathematically tractable control model to show that internal feedback has an indispensable role in achieving state estimation, localization of function (how different parts of the cortex control different parts of the body), and attention, all of which are crucial for effective sensorimotor control. This control model can explain anatomical, physiological, and behavioral observations, including motor signals in the visual cortex, heterogeneous kinetics of sensory receptors, and the presence of giant cells in the cortex of humans as well as internal feedback patterns and unexplained heterogeneity in neural systems.

From Pier 13 to the world