It operates like a giant version of the autocomplete technology that suggests the next word as you type out an email or an instant message on your smartphone. Given the sequence “Tom Cruise is a ____,” it might guess “actor.”
As you might imagine, my initial reaction was: No! No! No! A thousand times NO! and then some. After all, I’ve been objecting to this characterization for a while now. After I’d peeled myself from the ceiling I decided to click on that link, and found something interesting, indeed.
This a bit of animation, which I can’t show here. It is showing us next-word generation starting with this question: Who is LeBron James? The AI has already replied: LeBron James is an American professional... The animation steps through the addition of new words and, shows us the shifting probability distribution as it moves along. We get these short paragraphs as we move down the page:
When artificial intelligence software like ChatGPT writes, it considers many options for each word, taking into account the response it has written so far and the question being asked.
It assigns a score to each option on the list, which quantifies how likely the word is to come next, based on the vast amount of human-written text it has analyzed.
ChatGPT, which is built on what is known as a large language model, then chooses a word with a high score, and moves on to the next one.
The model’s output is often so sophisticated that it can seem like the chatbot understands what it is saying — but it does not.
Every choice it makes is determined by complex math and huge amounts of data. So much so that it often produces text that is both coherent and accurate.
Yes! This is just the kind of thing we need. And we need more of it. Much more. And it gets better as we go on.
“This is terra incognita,” Dr. Sejnowski said. “Humans have never experienced this before.”
Let’s go back to Cade Metz. He continues on:
When you chat with a chatbot, the bot is not just drawing on everything it has learned from the internet. It is drawing on everything you have said to it and everything it has said back. It is not just guessing the next word in its sentence. It is guessing the next word in the long block of text that includes both your words and its words.
The longer the conversation becomes, the more influence a user unwittingly has on what the chatbot is saying. If you want it to get angry, it gets angry, Dr. Sejnowski said. If you coax it to get creepy, it gets creepy.
The alarmed reactions to the strange behavior of Microsoft’s chatbot overshadowed an important point: The chatbot does not have a personality. It is offering instant results spit out by an incredibly complex computer algorithm.
I suppose that makes sense even if you either knew about shifting probability distributions already or if you linked out to that wonderful animation. But the more you know, the better.
Metz goes on:
But there’s a caveat to this reassurance: Because chatbots are learning from so much material and putting it together in such a complex way, researchers aren’t entirely clear how chatbots are producing their final results. Researchers are watching to see what the bots do and learning to place limits on that behavior — often, after it happens.
Microsoft and OpenAI have decided that the only way they can find out what the chatbots will do in the real world is by letting them loose — and reeling them in when they stray. They believe their big, public experiment is worth the risk.
That’s how it is for now. And, if you’ve played with ChatGPT you know that have the opportunity to give a thumbs up or thumbs down to each response. OpenAI is, of course, logging that information and using it to further tune ChatGPT.
I have come to think of these large so-called Foundation Models as the digital wilderness. We need to explore it, stake out claims, clear the land, and start building on it. Ultimately, “building on it” is going to mean linking it to a world model. We’re not there yet, alas.
Here's the last words in the article:
“This is terra incognita,” Dr. Sejnowski said. “Humans have never experienced this before.”
Some of my thinking anyhow, but by no means all of it. My thinking about the Chatster goes off in many directions, too many to chase down and corral for a single article. Here it is:
Much of the article revolves around the question: What’s ChatGPT doing? I presented an idea that David Marr advanced back in the late 1970s and early 1980s: We must describe and analyze the behavior of complex information systems – he was a neuroscientist interested in vision – on several levels. I think we need to do the same with large language models, of which ChatGPT is now the most widely known example.
The company line on LLMs is that they work by statistically guided next-token prediction. I don’t doubt that, but I don’t find it very helpful either. It’s like saying a laptop computer works by executing a fetch-execute cycle. Well, yes it does, and so does every other digital computer. More to the point, that’s how every program is run, whether it’s the operating system, a word processor, a browser, a printer driver, etc. That’s what’s going on at the bottom level.
In the case of a word processor, the top-level processes include such things as: create a new document, save a document, cut text, past text, check the spelling, apply a style to a block of text, and so forth. Those are actions taken by the user. What happens between those actions and the bottom-level fetch-execute is defined by processes implemented in low-level and high-level languages. Each of those processes was programmed by a human programmer. So, in theory, we know everything about what’s going on in a word processor, or, for that matter, any other kind of program.
Things are quite different with LLMs. At the top-level users are issuing prompts and the LLM is responding to them. How does it do that? By issuing word after word after word based on the statistical model it built up during training. What happens in between the bottom level and the top level?
We don’t know. And, all too often, we don’t care. As long as it runs and does impressive things, we don’t care how it works.
Perhaps the fascinating work Peter Gärdenfors has being doing in semantics can help. He has been developing a geometric concept of meaning. His two books:
Conceptual Spaces: The Geometry of Thought, MIT 2000. The Geometry of Meaning: Semantics Based on Conceptual Spaces, MIT 2014.
On the symbolic level, searching, matching, of symbol strings, and rule following are central. On the subconceptual level, pattern recognition, pattern transformation, and dynamic adaptation of values are some examples of typical computational processes. And on the intermediate conceptual level, vector calculations, coordinate transformations, as well as other geometrical operations are in focus. Of course, one type of calculation can be simulated by one of the others (for example, by symbolic methods on a Turing machine). A point that is often forgotten, however, is that the simulations will, in general be computationally more complex than the process that is simulated.
The top-level processes of LLMs, such as ChatGPT, are operating at the symbolic level. Those processes are to be described by grammars at the sentence level and by various kinds of discourse models above the sentence level. My 3QD article presents some evidence about how ChatGPT structures stories. That behavior is symbolic and so has to be understood in terms of actions on and with symbols. See, e.g.:
Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy, Emergent linguistic structure in artificial neural networks trained by self-supervision, PNAS, Vol. 117, No. 48, June 3, 2020, pp. 30046-30054, https://doi.org/10.1073/pnas.1907367117.
Andrew M. Saxea, James L. McClelland, and Surya Gangulic, A mathematical theory of semantic development in deep neural networks. PNAS, June 4, 2019, Vol. 116, No. 23, 11537-11546, https://www.pnas.org/content/116/23/11537.
What’s going on at the subconceptual level, that is, the bottom level, and the intermediate level? For that I want to turn to Stephen Wolfram.
Wolfram on ChatGPT
Wolfram has written a long, quasi-technical, and quite useful article, What Is ChatGPT Doing … and Why Does It Work? He makes extensive use of concepts from complex dynamics in his account. For the sake of argument let’s say that’s what ChatGPT is doing at the bottom-level. Perhaps between these two we have Gärdenfors’”intermediate conceptual level” with its “vector calculations” and “other geometrical operations.”
Let’s scroll down through Wolfram’s article to the section, “Meaning Space and Semantic Laws of Motion.” Wolfram observes:
We discussed above that inside ChatGPT any piece of text is effectively represented by an array of numbers that we can think of as coordinates of a point in some kind of “linguistic feature space”. So when ChatGPT continues a piece of text this corresponds to tracing out a trajectory in linguistic feature space.
Given that the idea of a linguistic feature space is very general, Gärdenfors’ geometric semantics is certainly an account of something that can be called a “linguistic feature space.”
Wolfram has been working on example where he follows GPT-2 from the prompt: “The best thing about AI...” After having shown illustrations of a feature space, he asks:
So what about trajectories? We can look at the trajectory that a prompt for ChatGPT follows in feature space—and then we can see how ChatGPT continues that (click on the diagrams to embiggen):
There’s certainly no “geometrically obvious” law of motion here. And that’s not at all surprising; we fully expect this to be a considerably more complicated story. And, for example, it’s far from obvious that even if there is a “semantic law of motion” to be found, what kind of embedding (or, in effect, what “variables”) it’ll most naturally be stated in.
In the picture above, we’re showing several steps in the “trajectory”—where at each step we’re picking the word that ChatGPT considers the most probable (the “zero temperature” case). But we can also ask what words can “come next” with what probabilities at a given point:
And what we see in this case is that there’s a “fan” of high-probability words that seems to go in a more or less definite direction in feature space. What happens if we go further? Here are the successive “fans” that appear as we “move along” the trajectory:
Keeping in mind that this is a space of very high dimensionality, are those "more or less definite directions in feature space" the sort of thing we'd find in Gärdenfors' conceptual spaces? Here’s what he says in The Geometry of Meaning (p. 21):
A central idea is that the meanings we use in communication can be described as organized in abstract spatial structures that are expressed in terms of dimensions, distances, regions, and other geometric notions. In addition, I also use some notions from vector algebra.
That surely sounds like it’s in the right ballpark. That does not mean, of course, that it is. But surely it is worth investigating.
The role of speculation in scientific investigation
The deep learning community puts on a great show of intellectual rigor. And in some ways, it is not merely a show. The rigor is there. The technology wouldn’t work as well as it does if it weren’t rigorous in some substantial way.
But there is little rigor that I can see in the way they think about language and texts. I see relatively little knowledge about linguistics, psycholinguistics, and related areas of cognitive science.
Nor is there much interest in figuring out what happens with those 175 billion parameters as they generate text. There is some work being done on reverse engineering (aka mechanistic interpretability) the operations of these engines. There needs to be more, much more – see this article by David Chapman for suggestions, Do AI as science and engineering instead.
Speculation is a necessary part of this process. In order to go boldly where none have gone before you are going to have to speculate. It can’t be helped. Sooner or later some speculation will turn out to be correct, that is, it will be supported by evidence. There is no way to determine that ahead of time. But make them as rigorous and detailed as you can. Speculation must be clear and crisp, otherwise it is not a reliable guide for thought.
“That which gives rise to the next token” is a space far exceeding the scope of next token itself:
- What’s in output so far - Embedding space of 1000s dimensions - Dynamics of “semantic motion”(@stephen_wolfram) in subconceptual space of 100s billions parameters
We need a theory for such “dynamics of subsemantics-subconceptual motion” such that it can afford us ideas similar to or better than transformers and higher-order programs running on them (eg RLRF, CAI, ICL, CoT).
But it may also be flat-out wrong. We’ll see when we get a better idea of how
inference works in the underlying language model.
* * * * *
Yes, I know that ChatGPT is trained by having it predict the next word,
and the next, and the next, for billions and billions of words. The result of
all that training is that ChatGPT builds up a complex structure of weights on
the 175 billion parameters of its model. It is that structure that emits word
after word during inference. Training and inference are two different
processes, but that point is not well-made in accounts written for the general
public.
Let's get back to the main thread.
I maintain, for example, that when ChatGPT begins a story with the words “Once
upon a time,” which it does fairly often, that it “knows” where it is going
and that its choice of words is conditioned on that “knowledge” as well as
upon the prior words in the stream. It has invoked a ‘story telling procedure’
and that procedure conditions its word choice. Just what that procedure is,
and how it works, I don’t know, nor do I know how it is invoked. I do know,
that it is not invoked by the phrase “once upon a time” since ChatGPT doesn’t
always use that phrase when telling a story. Rather, that phrase is called up
through the procedure.
Consider an analogy from jazz. When I set out to improvise a solo on, say, “A
Night in Tunisia,” I don’t know what notes I’m going to play from moment to
moment, much less do I know how I’m going to end, though I often know when I’m
going to end. How do I know that? That’s fixed by the convention in place at
the beginning of the tune; that convention says that how many choruses you’re
going to play. So, I’ve started my solo. My note choices are, of course,
conditioned by what I’ve already played. But they’re also conditioned by my
knowledge of when the solo ends.
Something like that must be going on when ChatGPT tells a story. It’s not
working against time in the way a musician is, but it does have a sense of
what is required to end the story. And it knows what it must do, what kinds of
events must take place, in order to get from the beginning to the end. In
particular, I’ve been working with
stories where the trajectories have five segments: Donné, Disturb, Plan, Execute, Celebrate. The whole trajectory is ‘in
place’ when ChatGPT begins telling the story. If you think of the LLM as a
complex dynamical system, then the trajectory is a valley in the system’s
attractor landscape.
Nor is it just stories. Surely it enacts a different trajectory when you ask
it a factual question, or request it to give you a recipe (like I recently
did, for Cornish pasty), or generate some computer code.
With that in mind, consider a passage from a recent video by Stephen Wolfram
(note: Wolfram doesn’t start speaking until about 9:50):
Starting at roughly 12:16, Wolfram explains:
It is trying write reasonable, it is trying to take an initial piece of
text that you might give and is trying to continue that piece of text in a
reasonable human-like way, that is sort of characteristic of typical human
writing. So, you give it a prompt, you say something, you ask something,
and, it’s kind of thinking to itself, “I’ve read the whole web, I’ve read
millions of books, how would those typically continue from this prompt
that I’ve been given? What’s the reasonable expected continuation based on
some kind of average of a few billion pages from the web, a few million
books and so on.” So, that’s what it’s always trying to do, it’s aways
trying to continue from the initial prompt that it’s given. It’s trying to
continue in a statistically sensible way.
Let’s say that you had given it, you had said initially, “The best think
about AI is its ability to...” Then ChatGPT has to ask, “What’s it going
to say next.”
I don’t have any problem with that (which, BTW, is similar to a passage near
the beginning of his recent article,
What Is ChatGPT Doing … and Why Does It Work?). Of course ChatGPT is “trying to continue in a statistically sensible way.”
We’re all more or less doing that when we speak or write, though there are
times when we may set out to be deliberately surprising – but we can set such
complications aside. My misgivings set in with this next statement:
Now one thing I should explain about ChatGPT, that’s kind of shocking when
you first hear about this. Is, those essays that it’s writing, it’s
writing at one word at a time. As it writes each word
it doesn’t have a global plan about what’s going to happen. It’s simply saying “what’s the best word to put down next based on what
I’ve already written?”
It's the highlighted passage that I find problematic. That story trajectory
looks like a global plan to me. It is a loose plan, it doesn’t dictate
specific sentences or words, but it does specify general conditions which are
to met.
Now, much later in his talk Wolfram will say something like this (I don’t have
the time, I’m quoting from his paper):
If one looks at the longest path through ChatGPT, there are about 400
(core) layers involved—in some ways not a huge number. But there are
millions of neurons—with a total of 175 billion connections and therefore
175 billion weights. And one thing to realize is that every time ChatGPT
generates a new token, it has to do a calculation involving every single
one of these weights.
If ChatGPT visits every parameter each time it generates a token, that sure
looks “global” to me. What is the relationship between these global
calculations and those story trajectories? I surely don’t know.
Perhaps it’s something like this: A story trajectory is a valley in the LLM’s attractor landscape. When it tells a story it enters the valley at one end and continues through to the end, where it exits the valley. That long circuit that visits each of those 175 billion weights in the course of generating each token, that keeps it in the valley until it reaches the other end.
I am reminded, moreover, of the late Walter Freeman’s conception of consciousness as
arising through discontinuous
whole-hemisphere states of coherence
succeeding one another at a “frame rate” of 6 Hz to 10Hz – something I discuss
in “Ayahuasca Variations” (2003). It’s the whole hemisphere aspect that’s striking (and somewhat
mysterious) given the complex connectivity across many scales and the
relatively slow speed of neural conduction.
LLMs are *not* simply “predicting the next statistically likely word”,
as the author says. Actually, nobody knows how LLMs work. We do know how
to train them, but we don’t know how the resulting models do what they
do.
Consider the analogy of humans: we know how humans arose (evolution via
natural selection), but we don’t have perfect models of how humans worked;
we have not solved psychology and neuroscience yet! A relatively simple
and specifiable process (evolution) can produce beings of extreme
complexity (humans).
Likewise, LLMs are produced by a relatively simple training process
(minimizing loss on next-token prediction, using a large training set from
the internet, Github, Wikipedia etc.) but the resulting 175 billion
parameter model is extremely inscrutable.
So the author is confusing the training process with the model.
It’s like saying “although it may appear that humans are telling jokes and
writing plays, all they are actually doing is optimizing for survival and
reproduction”. This fallacy occurs throughout the paper.
This is the why the field of “AI interpretability” exists at all: to probe
large models such as LLMs, and understand how they are producing the
incredible results they are producing.
I don’t have any reason to think Wolfram was subject to that confusion. But I
think many people are. I suspect that the general public, including many
journalists reporting on machine learning, aren’t even aware of the
distinction between training the model and using it to make inferences. One
simply reads that ChatGPT, or any other comparable LLM, generates text by
predicting the next word.
2.21.23: The conversation at LessWrong has been very helpful. Here's a reply I just left there:
Quick reply, after doing a bit of reading and recalling a thing or two: In a 'classical' machine we have a clean separation of process and memory. Memory is kept on the paper tape of our Turing Machine and processing is located in, well, the processor. In a connectionist machine process and memory are all smushed together. GPTs are connectionist virtual machines running on a classical machine. The "plan" I'm looking for is stored in the parameter weights, but it's smeared over a bunch of them. So this classical machine has to visit every one of them before it can output a token.
So, yes, purely next token prediction. But the prediction cycle, in effect, involves 'reassembling' the plan each time through.
To my mind, in order to say we "understand" how this puppy is telling a story, we need to say more than it's a next-token-prediction machine. We need to say something about how that "plan" is smeared over those weights. We need to come up with concepts we can use in formulating such explanations. Maybe the right concepts are just laying scattered about in dusty old file cabinets someplace. But, I'm thinking this is likely, we have to invent some new ones as well.
Wolfram was trained as a physicist. The language of complex dynamics is natural to him, whereas it's a poorly learned third or fourth language for me, So he talks of basins of attractors and attractor landscapes. As far as I can tell, in his language, those 175B parameters can be said to have an attractor landscape. When ChatGPT tells a story it enters the Story Valley in that landscape and walks a path through that valley. When its done with the story, it exits that valley. There are all kinds of valleys (and valleys within valleys (and valleys within them)) in the attractor landscape, for all kinds of tasks.
FWIW, the human brain has roughly 86B neurons. Each of those is connected with roughly 10K other neurons. Those connections are mediated by upward of a 100 different chemicals. And those neurons are surrounded by glial cells. In the old days researchers thought those glial cells were like packing peanuts for the neural net. We now know better and are beginning to figure out what they're doing. Memory is definitely part of their story. So we've got to add them into the mix. How many glial cells per neuron? There might be a number in the literature, but I haven't checked. Anyhow, the number of parameters we need to characterize a human brain is vast.
