Sabrina Hossenfelder looks at Claude 3.5 Haiku and discovers that it lacks self-awareness

In this video Hossenfelder examines a paper recently published by Anthropic: On the Biology of a Large Language Model. She has a number of interesting observations. After reiterating that the underlying LLM is in the business of next token prediction, she reviews an example of arithmetic reasoning starting at about 1:54. After following the example more or less step by step she observes (c. 2:45):

It’s basically a heuristic text-based approximation. It’s doing maths by free-associating numbers until the right one just sort of vibes into place. But here is the kicker. If you ask Claude how it arrived at that result it says “I added the ones (6+9=15), carried the 1, then added the tens (3+5+1=9), resulting in 95.” Which is not what it did, not even remotely. It answers this question separately, giving you again, a text prediction for the answer. And I think that this shows very clearly that Claude has no self-awareness. It doesn’t know what it's thinking about. What it tells you it’s doing is completely disconnected from what it’s actually doing.

That makes sense to me. Claude really "doesn’t know what it's thinking about." Why would we think otherwise? Well, I suppose, because it tells us what it did and we treat that statement as equivalent to what humans do when they report how they accomplished some task. The thing is, humans don't necessarily know what they're doing either.

Consider and example from Piaget's 1976 book, The Grasp of Consciousness. Here's how I reported that experiment in a paper I published some years ago, First Person: Neuro-Cognitive Notes on the Self in Life and in Fiction (2000):

Let us begin with an experiment conducted by Jean Piaget as part of an investigation into consciousness. In this experiment children were asked to crawl for about 10 meters and then to describe what they had just done (Piaget, 1976, pp. 1 ff.). Four-year olds generally said either that they first moved one arm, then the other, then one leg, then the other, or legs first and then arms. Piaget called this a Z pattern. That is not, in fact, how any of them actually crawled. What they actually did was either to first move one arm, then the opposite leg, then the other arm, then the opposite leg, or the same pattern beginning with a leg. Piaget called this an X pattern. It isn't until children are seven or older that they can describe this X pattern. [...]

What is striking is that the younger children's verbal account of such a basic act is simply wrong. In order to execute the crawl there must be some brain tissue devoted to schemas regulating the appropriate actions; for crawling isn't a spinal reflex. But those regulating schemas must in some way be distinct from the schemas underlying the younger children's verbal accounts, otherwise those accounts would be more accurate. My first point is simply that we are here dealing with two different neural schemas for the same action and that one of them is grossly simplified and thus incapable of actually regulating the behavior it represents.

There are many ways in which the behavior of these four-year olds is quite different from Claude's, but I see no need to list them all. They're obvious enough. The basic point, as I say in that second paragraph, is that the young children got it wrong. And, while older children do get it right, there is a separate body of research that shows that the kind of behavior those four-year olds exhibit is common in various domains.

That research is about what is called the introspection illusion, which Wikipedia characterizes as follows:

The introspection illusion is a cognitive bias in which people wrongly think they have direct insight into the origins of their mental states, while treating others' introspections as unreliable. The illusion has been examined in psychological experiments, and suggested as a basis for biases in how people compare themselves to others. These experiments have been interpreted as suggesting that, rather than offering direct access to the processes underlying mental states, introspection is a process of construction and inference, much as people indirectly infer others' mental states from their behaviour.

Notice that last statement, “introspection is a process of construction and inference.” Isn't that what Claude was doing? It provides a plausible account of its activity, not on the basis of some examination of that activity, but rather, based on what it knows about how arithmetic is (supposed to be) done. Claude doesn't have introspective awareness of what it is going, but then neither do humans, at least not in some wide variety of cases.

Let’s return to Hossenfelder, who goes on to assert: “I’d say that self-awareness is a precondition for consciousness.[1] So this model is nowhere near conscious.” I agree with her that Claude is not conscious, but not for the reason she gives. Those four-year olds in Piaget's experiments were certainly conscious, but they lacked (a certain kind of) self-awareness. I'm inclined to think that self-awareness and consciousness are distinct mental phenomena. On consciousness, I favor the view expressed by William Powers in his 1973 book, Behavior: The Control of Perception. Powers’s account is subtle, more than I can explain here (I do explain it in a post from 2022). Suffice it to say that Powers’s account is grounded in the behavior architecture of the brain. LLMs simply don't that the required architecture. They may well talk as though they’re conscious, but they’re just faking it. It’s empty talk. 

* * * * *

[1] FWIW, I suspect that this mistaken belief is widespread.