Coming to terms with The God Test, Reset? – [GT-4]

When I first started writing about Robert Wright’s The God Test I told myself: “I’m going to end up publishing a review in 3 Quarks Daily. I’ve got lots of time. I’ll write a series of blog posts dealing with the book which I can reference in the review proper. I’ve got a couple, three weeks. Piece of cake.” I wake up this morning, realize the review’s due this coming Sunday, I’ve only written four blog posts [the ones tagged with “GT-X”, the others are along for the ride], not yet finished the book, and ... Yikes! I’m not ready.

One problem is that I can’t help but see the book on two levels. On the one hand it’s a book about the current AI “revolution.” But it’s also an example of how our society is coming to grips with the technology. In the first case I’m assessing Wright’s work, the work of an individual journalist. In the second case I’m looking at the culture and society of which Wright is a member. That’s very different.

Starting at that first level, how well does Wright explain the technology and its evolution? That’s one level. But it’s also about what AI portends for the future and how we should deal with it. Well, we don’t really know the future, do we? These things are hard to predict. Moreover, surely what happens depends, in large extent, on how we decided to deal with it, no? And that’s why Wright is writing the book, to influence how we deal with the technology. That depends, in turn, on the capabilities of the technology itself, which we don’t really know, do we? Which is hotly debated. So in that context, on THAT level, the book becomes an example of how the culture, our culture (whatever that culture is, however you delimit it) is dealing with the technology. But because not even the experts know how the technology works, it’s a mess, and I can’t separate the two levels: 1) What’s he say about the technology, 2) Why’s he saying it (this is the society/culture level)?

Do you find that previous paragraph difficult and confused/confusing? Welcome to the club.

In a way what I want to do is factor the book into two components, one which I attribute to Wright the individual journalist, and the other to the culture in which he’s working. The culture provides him the conceptual tools he’s working with and the book is what he makes of them. What I really want to say is that Wright’s work is good – I like him, but he’s got an aggravating and invasive interview style which drives me bonkers sometimes – but the culture has given him lousy tools. The upshot: I have lots of problems with the book. But it’s NOT HIS FAULT. It’s the culture’s fault. It’s out fault. WE’RE NOT READY.

[And I’ve not even finished the book!]

Wright is a journalist. He’s got a number of beats. I first encountered with in The New Republic, where he was writing about politics and international affairs. He still does that. But he also wrote a book about evolutionary psychology. Which I’ve not read. That’s become a standard kind of book, a journalist explaining and interpreting a technical subject for a general audience. Back in 1984 he wrote an article on AI for The Wilson Quarterly, “Thinking Machines,” which he mentions in The God Test. I’ve read it, not back then, but now. It’s pretty good. But things have changed a lot since then. The technology Wright wrote about in that article could not have produced ChatGPT and the cultural explosion that’s followed. The seeds were there, but there was no way to predict... And it’s that explosion that motivated The God Test.

The thing is, the experts who created that technology, they knew how it worked. So Wright was in the business of explaining expert knowledge for a lay audience. The current technology is quite different. They experts know how they create the large language models that power these chatbots, and Wright talks about that. But they don’t know how those language models work. On that score, there’s no expert knowledge for Wright to interpret and explain. There’s just a black hole, a Rorschach blot.

Framing my discussion of The God Test, Part 1: Rorschach, reason, and whaling – [GT-3]

I’ve got to bite the bullet: I’m just going to have to go through a bunch of (preliminary) stuff before I can really engage with The God Test. My current target is to be in a position to publish a proper review of the book in 3 Quarks Daily for the week of August 9.

Rorschach Recap

I want start by recapping the Rorschach metaphor I introduced in the previous post, More on how I’m approaching The God Test – Rorschach! [GT-2]. What I like about it is that has a shape, there’s something there, but it’s not clear what. So we have little choice but to project onto it in order to (begin to) make sense of it.

First: It is a new kind of thing, an artifact we can converse with in an open-ended and natural way. The steam engine was the same kind of thing. It was an inanimate object that moved over the surface of the earth under its own power. Previously only animals (& humans as animals) had that power. So it becomes an iron horse. Just what are AIs? What’s their nature? That’s one thing.

Second: How it works is opaque. We know how to create large language models (LLMs), but we don’t know how they work. That’s new. We may not have understood the deep physics of the steam engine, but we certainly knew how they worked.

Third: We don’t know what they portend for the future. To some extent this is a function of the first two: How can we, how should we, interact. But it is also a function of the future, which is undetermined. We just don’t know.

Rhetorical force over reason

This is an argument I made in the first working paper I published after the release of ChatGPT in November of 2022: ChatGPT intimates a tantalizing future; its core LLM is organized on multiple levels; and it has broken the idea of thinking (February 6, 2023).

What do I mean by that, has broken the idea of thinking? Prior to ChatGPT it was obvious that humans could think and computers could not. [Yeah, I know, there’s Deep Blue defeating Kasparov in chess. That just changes the dates, not the argument.] The difference in performance was so obvious that the fact that we don’t really know how humans think wasn’t much of an issue. Now it is. Sure, we can still say that we can think and the AI’s can’t, but that’s just a line and without good explanations on both sides of the line, it seems a bit arbitrary, if not desperate.

I made a particular argument about Searles’ (in)famous Chinese Room thought experiment. I read it when it was first published in Brain and Behavioral Science in 1980. I wasn’t impressed. Why not? He didn’t say anything about any of the techniques used in AI or computational linguistics (CL). How could anyone possibly take that seriously?

He talked about intention, that’s how. Meaning requires intention and only living things can have intention, a remark he made at the end of the article. Without intention the most you get is syntax, but no meaning. Searle could get away with that because, in the first place, the concept of intention has a long history within philosophy – it has a subtle meaning, but that can wait for a later post – and so philosophers, his main audience, were comfortable with it. That’s one thing.

But there’s something more important, something that we can see only in retrospect, and that’s the simple fact computers very obviously could not translate from Chinese into English or into any other language. That difference carried tremendous weight. We don’t have a subtle behavioral difference between computers and humans that requires a subtle and sophisticated argument. To a first approximation, almost any argument would do. As far as I was concerned, “intention” was just a fancy word for something we don’t understand. But that’s not an argument anyone needs to take seriously. The behavioral distance is quite sufficient to carry the argument for those who insist that computers can’t and will never be able to think like humans.

Now the behavioral evidence has changed. Sure, differences remain, but the evidence is shifting. The old arguments remain and those who believed them still do so, but it’s getting harder. The need for explicit arguments grounded in explicit accounts of computers, and also brains, is growing.

Whaling and expertise

What’s an expert in machine learning and LLMs actually expert in? For some time now I’ve been arguing that investing in AI is like investing in a whaling venture where the captain and crew of the ship know all there is to know about the ship and how to handle it but know little or nothing about whales and their behavior and about navigating around the Cape Horn and in the South Pacific, where the whales live. What are the chances of that voyage being successful? Not very good.

The people who have created the current AI technology are like that captain and crew. The know how to sail the ship. But they don’t know much about language or cognition. They don’t actually know much about the human mind. Here my point is not about the fact that the models are opaque, but that human language and cognition are highly structured and they don’t believe that one needs to know (much of) anything about that not only to build AI but to make confident prediction about the future of AI.

Gary Marcus, Subbarao Kambhampati, and others have been consistently arguing that, yes, the current technology is remarkable, but we are going to have to adopt classical symbolic techniques if we are to fully develop the technology so that we have accurate and safe systems. Marcus is arguing from his knowledge of human language and cognition. As far as I can tell, Wright doesn’t take that seriously. I know that he had Marcus on his NonZero podcast, and that he lists Marcus in his acknowledgements, but that he doesn’t discuss Marcus’s ideas. I conclude that he doesn’t take that line of argument seriously.

That’s a mistake, but this is not the place to make my own arguments on this issue. My point is simply that expertise in AI is no generally construed to encompass knowledge of, expertise in, human cognition and language. I can’t see how that is going to work out well in the future.

[Note: If you’re curious about my views, on this subject, read the article linked in the first paragraph of this section. My views all over the place here at New Savanna, particularly around the work of the mathematician Miriam Yevick. Also, check out the experimental work I’ve done with LLMs.]

More on how I’m approaching The God Test – Rorschach! [GT-2]

I’m still trying to figure out how to approach Robert Wright’s The God Test.

How LLMs work

In my previous post – How will I handle The God Test? [GT-1] – I expressed misgivings about how Wright explains the technology. Those misgivings haven’t disappeared. However, Bert Idem has published a useful review at Finite Ape in which he addresses some of those issues in detail. Specifically:

Now, about the history of AI, the story he tells is actually great and it is certainly more than what most non-technical people know about LLMs. However, there are three places where I think the framing goes wrong or at least leaves out context that matters:

• LLMs did not discover the meanings of words on their own by accident. They were designed on top of ideas from older models that were specifically trained to learn the meanings of words.
• Similarly, computer vision models didn’t find out how to “view” an image like we do. Instead, the classical CNN models were heavily inspired by biological vision itself.
• LLM weight training is simply gradient-based optimization and the process has nothing to do with evolution. Of course, we can make a parallel between any kind of change and evolution but then, in that sense, everything evolves and it is not useful to talk about evolution.

I agree with Idem on those three issues, not so sure about the history part. While I may return to some of these issues later on, this will serve as a place holder.

A Rorschach test

There’s something else going on, but I’m not quite sure how to conceptualize it. It seems to me that AI is functioning something like a Rorschach test which, as you may know, is a psychological instrument intended to elicit (potentially) revealing responses from a person. It’s a projective test.

A person is shown a series of ink blot images, like this one (generated by ChatGPT):

They are asked what that they see in the image, what it means to them. Since the image is, though not formless, its form is not that of any specific animal, vegetable, mineral, person, or anything else. It’s just a blot. Whatever the person says about the blot, however they interpret it, that must reveal something about them. Why? Because whatever they see in the blot, isn’t really there.

Broadly and crudely speaking, AI has become something of a cultural Rorschach test.

Understanding computers & LLMs

Until ChatGPT was released in late November of 2022, most people knew very little to nothing about AI. Oh, they may have seen “intelligent” computers and robots in science fiction movies, but that’s science fiction and only tangentially related to AI considered as a line of research dating back to the 1950s. Many people would have heard about IBM’s Deep Blue beating Gary Kasparov in chess in 1997 and then, in 2011, when IBM’s Watson beat Ken Jennings and Brad Rutter in Jeopardy. Those were real AI systems, standing on research extending back decades, but as far as most people were concerned, they were one-off PR stunts. Just how they worked, who cares? They’re computers, and computers are magic, no?

As far as most of us are concerned, computers are magic. Somewhere “out there” someone knows how these things work, but we don’t need to know any of that. It’s complicated, but computers do what they’re programmed to do, no? Yes, but not LLMs.

And that’s the tricky part. LLMs, large language models, aren’t like other computer systems. They aren’t programmed in the way that word processors, photo editors, or phones are programmed. LLMs aren’t programmed at all, not in the ordinary sense of programming – something I may or may not get into in a later post. As far as most users are concerned, how ChatGPT, or Claude, or Gemini work, that’s no more interesting than how a word processor works. It just does. It’s more magic.

But if you have a strong philosophical streak, if you are interested in the mind, in technology, in the technology in the future, then you may not be content with writing LLMs off as just another kind of magic. You want to know what’s going on inside, 1) because you want to know (curiosity), and 2) because you want to know how the technology is going to develop in the future (engagement). Now things get interesting? Why? Because even the people who have created the technology don’t know how it works.

Oh, they know how the transformer program works. That’s the program that creates the language model. It creates the model by performing a (certain kind of) statistical analysis of a huge body of texts, effectively the entire internet. When a person prompts the model with some statement, the model responds by a statement of its own. No one know just how the model does that. That’s a mystery, a deep black hole in the technology ecosystem.

AI as a Rorschach test

If you aren’t content to believe in magic, then you have to come up with something to fill that black hole in your, in our, understanding. This is where the Rorschach aspect of AI reveals itself. To a first approximation, what each of us uses to paper over that black hole has as much to do with ourselves as with AI.

Why do I say, “To a first approximation”? It’s a rhetorical device to get things started. It puts us all in the same boat, despite our different backgrounds. However, whatever LLMs are, they are not magic. It is possible, in principle, to construct a technical account of what they’re up to, but no one knows how to do that, yet. Not even the people in the AI labs who create these beasts.

Those of us who are trying to figure out how LLMs work have widely varying backgrounds. In particular, we have widely varied technical backgrounds and we bring those backgrounds to bear when we think about what LLMs are doing. Those backgrounds influence how we interpret the AI-blot. Wright is a journalist with a wide range of interests, including politics, international affairs, evolutionary psychology, cultural evolution, and Buddhism. As far as I can tell there isn’t much there that’s directly relevant to understanding the mechanisms of LLMs, but he’s done a lot of reading and talked with a lot of experts to fill in the gaps.

My background is quite different. While I happen to know quite a bit about cultural evolution, cognitive psychology, neuroscience, and various other things, my background in computational semantics puts me much closer to LLMs than Wright’s knowledge of evolutionary psychology puts him. Still, like him, I’ve done a lot of reading and talked with experts. In particular, I’ve been collaborating with Ramesh Viswanathan for the last three years. He’s an expert in machine vision Goethe University Frankfurt. He’s got a background in mathematics and AI that I don’t have. Still, there are things he doesn’t know, things he’s trying to figure out. 

We are all making stuff up.

To some extent, then, AI is a Rorschach test about how beliefs about the human mind, and human nature. When we try to figure out how the LLM is working we’re also, if only implicitly, trying to figure out how we work, internally, as well. The whole discourse about AL alignment is as much a discourse about us as it is about AI. 

The Future

And even if we knew much more about how LLMs work internally we still wouldn’t know how the technology will develop in the future. We? You, me, Robert Wright, Ramesh Viswanathan, Gary Marcus, Tyler Cowen, Geoffrey Hinton, Sam Altman, Dario Amodei, Nick Bostrom, Eliezer Yudkowsky, all of us who are trying to figure it out. We don’t know what will happen. That’s where we’re projecting like mad. We’re hallucinating, to borrow a term from AI-speak. 

Thus AI is also a Rorschach test for our visions of the future. When we imagine the future of AI, we’re also imagining our future. Like the two sides of a coin, the two cannot be separated. 

The tricky part, the important part, is that the future development of AI is not predestined. It depends on the choices we make, now and in the near future. We can easily and often do imagine things that will not be possible because that’s just not how the world works. But the laws of how the world works are open to a wide range of possibilities. The boundary between the possible and the impossible is fuzzy at best.

Where, and how, does Wright draw that boundary? Perhaps that’s what I’ll be trying to figure out.

More later.