If a million monkeys are typing out code, how long before they produce a coherent program that does something useful?

Yesterday Eric Jang and Santiago Renteria had an interesting conversation in the Twitterverse about a recent blog post in which Scott Aaronson discussed DeepMind’s AlphaCode paper. Aaronson seemed overly sensitive about recent criticisms of that work. Here's what he said:

Yes, I realize that AlphaCode generates a million candidate programs for each challenge, then discards the vast majority by checking that they don’t work on the example data provided, then still has to use clever tricks to choose from among the thousands of candidates remaining. I realize that it was trained on tens of thousands of contest problems and millions of solutions to those problems. I realize that it “only” solves about a third of the contest problems, making it similar to a mediocre human programmer on these problems. I realize that it works only in the artificial domain of programming contests, where a complete English problem specification and example inputs and outputs are always provided.

Forget all that. Judged against where AI was 20-25 years ago, when I was a student, a dog is now holding meaningful conversations in English. And people are complaining that the dog isn’t a very eloquent orator, that it often makes grammatical errors and has to start again, that it took heroic effort to train it, and that it’s unclear how much the dog really understands.

Well, yes. What AlphaCode does really is remarkable. But, Aaronson goes on to note in his next paragraph:

It’s not obvious how you go from solving programming contest problems to conquering the human race or whatever, but I feel pretty confident that we’ve now entered a world where “programming” will look different.

And that too. Forget about "conquering the human race or whatever." That's self-indulgent tech-bro nonsense. But changing how code gets written? Sure, that's happening.

The thing is, we're moving into new and unexplored territory – we have been for decades now – and we don't know how to evaluate what we discover. As I said the other day, the space seems to get larger as we move into it. We have no metrics, not even informal ones. So the tendency seems to be that we go binary: Either we trivialize the result (that's nothing) or we exaggerate its significance (the Singularity is coming!).

Well, no, it's not either of those things. It's somewhere in between. Just where, we haven't a clue. I found myself in that situation when GPT-3 came out. I dealt with it by writing 14K words: GPT-3: Waterloo or Rubicon? Here be Dragons. That paper is somewhere between nothing to see here and it's alive! But where? How do we chart this new territory?

Postscript:

GPT-3 achieves with roughly 12 cybernetic monkeys in finite time what would have taken aeons with a markovian macaque army.

— Santiago Renteria (@renatrigiorese) April 12, 2022

Postscript 2: Matt Yglesias has a column that inadvertently illustrates the problem, The case for Terminator analogies. He thinks the AI alignment problem is real, but that Skynet in the Terminator films is not what the theorists of alignment think about. He's right about that. In the course of developing his argument he says:

A lot of smart people like to argue about the timelines here, so as someone who is not so smart, I would just make two observations: (1) we have a clear precedent for very rapid progress in the field of computers, and (2) AI does appear to be progressing very rapidly recently. A French AI beat eight world champions at bridge two weeks ago. Last week, OpenAI released DALL-E-2, which draws images based on natural language input, and Google released PaLM, which seems like a breakthrough in computer reasoning.

There's the problem. Yes, there's been rapid progress – the bridge example is particularly interesting. But how do we calibrate it? This is going to keep cropping up.

One issue is that the idea of progress seems like linear movement along a single dimension. Whatever is going on, it is multidimensional. Perhaps we should start thinking in terms of adding new dimensions to the space rather than simply advancing on one. 

FWIW, back in the 1990s David Hays undertook an extensive review of the (largely empirical) literature on cultural complexity: The Measurement of Cultural Evolution in the Non-Literate World. He started out looking for one variable underlying the various metrics that had been proposed and explored. He concluded that that was not possible, at least not at the time. Instead he identified eleven aspects, as he called them, for socio-cultural organization. In effect, he argued that cultural complexity involves eleven dimensions. He rated each of several hundred societies on those aspects.

The machine in my mind: Lessons learned

Several days ago I recounted how my thinking on the distinction between minds and machines had evolved: The machine in my mind, my mind on the machine: Will we ever build a machine to equal the human brain? This is a conclusion to that. My point was simply that I have always believed that we cannot not make an artificial intelligence as powerful as a real human brain. Some other things are worth noting.

You can’t predict the course of intellectual development

This is well-known. But it’s one thing to know it from knowing intellectual history. Knowing it from having lived it is different.

Not being able to predict means you can’t project the future evolution of current lines of research. It also means that you can’t predict what new possibilities will appear out of nowhere. At the time in 1970s when I was predicting the development of a computer system capable to reading a Shakespeare play, I was not predicting that one day I would be thinking seriously about human origins and offer a concrete hypothesis. But that’s what I did in my book about music: Beethoven’s Anvil (2001). Nor, for that matter, did I imagine that I would one day thinking about the metaphysical structure of the world as a high-dimensional space constructed by an AI engine.

The space gets larger

As we come to know more, the space gets larger. As a crude analogy, here’s something I stuck into my post on the meaning of “understand”:

Let's assume for a minute that we're going to rate understanding on a scale, say, from 1 to 10. GPT-3 rates, say, 3. Along comes Pathways and it's clearly better than GPT-3. Where does it go? 4? 5? 6?

No.

Given that considerable distance remains between Pathways and humans, I'd say that a 1-10 scale is insufficient. Let's make it 1-100. GPT-3 goes in at 23 and Pathways at, say, 38.

That is to say, each time one of these remarkable results comes in I think it enlarges our sense of the measure space. Maybe it even forces us to start adding dimensions. It just makes the world larger.

The space of possible models for human intelligence, not to mention models for artificial intelligence, is now much larger than it was in the 1970s. Will it keep getting larger and larger as our knowledge grows, or will the time come when we have bounded the space? How will we know?

The very idea of such a space, of course, implies computational understanding. It certainly isn’t a physical space. What kind of space is it? Conceptual? Imaginary? The very fact that we can conceive of such a space, imagine it, depends on computation. Computer programs can search spaces, at least in AI. But I don’t think the idea. Is due to AI. When I took a course in computer programming during my undergraduate years at Johns Hopkins we wrote a program to search for values on a hidden surface. [For that matter, we also did a program to play tic-tac-toe.]

My basic conceptual ontology remains

As far as I can tell, despite the collapse of my dream for that Shakespeare-reading computer system, my basic underlying conceptual ontology remains the same. That’s why I continue to believe that we will not be able to fully simulate a human brain in an artificial system.

We don’t have a language, a conceptual system, in which to describe that ontology, though we worked on it in Hays’s research group – see the concept of assignment in this working paper, Ontology in Cognition: The Assignment Relation and the Great Chain of Being, this technical report for the Center for Integrated Manufacturing at RPI, Ontology in Knowledge Representation for CIM, and this encyclopedia article, Ontology of Common Sense. I don’t know just when and how my ontology developed, but I’d point to my undergraduate years at Johns Hopkins:

• Lévi-Strauss on myth
• taking a course in computer programming
• encountering Chomsky’s ideas about language
• reading Karl Pribram on neural holography in Scientific American in 1969, which linked up with Lévi-Strauss’s idea of the totemic operator
• encountering “Kubla Khan”

It was while working on a master’s thesis about “Kubla Khan” that all those things came together. That’s the underlying ontology that was in place when I met David Hays at Buffalo in spring on 1974. I tell that story in an article I published originally in 1975 and have since updated and revised, Touchstones.

* * * * *

What do I mean by “basic underlying conceptual ontology?” Think of a set of building blocks, Lego pieces, Erector set components, or, for that matter, the various components that go into the construction of, say, actual buildings, whatever. There is a finite set of distinct different types of objects in these various collections. That set of types is your ontology. This set of types places constraints on what you can build. But what you can actually build depends on your imagination and determination, plus, of course, having enough tokens of each type to complete the job.

My set of conceptual Lego pieces was complete by the time I completed my master’s thesis on “Kubla Khan” in 1972. It was rich enough that I was able to learn Hays’s computational semantics and, on that basis, imagine Prospero, the system that could “read” Shakespeare. When the possibility of actually constructing Prospero disappeared, the set of conceptual Lego pieces – my conceptual ontology – remained unchanged. But my sense of what one can build with those pieces changed.

When I began (email) conversations with Walter Freeman about the complex dynamics of the nervous system, I was able to do so with that set of conceptual Lego pieces (ontology) – though, keep in mind, I don’t command the underlying mathematics and so have to work analogy and metaphor. That same conceptual ontology has allowed me to conceive of attractor nets, networks of logical operators over attractors in various attractor landscapes, where each landscape corresponds to a neurofunctional area in the brain. When I began thinking seriously about deep learning and artificial neural nets, I did so in terms those ontological primitives. They allowed me to see, both that GPT-3 represents a conceptual advance, and that such technology is not sufficient in itself.

Remember, finally, that that conceptual ontology took shape through investigating the form and meaning of “Kubla Khan.” That ontology was ‘designed,’ if you will, to encompass a rich example of verbal artistry. It ranges over neurons, logical operators, poems, and more. 

What has happened over the course of my career is the my sense of what can be built within this ontology has changed. Yes, I have had to drop Prospero and things ‘like’ it from the list, but I have added things to the list as well, such as the origins of human thought and attractor nets. On the whole, my sense is that the space of possible constructs has grown larger and more various.

What does "understand" mean? Can Google's Pathways ascend the stairway to AGI heaven?

Over at Marginal Revolution Alex Tabarrok posted about Pathways, The Chinese Room Thinks. He claims:

It seems obvious that the computer is reasoning. It certainly isn’t simply remembering. It is reasoning and at a pretty high level! To say that the computer doesn’t “understand” seems little better than a statement of religious faith or speciesism. [...] The sheer ability of AI to reason, counter-balances our initial intuition, bias and hubris, making the defects in Searle’s argument easier to accept.

Hmmm... I'm not so sure. I commented:

Well, back in the early-1970s ARPA (now DARPA) sponsored The Speech Understanding Project. It was a five-year effort spread over, I believe, four research groups. The goal was to produce a question-answering system (a term of art) that would take a spoken-language question as input and produce the correct answer to the question. The problem domain was defined by a database of navy ships. So you ask "When was the USS Forestal commissioned?" If it answers "1955," that's scored as a yes. A correct answer was taken to mean that the system had understood the question.

Is that understanding? Well, it's something. But it's hard to say that it is understanding in any deep sense. Still, that required a major research effort involving 10s if not (low) 100s of people and much of the work was quite interesting, if you've got a taste for that sort of thing, which I did in those days. What GPT-3 did in response to Jerry Seinfeld's bit about the Roosevelt tramway is a deeper kind of understanding. As is Pathways' performance in the examples Alex posted above.

How much deeper? If, on a scale of one to ten, we say ARPA's speech understanding systems work at level 1, where do we put Pathways? Does such a metric make sense? Is understanding one-dimensional?

If you ask me to explain Einstein's theory of relativity, I can certainly produce something. But it won't be very deep. Certainly better than 1, but probably not a 5, depending on how you define those levels. In what sense do I understand the theory of relativity?

(BTW, YouTube has a bunch of videos where something is explained on five levels: 1 grade school, 2 high school, 3) college graduate, 4) graduate student, 5) professional.)

What about Steven Spielberg's Jaws? We can ask questions: What was the sheriff's name? Who was the second person killed? How many people were killed? What animal did the killing? Those are relatively easy questions. These are more difficult: Why was the mayor reluctant to close the beach? Quint was haunted by an experience he had as a young adult. Tell us about that experience? Why did it haunt him? I suspect that most people who've seen the movie would have little trouble answering those questions.

And then there's this sort of question, one that I set out to answer: How does the movie exhibit Girard's theory of sacrifice? Of course, one might argue that it doesn't exhibit Girard's theory at all, and some, of course, would say that Girard's theory doesn't hold water. What kind of understanding is going on there? I suspect that most people who've seen Jaws could not deal with those questions. They know little, if anything at all, about Girard and likely don't care much for that sort of thing. Does that mean they don't understand the movie? To say so would be perverse.

But I also think that most people who've seen the movie understand it on a deeper level than is required to answer the first set of questions about the film. And they probably cannot verbalize much of their understanding. Does that mean they don't really understand the movie?

It's not at all clear to me just what one is claiming when one says that Pathways understands. For that matter, what is one claiming when one says that Pathways doesn't really understand?

It seems to me that we're caught between a rock and a hard place. The rock is our current conceptual system, which seems to treat understanding either as a binary variable – one either understands or one doesn't – or as a one-dimensional variable where the distinctions between one level and the next are not at all clear. The hard place is to start coming up with a new conceptual vocabulary that gives us a finer-grained an more useful way to comprehend and think about what these alien mentalities – is that the word we want? – are doing. I think we've got to start investigating that hard place. 

 Here's my notice of Pathways.

 * * * * *

Addendum: Let's assume for a minute that we're going to rate understanding on a scale, say, from 1 to 10. GPT-3 rates, say, 3. Along comes Pathways and it's clearly better than GPT-3. Where does it go? 4? 5? 6?

No.

Given that considerable distance remains between Pathways and humans, I'd say that a 1-10 scale is insufficient. Let's make it 1-100. GPT-3 goes in at 23 and Pathways at, say, 38.

That is to say, each time one of these remarkable results comes in I think it enlarges our sense of the measure space. Maybe it even forces us to start adding dimensions. It just makes the world larger.

* * * * *

Gary Marcus comments:

This week seems like win for AI, but it's actually a step back:
• No info about training set [Dall-E]
• Sparse disclosure of methods and errors [both]
• Anecdotal data only [PaLM jokes]
• Cherry-picking [both]
• No access for scientific community [both] https://t.co/HCn1g5yG7L

— Gary Marcus 🇺🇦 (@GaryMarcus) April 7, 2022