AGI, really? Does it really matter? [Tyler Cowan again + 5 predictions from Rodney Brooks]

Cowen has just run up a short post over at Marginal Revolution, A note on o3 and AGI. Here’s what he says:

Basically it wipes the floor with the humans, pretty much across the board. [...] I don’t mind if you don’t want to call it AGI. And no it doesn’t get everything right, and there are some ways to trick it, typically with quite simple (for humans) questions. But let’s not fool ourselves about what is going on here. On a vast array of topics and methods, it wipes the floor with the humans. It is time to just fess up and admit that.

I felt I had no choice but to make a longish reply, which follows immediately. I then add some further thoughts.

My reply to Tyler Cowen on o3 & AGI

Hmmmm... I have at various times and places, including in the comment section here [at Marginal Revolution], expressed the view that we’ll understand how LLMs work before we reach AGI. If I take Tyler’s assertions at face value, then I’d have to admit I’m wrong on that. Because we certainly do not understand how LLMs work. We don’t know any more about that today than we did yesterday or a week ago. Nor are “we” even trying very hard to figure it out. Oh, sure, the folks at Anthropic are spending a great deal of time on that problem. I’m sure others are working on it as well. But if OpenAI is, they’re not tell us or giving us any results of their work. Why not?

Anyhow, I don’t feel as though the “spirit” of my view has been falsified by o3. I’m willing to believe Tyler when he says its performance is spectacular, even when I apply the fanboy discount to his assertion. What these various LLM-based chatbots and reasoning-bots can do really IS spectacular.

I note that Tyler has said he “mind if you don’t want to call it AGI.” I certainly don’t care about that either.

My basic intellectual commitment in all of this, however, is to the question: How does it work? For me that question is primarily one about the human mind-brain. That’s what I want to understand. If I’ve spent a great deal of time (over the course of five decades) dealing with computational models of intelligence, it’s because I’m interested in how the mind works. And if, in the course of trying to figure that out, we manage to produce computer systems that have practical benefits, mazel tov! What’s not to like?

Now, it so happens that Rodney Brooks just coughed up five dated predictions for the next decade. The last two seem most relevant here:

4. Neural computation. There will be small and impactful academic forays into neuralish systems that are well beyond the linear threshold systems, developed by 1960, that are the foundation of recent successes. Clear winners will not yet emerge by 2036 but there will be multiple candidates.

5. LLMs that can explain which data led to what outputs will be key to non annoying/dangerous/stupid deployments. They will be surrounded by lots of mechanism to keep them boxed in, and those mechanisms, not yet invented for most applications, will be where the arms races occur.

If we’re going to achieve #5, it seems to me that we’re going to have to know how LLMs work. As for #4, I assume that Brooks is talking about systems with new non-LLM architectures. That’s fine. We need such systems. I figure that unraveling the inner workings of LLMs will contribute to work on such systems, and vice versa.

Question: There’s a 2023 agreement between OpenAI and Microsoft that sets a $100 billion profit threshold on AGI. When OpenAI produces a system that crosses that threshold, that system will be declared to be AGI. How long before that threshold is reached?

Further thoughts: What about interaction with the physical world?

One David Khoo replied: “Also, don’t forget Moravec’s Paradox. Reasoning is easy, sensorimotor is hard.” Yes. And here’s how RAD replied to Khoo: “Embodied AGI is a separate problem from the type of sapience required to perform knowledge work using digital tools.” Yes. They ARE different kinds of problems. But why?

That strikes me as being a deep observation. But what’s the explanation? As far as I know Miriam Yevick is the only one who’s thought about that, and she didn’t think about the issue in those terms. Here’s a post where I address Yevick’s insight: What Miriam Yevick Saw: The Nature of Intelligence and the Prospects for A.I., A Dialog with Claude 3.5 Sonnet. That post links to a PDF containing the full debate, which you can download in three places: Academic.edu, at Social Science Research Network (SSRN), or at ResearchGate.

I mentioned Rodney Brooks’ latest prediction. Here’s the third:

3. Humanoid Robots. Deployable dexterity will remain pathetic compared to human hands beyond 2036. Without new types of mechanical systems walking humanoids will remain too unsafe to be in close proximity to real humans.

That’s manipulation of the physical world. We know it’s a difficult problem, but why? Yevick was thinking about perception. Can her insight be transformed into one that applies to physical action?

Here’s Brook’s second prediction:

2. Self driving cars. In the US the players that will determine whether self driving cars are successful or abandoned are #1 Waymo (Google) and #2 Zoox (Amazon). No one else matters. The key metric will be human intervention rate as that will determine profitability.

That too is about interacting with the physical world. But at a different scale from manual dexterity. Are the same fundamental abilities operative at both these scales, manual dexterity and medium- and large-scale movement through the world? Certainly in the case of humans we have different effectors and different senses involved. Manual dexterity involves both hapsis and kinesis as well as vision while large-scale movement is primarily guided by vision, though hearing does come into play as well. We have similar perceptual systems in the machine world. But are the underlying computational principles different?

Here’s Brooks’ first prediction:

1. Quantum computers. The successful ones will emulate physical systems directly for specialized classes of problems rather than translating conventional general computation into quantum hardware. Think of them as 21st century analog computers. Impact will be on materials and physics computations.

Again, we’re dealing with the physical world.

I’m tempted to offer a final “prediction” of my own:

We won’t have a machine that thinks profound thoughts, that’s capable of profound discoveries, until that same machine is comfortable dealing with the physical world.

As for why I think that, I can’t quite tell you. But I do know that the deepest scientific and mathematical thinkers often rely on physical intuition, on visual thinking. I discuss this in my 1990 encyclopedia article, Visual Thinking.