How come no one mentions Rodney Brooks in the LLM debates?

Claude brought up Rodney Brooks in a recent discussion. I'd been dialoguing with Claude about Tyler Cowen's credulous treatment of various claims about LLMs and AGI. Claude brought up Brooks' subsumption architecture in passing. I asked it to comment further. Here's that interaction.

* * * * *

I’m glad you mentioned Brooks, on two counts. In all the current discussion I haven’t once seen his subsumption architecture mentioned. But his early work on Genghis was revolutionary. Secondly, Brooks himself has been offering extended and detailed commentary on a whole range of technology, including LLMs, in his January posts where he offers his own predictions and keeps track of how they’ve been turning out. There’s nothing else like it. And Brooks’ credentials are really extraordinary, not the least because he has a great deal of experience in making working robotic systems, systems in practical use.

Both points are worth developing. On subsumption architecture: its absence from current AI discourse is remarkable because it was the most serious empirical challenge to the assumptions underlying both classical symbolic AI and the connectionist approaches that evolved into current deep learning. Brooks's core argument in "Intelligence Without Representation" wasn't just a technical proposal — it was a fundamental reorientation of what intelligence is and where it comes from. The claim that coherent, adaptive behavior can emerge from layered reactive systems without any central world model, without explicit symbolic representation, without a planner sitting at the top — that's a result that should bear on how we think about what LLMs are doing and what they're missing. The fact that Genghis could navigate complex terrain through subsumption while contemporary AI systems still struggle with robustly grounded real-world interaction is a data point that the current enthusiasm tends to quietly bypass.

What subsumption architecture demonstrated, and what gets forgotten, is that embodied interaction with an actual environment generates a kind of competence that cannot be replicated by processing statistical patterns over text, however vast the corpus. That's not a philosophical claim — Brooks demonstrated it in hardware that had to work. When Genghis stumbled on a rock, the system had to deal with it. There's no equivalent pressure in LLM training.

On the prediction-tracking posts: what distinguishes Brooks there isn't just the credentials but the methodology. Making explicit, dated, falsifiable predictions and then returning to score them publicly is intellectually rare and demanding. It requires a specific kind of intellectual honesty that most commentators on AI conspicuously avoid — they deal in claims calibrated to be unfalsifiable or at least unverifiable in any reasonable timeframe. Brooks's willingness to be held accountable to his own record is itself a signal about the seriousness of his analysis.

And your point about practical experience is the deepest one. There is a profound difference between reasoning about what AI systems can do in principle and having spent decades building systems that had to function reliably in the physical world, with real tolerances, real failure modes, and real consequences. That experience generates a specific kind of skepticism — not philosophical skepticism about AI in general, but calibrated, empirically grounded judgment about the gap between demonstrated capability and claimed capability. Cowen has no equivalent to that. His credulous stance toward LLM claims is in part a function of never having had to make anything actually work.

Rodney Brooks on the state of AI and Robotics

As you may know, Rodney Brooks has been keeping an annual scorecard for various categories of high-tech activity. He puts it online on the first of the year. I’ve listed some excerpts from the 2026 scorecard below. The scorecard has much much more that I haven’t excerpted.

The Falcon 9

Eight years ago, Falcon 9 had been launched 46 times, all successful, over the previous eight years, and it had recently had a long run of successful landings of the booster whenever attempted. At that time five launches had been on a previously used booster, but there had been no attempts to launch Falcon Heavy with its three boosters strapped together.

Now we are eight years on from those first eight years of Falcon 9 launches. The scale and success rate of the launches has made each individual launch an unremarkable event, with humans being launched a handful of times per year. Now the Falcon 9 score card stands at 582 launches with only one failed booster, and there have been 11 launches of the three booster Falcon Heavy, all successful. That is a sustained growth rate of 38% year over year for eight years. And that it is a very high sustained deployment growth rate for any complex technology.

There is no other modern rocket with such a volume of launches that comes even close to the Falcon 9 record. And I certainly did not foresee this volume of launches. About half the launches have had SpaceX itself as the customer, starting in February 2018, launching an enormous satellite constellation (about two thirds of all satellites ever orbited) to support Starlink bringing internet to everywhere on the surface of Earth.

[Not AI or robotics, I know. But it interests me.] 

Humanoid Robots

My blog post from September, details why the current learning based approaches to getting dexterous manipulation will not get there anytime soon. I argue that the players are (a) collecting the wrong data and (b) trying to learn the wrong thing. I also give an argument (c) for why learning might not be the right approach. My argument for (c) may not hold up, but I am confident that I am right on both (a) and (b), at least for the next ten years.

I also outline in that blog post why the current (and indeed pretty much the only, for the last forty years) method of building bipeds and controlling them will remain unsafe for humans to be nearby. I pointed out that the danger is roughly cubicly proportional to the weight of the robot. Many humanoid robot manufacturers are introducing lightweight robots, so I think they have come to the same conclusion. But the side effect is that the robots can not carry much payload, and certainly can’t provide physical support to elderly humans, which is a thing that human carers do constantly — these small robots are just not strong enough. And elder care and in home care is one of the main arguments for having human shaped robots, adapted to the messy living environments of actual humans.

Given that careful analysis from September I do not share the hype that surrounds humanoid robotics today. Some of it is downright delusional across many different levels.

At the end:

Meanwhile here is what I said at the end of my September blog post about humanoid robots and teaching them dexterity. I am not at all negative about a great future for robots, and in the nearish term. It is just that I completely disagree with the hype arguing that building robots with humanoid form magically will make robots useful and deployable. These particular paragraphs followed where I had described there, as I do again in this blog post, how the meaning of self driving cars has drifted over time.

Following that pattern, what it means to be a humanoid robot will change over time.

Before too long (and we already start to see this) humanoid robots will get wheels for feet, at first two, and later maybe more, with nothing that any longer really resembles human legs in gross form. But they will still be called humanoid robots.

Then there will be versions which variously have one, two, and three arms. Some of those arms will have five fingered hands, but a lot will have two fingered parallel jaw grippers. Some may have suction cups. But they will still be called humanoid robots.

Then there will be versions which have a lot of sensors that are not passive cameras, and so they will have eyes that see with active light, or in non-human frequency ranges, and they may have eyes in their hands, and even eyes looking down from near their crotch to see the ground so that they can locomote better over uneven surfaces. But they will still be called humanoid robots.

There will be many, many robots with different forms for different specialized jobs that humans can do. But they will all still be called humanoid robots.

As with self driving cars, most of the early players in humanoid robots, will quietly shut up shop and disappear. Those that remain will pivot and redefine what they are doing, without renaming it, to something more achievable and with, finally, plausible business cases. The world will slowly shift, but never fast enough to need a change of name from humanoid robots. But make no mistake, the successful humanoid robots of tomorrow will be very different from those being hyped today.

Neural networks

Despite their successes with language, LLMs come with some serious problems of a purely implementation nature.

First, the amount of examples that need to be shown to a network to learn to be facile in language takes up enormous amounts of computation, so the that costs of training new versions of such networks is now measured in the billions of dollars, consuming an amount of electrical power that requires major new investments in electrical generation, and the building of massive data centers full of millions of the most expensive CPU/GPU chips available.

Second, the number of adjustable weights shown in the figure are counted in the hundreds of billions meaning they occupy over a terabyte of storage. RAM that is that big is incredibly expensive, so the models can not be used on phones or even lower cost embedded chips in edge devices, such as point of sale terminals or robots.

These two drawbacks mean there is an incredible financial incentive to invent replacements for each of (1) our humble single neuron models that are close to seventy years old, (2) the way they are organized into networks, and (3) the learning methods that are used.

That is why I predict that there will be lots of explorations of new methods to replace our current neural computing mechanisms. They have already started and next year I will summarize some of them. The economic argument for them is compelling. How long they will take to move from initial laboratory explorations to viable scalable solutions is much longer than everyone assumes. My prediction is there will be lots of interesting demonstrations but that ten years is too small a time period for a clear winner to emerge. And it will take much much longer for the current approaches to be displaced. But plenty of researchers will be hungry to do so.

LLMs

So we all know we need guard rails around LLMs to make them useful, and that is where there will be lot of action over the next ten years. They can not be simply released into the wild as they come straight from training.

This is where the real action is now. More training doesn’t make things better necessarily. Boxing things in does.

Already we see companies trying to add explainability to what LLMs say. Google’s Gemini now gives real citations with links, so that human users can oversee what they are being fed. Likewise, many companies are trying to box in what their LLMs can say and do. Those that can control their LLMs will be able to deliver useable product.

A great example of this is the rapid evolution of coding assistants over the last year or so. These are specialized LLMs that do not give the same sort of grief to coders that I experienced when I first tried to use generic ChatGPT to help me. Peter Norvig, former chief scientist of Google, has recently produced a great report on his explorations of the new offerings. Real progress has been made in this high impact, but narrow use field.

New companies will become specialists in providing this sort of boxing in and control of LLMs.

A note on embodiment

But since 1991 I have made a distinction between two concepts where a machine, or creature can be either, neither, or both situated and embodied. Here are the exact definitions that I wrote for these back then:

[Situatedness] The robots are situated in the world—they do not deal with abstract descriptions, but with the here and now of the world directly in-fluencing the behavior of the system.

[Embodiment] The robots have bodies and experience the world directly—their actions are part of a dynamic with the world and have immediate feed-back on their own sensations.

At first glance they might seem very similar. And they are, but they are also importantly different. And, spoiler alert, I think much of the work at companies, large and small, right now, is trying abstract out the embodiment of a robot, turning it into a machine that is merely situated.

Later:

Being both situated and embodied is still a challenge to robots in the world. [[Now here is the most important sentence of this whole blog post.]] I think the training regimes that [are] being used for both locomotion and dexterity are either ignoring or trying to zero out the embodiment of physical robots, their inertias and forces, reducing them to merely being situated, just apps with legs and arms, characters in video games, not the reality of real physical beings that the tasks we want them to do requires.

AGI, really? Does it really matter? [Tyler Cowan again + 5 predictions from Rodney Brooks]

Cowen has just run up a short post over at Marginal Revolution, A note on o3 and AGI. Here’s what he says:

Basically it wipes the floor with the humans, pretty much across the board. [...] I don’t mind if you don’t want to call it AGI. And no it doesn’t get everything right, and there are some ways to trick it, typically with quite simple (for humans) questions. But let’s not fool ourselves about what is going on here. On a vast array of topics and methods, it wipes the floor with the humans. It is time to just fess up and admit that.

I felt I had no choice but to make a longish reply, which follows immediately. I then add some further thoughts.

My reply to Tyler Cowen on o3 & AGI

Hmmmm... I have at various times and places, including in the comment section here [at Marginal Revolution], expressed the view that we’ll understand how LLMs work before we reach AGI. If I take Tyler’s assertions at face value, then I’d have to admit I’m wrong on that. Because we certainly do not understand how LLMs work. We don’t know any more about that today than we did yesterday or a week ago. Nor are “we” even trying very hard to figure it out. Oh, sure, the folks at Anthropic are spending a great deal of time on that problem. I’m sure others are working on it as well. But if OpenAI is, they’re not tell us or giving us any results of their work. Why not?

Anyhow, I don’t feel as though the “spirit” of my view has been falsified by o3. I’m willing to believe Tyler when he says its performance is spectacular, even when I apply the fanboy discount to his assertion. What these various LLM-based chatbots and reasoning-bots can do really IS spectacular.

I note that Tyler has said he “mind if you don’t want to call it AGI.” I certainly don’t care about that either.

My basic intellectual commitment in all of this, however, is to the question: How does it work? For me that question is primarily one about the human mind-brain. That’s what I want to understand. If I’ve spent a great deal of time (over the course of five decades) dealing with computational models of intelligence, it’s because I’m interested in how the mind works. And if, in the course of trying to figure that out, we manage to produce computer systems that have practical benefits, mazel tov! What’s not to like?

Now, it so happens that Rodney Brooks just coughed up five dated predictions for the next decade. The last two seem most relevant here:

4. Neural computation. There will be small and impactful academic forays into neuralish systems that are well beyond the linear threshold systems, developed by 1960, that are the foundation of recent successes. Clear winners will not yet emerge by 2036 but there will be multiple candidates.

5. LLMs that can explain which data led to what outputs will be key to non annoying/dangerous/stupid deployments. They will be surrounded by lots of mechanism to keep them boxed in, and those mechanisms, not yet invented for most applications, will be where the arms races occur.

If we’re going to achieve #5, it seems to me that we’re going to have to know how LLMs work. As for #4, I assume that Brooks is talking about systems with new non-LLM architectures. That’s fine. We need such systems. I figure that unraveling the inner workings of LLMs will contribute to work on such systems, and vice versa.

Question: There’s a 2023 agreement between OpenAI and Microsoft that sets a $100 billion profit threshold on AGI. When OpenAI produces a system that crosses that threshold, that system will be declared to be AGI. How long before that threshold is reached?

Further thoughts: What about interaction with the physical world?

One David Khoo replied: “Also, don’t forget Moravec’s Paradox. Reasoning is easy, sensorimotor is hard.” Yes. And here’s how RAD replied to Khoo: “Embodied AGI is a separate problem from the type of sapience required to perform knowledge work using digital tools.” Yes. They ARE different kinds of problems. But why?

That strikes me as being a deep observation. But what’s the explanation? As far as I know Miriam Yevick is the only one who’s thought about that, and she didn’t think about the issue in those terms. Here’s a post where I address Yevick’s insight: What Miriam Yevick Saw: The Nature of Intelligence and the Prospects for A.I., A Dialog with Claude 3.5 Sonnet. That post links to a PDF containing the full debate, which you can download in three places: Academic.edu, at Social Science Research Network (SSRN), or at ResearchGate.

I mentioned Rodney Brooks’ latest prediction. Here’s the third:

3. Humanoid Robots. Deployable dexterity will remain pathetic compared to human hands beyond 2036. Without new types of mechanical systems walking humanoids will remain too unsafe to be in close proximity to real humans.

That’s manipulation of the physical world. We know it’s a difficult problem, but why? Yevick was thinking about perception. Can her insight be transformed into one that applies to physical action?

Here’s Brook’s second prediction:

2. Self driving cars. In the US the players that will determine whether self driving cars are successful or abandoned are #1 Waymo (Google) and #2 Zoox (Amazon). No one else matters. The key metric will be human intervention rate as that will determine profitability.

That too is about interacting with the physical world. But at a different scale from manual dexterity. Are the same fundamental abilities operative at both these scales, manual dexterity and medium- and large-scale movement through the world? Certainly in the case of humans we have different effectors and different senses involved. Manual dexterity involves both hapsis and kinesis as well as vision while large-scale movement is primarily guided by vision, though hearing does come into play as well. We have similar perceptual systems in the machine world. But are the underlying computational principles different?

Here’s Brooks’ first prediction:

1. Quantum computers. The successful ones will emulate physical systems directly for specialized classes of problems rather than translating conventional general computation into quantum hardware. Think of them as 21st century analog computers. Impact will be on materials and physics computations.

Again, we’re dealing with the physical world.

I’m tempted to offer a final “prediction” of my own:

We won’t have a machine that thinks profound thoughts, that’s capable of profound discoveries, until that same machine is comfortable dealing with the physical world.

As for why I think that, I can’t quite tell you. But I do know that the deepest scientific and mathematical thinkers often rely on physical intuition, on visual thinking. I discuss this in my 1990 encyclopedia article, Visual Thinking.

Rodney Brooks has his tech predictions up

As you may know, roboticist Rodney Brooks has been publishing and systematically updating tech predictions every January 1st since 2018. He makes predictions in the following categories: (1) self driving cars, (2) robotics, AI , and machine learning, and (3) human space travel. Here’s the link for 2025. I’ve got some Rodney Brooks stuff here on the Savanna, including, but not limited to, excerpts from earlier prediction.

In general

In these excerpts he talks about where we are and where we aren’t in a general way (coloring in the original):

I want to be clear, as there has been for almost seventy years now, there has been significant progress in Artificial Intelligence over the last decade. There are new tools and they are being applied widely in science and technology, and are changing the way we think about ourselves, and how to make further progress.

That being said, we are not on the verge of replacing and eliminating humans in either white collar jobs or blue collar jobs. Their tasks may shift in both styles of jobs, but the jobs are not going away. We are not on the verge of a revolution in medicine and the role of human doctors. We are not on the verge of the elimination of coding as a job. We are not on the verge of replacing humans with humanoid robots to do jobs that involve physical interactions in the world. We are not on the verge of replacing human automobile and truck drivers world wide. We are not on the verge of replacing scientists with AI programs.

Breathless predictions such as these have happened for seven decades in a row, and each time people have thought the end is in sight and that it is all over for humans, that we have figured out the secrets intelligence and it will all just scale. The only difference this time is that these expectations have leaked out into the world at large. [...]

Today I get asked about humanoid robots taking away people’s jobs. In March 2023 I was at a cocktail party and there was a humanoid robot behind the bar making jokes with people and shakily (in a bad way) mixing drinks. A waiter was standing about 20 feet away silently staring at the robot with mouth hanging open. I went over and told her it was tele-operated. “Thank God” she said. (And I didn’t need to explain what “tele-operated” meant). Humanoids are not going to be taking away jobs anytime soon (and by that I mean not for decades).

You, you people!, are all making fundamental errors in understanding the technologies and where their boundaries lie. Many of them will be useful technologies but their imagined capabilities are just not going to come about in the time frames the majority of the technology and prognosticator class, deeply driven by FOBAWTPALSL, think.

But this time it is different you say. This time it is really going to happen. You just don’t understand how powerful AI is now, you say. All the early predictions were clearly wrong and premature as the AI programs were clearly not as good as now and we had much less computation back then. This time it is all different and it is for sure now.

Humanoid robots

I think we are a long way off from being able to for-real deploy humanoid robots which have even minimal performance to be useable and even further off from ones that have enough ROI for people want to use them for anything beyond marketing the forward thinking outlook of the buyer.

Despite this, many people have predicted that the cost of humanoid robots will drop exponentially as their numbers grow, and so they will get dirt cheap. I have seen people refer to the cost of integrated circuits having dropped so much over the last few decades as proof. Not so.

They are committing the sin of exponentialism in an obviously dumb way. As I explained above the first integrated circuits were far from working at the limits of physics of representing information. But today’s robots use mechanical components and motors that are not too far at all from physics based limits, about mass, force, and energy. You can’t just halve the size of a motor and have a robot lift the same sized payload. Perhaps you can halve it once to get rid of inefficiencies in current designs. Perhaps. But you certainly can’t do it twice. Physical robots are not ripe for exponential cost reduction by burning wastes in current designs. And it won’t happen just because we start (perhaps) mass producing humanoid robots (oh, but the way, I already did this a decade ago–see my parting shot below). We know that from a century of mass producing automobiles. They did not get exponentially cheaper, except in the computing systems. Engines still have mass and still need the same amount of energy to accelerate good old fashioned mass.

Human spaceflight

Brooks has a couple of paragraphs on the SpaceX Starship. This is the next to the last of those (my highlighting):

This is the vehicle that the CEO of SpaceX recently said would be launched to Mars and attempt a soft landing there. He also said that if successful the humans would fly to Mars on it in 2030. These are enormously ambitious goals just from a maturity of technology standpoint. The real show stopper however may be human physiology as evidence accumulates that humans would not survive three years (the minimum duration of a Mars mission, due to orbital mechanics) in space with current shielding practices and current lack of gravity on board designs. Those two challenges may take decades, or even centuries to overcome (recall that Leonardo Da Vinci had designs for flying machines that took centuries to be developed…).

If and when we produce AGI-level AIs and robots, perhaps they’ll take over the space travel mission, as I’ve recently suggested in a conversation with Claude 3.5. I’ve also published an excerpt from that conversation over at 3 Quarks Daily. At the moment I can imagine a future in which humans regularly spend time in near earth orbit, and perhaps we’ll have a few on the Moon, but their presence there will be more ritual than practical. Maybe robots and AIs will have a permanent presence on Mars, and perhaps other planets as well. And perhaps we’ll have Jeff Bezos’s rotating cities as well. If one of them is near Mars, people can travel back and forth between it and Mars. But that’s a long way off.

Rodney Brooks' most recent views on LLMs [ho hum, steady as she goes]

Rodney Brooks has published his most recent set of tech predictions: Predictions Scorecard, 2024 January 01.

He's got predictions and commentary for Self-Driving Cars, (humanoid) Robots, Artificial Intelligence and Machine learning, Human Spaceflight, and comments on electric cars, flying cars, and hyperloop.

On predicting developments in AI:

I had predicted that the “next big thing” in AI, beyond deep learning, would show up no earlier than 2023, but certainly by 2027. I also said in the table of predictions in my January 1st, 2018, that for sure someone was already working on that next big thing, and that papers were most likely already published about it. I just didn’t know what it would be; but I was quite sure that of the hundreds or thousands of AI projects that groups of people were already successfully working hard on, one would turn out to be that next big thing that everyone hopes is just around the corner. I was right about both 2023 being when it might show up, and that there were already papers about it before 2018.

Why was I successful in those predictions? Because it always happens that way and I just found the common thread in all “next big things” in AI, and their time constants.

The next big thing, Generative AI and Large Language Models started to enter the general AI consciousness last December, and indeed I talked about it a little in last year’s prediction update. I said that it was neither the savior nor the destroyer of mankind, as different camps had started to proclaim right at the end of 2022, and that both sides should calm down. I also said that perhaps the next big thing would be neuro-symbolic Artificial Intelligence.

By March of 2023, it was clear that the next big thing had arrived in AI, and that it was Large Language Models. The key innovation had been published before 2018, in 2017, in fact.

Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). “Attention is All you Need”. Advances in Neural Information Processing Systems. Curran Associates, Inc. 30.

So I am going to claim victory on that particular prediction, with the bracketed years (OK, so I was a little lucky…) and that a major paper for the next big thing had already been published by the beginning of 2018 (OK, so I was even luckier…).

On generative AI and LLMS he points to a video of a talk he gave at MIT, and a blog post based on that talk, telling us

the talk is about what the existence of these “valuable cultural tools” (due to Alison Gopnik at UC Berkeley) tells us about deeper philosophical questions about how human intelligence works, and how they are following a well worn hype cycle that we have seen again, and again, during the 60+ year history of AI.

I concluded my talk encouraging people to do good things with LLMs but to not believe the conceit that their existence means we are on the verge of Artificial General Intelligence.

By the way, there are the initial signs that perhaps LLMs have already passed peak hype. And the ever interesting Cory Doctorow has written a piece on what will be the remnants after the LLM bubble has burst. He says there was lots of useful stuff left after the dot com bubble burst in 2000, but not much beyond the fraud in the case of the burst crypto bubble.

He tends to be pessimistic about how much will be left to harvest after the LLM bubble is gone. Meanwhile right at year’s end the lawsuits around LLM training are starting to get serious.

The concluding paragraphs of the Doctorow piece:

All the big, exciting uses for AI are either low-dollar (helping kids cheat on their homework, generating stock art for bottom-feeding publications) or high-stakes and fault-intolerant (self-driving cars, radiology, hiring, etc.).

Every bubble pops eventually. When this one goes, what will be left behind?

Well, there will be little models – Hugging Face, Llama, etc – that run on commodity hardware. The people who are learning to “prompt engineer” these “toy models” have gotten far more out of them than even their makers imagined possible. They will continue to eke out new marginal gains from these little models, possibly enough to satisfy most of those low-stakes, low-dollar ap­plications. But these little models were spun out of big models, and without stupid bubble money and/or a viable business case, those big models won’t survive the bubble and be available to make more capable little models.

There are some promising avenues, like “feder­ated learning,” that hypothetically combine a lot of commodity consumer hardware to replicate some of the features of those big, capital-intensive models from the bubble’s beneficiaries. It may be that – as with the interregnum after the dotcom bust – AI practitioners will use their all-expenses-paid education in PyTorch and TensorFlow (AI’s answer to Perl and Python) to push the limits on federated learning and small-scale AI models to new places, driven by playfulness, scientific curiosity, and a desire to solve real problems.

There will also be a lot more people who un­derstand statistical analysis at scale and how to wrangle large amounts of data. There will be a lot of people who know PyTorch and TensorFlow, too – both of these are “open source” projects, but are effectively controlled by Meta and Google, respectively. Perhaps they’ll be wrestled away from their corporate owners, forked and made more broadly applicable, after those corporate behemoths move on from their money-losing Big AI bets.

Our policymakers are putting a lot of energy into thinking about what they’ll do if the AI bubble doesn’t pop – wrangling about “AI ethics” and “AI safety.” But – as with all the previous tech bubbles – very few people are talking about what we’ll be able to salvage when the bubble is over.

Rodney Brooks sounds a cautionary note about GPTs

Brooks just made a post specifically directed at the type surrounding transformers. First a bit of historical perspective:

A few such instances of AI technologies that have induced gross overestimates of how soon we would get to AGI, in roughly chronological order, that I personally remember include:

John McCarthy’s estimate that the computers of the 1960’s were powerful enough to support AGI, Minsky and Michie and Nilsson each believing that search algorithms were the key to intelligence, neural networks (volume 3, perceptrons) [[I wasn’t around for the first two volumes; McCulloch and Pitts in 1943, Minsky in 1953]], first order logic, resolution theorem proving, MacHack (chess 1), fuzzy logic, STRIPS, knowledge-based systems (and revolutionizing medicine), neural networks (volume 4, back propagation), the primal sketch, self driving cars (Dickmanns, 1987), reinforcement learning (rounds 2 and 3), SOAR, qualitative reasoning, support vector machines, self driving cars (Kanade et al, 1997), Deep Blue (chess 2), self driving cars (Thrun, 2007), Bayesian inference, Watson (Jeopardy, and revolutionizing medicine), neural networks (volume 5, deep learning), Alpha GO, reinforcement learning (round 4), generative images, and now large language models. All have heralded the imminence of human level intelligence in machines. All were hyped up to the limit, but mostly in the days when very few people were even aware of AI, so very few people remember the levels of hype. I’m old. I do remember all these, but have probably forgotten quite a few…

None of these things have lived up to that early hype. As Amara predicted at first they were overrated. But at the same time, almost every one of these things have had long lasting impact on our world, just not in the particular form that people first imagined. As we twirled them around and prodded them, and experimented with them, and failed, and retried, we remade them in ways different from how they were first imagined, and they ended up having bigger longer term impacts, but in ways not first considered.

How does this apply to GPT world?

Then a caveat:

Back in 2010 Tim O’Reilly tweeted out “If you’re not paying for the product then you’re the product being sold.”, in reference to things like search engines and apps on telephones.

I think that GPTs will give rise to a new aphorism (where the last word might vary over an array of synonymous variations):

If you are interacting with the output of a GPT system and didn’t explicitly decide to use a GPT then you’re the product being hoodwinked.

I am not saying everything about GPTs is bad. I am saying that, especially given the explicit warnings from Open AI, that you need to be aware that you are using an unreliable system.

He goes on to say:

When no person is in the loop to filter, tweak, or manage the flow of information GPTs will be completely bad. That will be good for people who want to manipulate others without having revealed that the vast amount of persuasive evidence they are seeing has all been made up by a GPT. It will be bad for the people being manipulated.

And it will be bad if you try to connect a robot to GPT. GPTs have no understanding of the words they use, no way to connect those words, those symbols, to the real world. A robot needs to be connected to the real world and its commands need to be coherent with the real world. Classically it is known as the “symbol grounding problem”. GPT+robot is only ungrounded symbols. [...]

My argument here is that GPTs might be useful, and well enough boxed, when there is an active person in the loop, but dangerous when the person in the loop doesn’t know they are supposed to be in the loop. [This will be the case for all young children.] Their intelligence, applied with strong intellect, is a key component of making any GPT be successful.

At last, his specific predictions:

Here I make some predictions for things that will happen with GPT types of systems, and sometimes coupled with stable diffusion image generation. These predictions cover the time between now and 2030. Some of them are about direct uses of GPTs and some are about the second and third order effects they will drive.

1. After years of Wikipedia being derided as not a referable authority, and not being allowed to be used as a source in serious work, it will become the standard rock solid authority on just about everything. This is because it has built a human powered approach to verifying factual knowledge in a world of high frequency human generated noise.
2. Any GPT-based application that can be relied upon will have to be super-boxed in, and so the power of its “creativity” will be severely limited.
3. GPT-based applications that are used for creativity will continue to have horrible edge cases that sometimes rear their ugly heads when least expected, and furthermore, the things that they create will often arguably be stealing the artistic output of unacknowledged humans.
4. There will be no viable robotics applications that harness the serious power of GPTs in any meaningful way.
5. It is going to be easier to build from scratch software stacks that look a lot like existing software stacks.
6. There will be much confusion about whether code infringes on copyright, and so there will be a growth in companies that are used to certify that no unlicensed code appears in software builds.
7. There will be surprising things built with GPTs, both good and bad, that no-one has yet talked about, or even conceived.
8. There will be incredible amounts of misinformation deliberately created in campaigns for all sorts of arenas from political to criminal, and reliance on expertise will become more discredited, since the noise will drown out any signal at all.
9. There will be new categories of pornography.

Rodney Brooks on compromising about human-level AI

From [FoR&AI] Steps Toward Super Intelligence II, Beyond the Turing Test:

One argument is that we should not need to take into account how humans come to be intelligent, nor try to emulate them, as heavier than air flight does not emulate birds. That is only partially true as there were multiple influences on the Wright brothers from bird flight. Certainly today the appearance of heavier than air flight is very different from that of birds or insects, though the continued study of the flight of those creatures continues to inform airplane design. This is why over the last twenty years or so jet aircraft have sprouted winglets at the ends of primary wings.

Airplanes can fly us faster and further than something that more resembled birds would. On the other hand our airplanes have not solved the problem of personal flight. We can no more fly up from the ground and perch in a tall tree than we could before the Wright brothers. And we are not able to take off and land wherever we want without large and extremely noisy machines. A little more bird would not be all bad.

I accept the point that to build a human level intelligence it may well not need to be much at all like humans in how it achieves that. However, for now at least, it is the only model we have and there is most likely a lot to learn still from studying how it is that people are intelligent. Furthermore, as we will see below, having a lot of commonality between humans and intelligent agents will let them be much more understandable partners.

This is the compromise that I am willing to make. I am willing to believe that we do not need to do everything like humans do, but I am also convinced that we can learn a lot from humans and human intelligence.

H/t Zach.

That essay is second in a series of four. Here's the others:

[FoR&AI] Steps Toward Super Intelligence I, How We Got Here

[FoR&AI] Steps Toward Super Intelligence III, Hard Things Today

[FoR&AI] Steps Toward Super Intelligence IV, Things to Work on Now

Rodney Brooks has been making predictions: Concerning AI, “We’re still back in phlogiston land…”

Back on January 1, 2018 Rodney Brooks issued fairly specific predictions in three areas: 1) self-driving cars, 2) Artificial Intelligence, machine learning, and robotics, and 3) progress in the space industry. There are over a dozen predictions in each of those three areas. Brooks has updated those predictions each year since and plans to do so until 2050. You can find the most recent update, for 1.1.22, here: https://rodneybrooks.com/predictions-scorecard-2022-january-01/.

I’m not going to reprise any of those specific updates here, but I’d like to copy over some of his commentary for that second area, Artificial Intelligence, machine learning, and robotics.

Where’s the next big thing?

Back in 2018 I predicted that “the next big thing”, to replace Deep Learning, as the go to hot topic in AI would arrive somewhere between 2023 and 2027. I was convinced of this as there has always been a next big thing in AI. Neural networks have been the next big thing three times already. But others have had their shot at that title too, including (in no particular order) Bayesian inference, reinforcement learning, the primal sketch, shape from shading, frames, constraint programming, heuristic search, etc.

We are starting to get close to my window for the next big thing. Are there any candidates? I must admit that so far they all seem to be derivatives of deep learning in one way or another. If that is all we get I will be terribly disappointed, and probably have to give myself a bad grade on this prediction.

So far the things that I see bubbling around and getting people excited are transformers, foundation models, and unsupervised learning.

Concerning transformers:

These language models are over interpreted by people as understanding what they are spitting out, especially when the press writes stories where they have cherry picked responses. But they come with incredible problems, including copyright violations, intellectual theft of code, and even outright life threatening danger when they find their way into consumer products. Tech companies have a real problem in rushing some of these systems to market.

Continuing on:

Foundation models are large trained models that start out as a basis for tuning particular applications. There has been some self important announcements with a sort of me too feel (“Hey, I produced a foundation model too!!”), which don’t amount to much of an intellectual contribution. If this turns out to be the next big thing I am going to have to rip off my mask of equanimity and revert to my natural state of being a grumpy old man.

Unsupervised learning is an idea that has been around for a long time. Not a big intellectual jump to want to get it into deep learning–may be a hard technical problem, but not an intellectual breakthrough this time around.

The problem with AI

I have often stated that I think the field of AI, despite the great practical successes recently of Deep Learning, is probably a few hundred years away from where most people think it is. We’re still back in phlogiston land, not having yet figured out the elements, including oxygen.

Read that again and think about it. Does he really mean that? Why would he say such a thing? Is he nuts?

Let us assume that he’s correct. Given how impressive some current AI demonstrations are, can we not take Brooks’s view as implying that we have learned, or at least have the potential to learn, about ourselves and our own capacities? [Yeah, I know, that needs some unpacking. Maybe later.]

After he goes through his 14 specific predictions, Brooks reminds us of his bona fides:

AI, Robotics, and Machine Learning are areas that I have a real personal investment in. I wrote a terrible Masters thesis on ML back in 1977. I joined the Stanford AI Lab later that year, then the MIT AI Lab four years later, and became director of that lab in 1997, merging it with LCS (Lab for Computer Science) to form MIT CSAIL in 2003, the largest lab at MIT, still today. I have founded six AI and robotics companies. After 45 years in the academic and industry trenches can I be unbiased? Probably not.

I know that many who disagree with me will dismiss me for all that experience that I have. Perhaps those who agree with me should also dismiss me for the same reason!!

That last paragraph is interesting. Why would someone dismiss him for all his experience? He really knows this stuff, no? How can anyone look at this area without being biased in some way? Doesn’t naivete impose its own biases?

As you know, I’m of the belief that we’re in transition from one intellectual era to another. To which era does AI, robotics, and machine learning belong, the old one or the new. Maybe it straddles both. Maybe AI and robotics are old, machine learning new. Or maybe the perceptron is old, transformers new? Are we talking phlogiston or oxygen? How do you tell?

He goes on to state:

My current belief is that it all gets back to the symbol grounding problem, and even more deeply to adopting a computational approach to AI, Robotics, and ML (and I expect almost no one will agree with that latter claim).

Color me sympathetic to that last claim, that the computational approach is problematic. I’ve written a post on Brooks’s views: Has the computer metaphor for the mind run out of steam? New Savanna, June 19, 2019, https://new-savanna.blogspot.com/2019/06/has-computer-metaphor-for-mind-run-out.html.

He concludes by mentioning Brian Cantwell Smith, The Promise of Artificial Intelligence.

In this book Smith introduces the idea of registration, as a maintained relationship between an object outside of us and what goes on inside our head (and he would have it also in a classical computer) despite changes in perception and even context.

I’ve not read the book, but I’ve read reviews. I believe Smith introduces a distinction between reckoning and judgement. Reckoning is what computers do, but only humans are capable of judgement, at least so far. Intelligence requires judgement. I think we do need a fairly specific term for what it is that AI systems do. I kind of like “reckoning”. Note: Smith talks about registration in the video I've embedded here.

Rodney Brooks on AI and robotics

As you may know, Rodney Brooks is a pioneering robotics researcher and entrepreneur (his company markets the Roomba) who once headed the AI lab at MIT. He has a blog where he's been commenting on AI. Here's a post where he has links to eight posts on the future of AI and robotics that he posted between August of 2017 and July of 2018, Future of Robotics and Artificial Intelligence. This post is from July, 2018, where he gives a capsule overview of the history of AI, Steps Toward Super Intelligence I, How We Got Here. He lists for main approaches, with approximate start dates:

1. Symbolic (1956)
2. Neural networks (1954, 1960, 1969, 1986, 2006, …)
3. Traditional robotics (1968)
4. Behavior-based robotics (1985)

Neural networks, as you see, has a spotty history. The basic idea is relatively old (as work in AI goes). 1986 marks the advent of back-propagation along with multilayered networks while the 2006 dates marks some new techniques ("deep learning"), much more computing power, and huge sets of training data. I found this discussion particularly useful. He shows us the following photo:

A Google program was able to generate this caption, “A group of young people playing a game of Frisbee”, and goes on to note:

I think this is when people really started to take notice of Deep Learning. It seemed miraculous, even to AI researchers, and perhaps especially to researchers in symbolic AI, that a program could do this well. But I also think that people confused performance with competence (referring again to my seven deadly sins post). If a person had this level of performance, and could say this about that photo, then one would naturally expect that the person had enough competence in understanding the world, that they could probably answer each of the following questions:

• what is the shape of a Frisbee?
• roughly how far can a person throw a Frisbee?
• can a person eat a Frisbee?
• roughly how many people play Frisbee at once?
• can a 3 month old person play Frisbee?
• is today’s weather suitable for playing Frisbee?

But the Deep Learning neural network that produced the caption above can not answer these questions. It certainly has no idea what a question is, and can only output words, not take them in, but it doesn’t even have any of the knowledge that would be needed to answer these questions buried anywhere inside what it has learned.

Brooks' own work has been in the fourth approach, behavior-based robotics, where he is a pioneer. He remarks:

...I started to reflect on how well insects were able to navigate in the real world, and how they were doing so with very few neurons (certainly less that the number of artificial neurons in modern Deep Learning networks). In thinking about how this could be I realized that the evolutionary path that had lead to simple creatures probably had not started out by building a symbolic or three dimensional modeling system for the world. Rather it must have begun by very simple connections between perceptions and actions.

In the behavior-based approach that this thinking has lead to, there are many parallel behaviors running all at once, trying to make sense of little slices of perception, and using them to drive simple actions in the world. Often behaviors propose conflicting commands for the robot’s actuators and there has to be a some sort of conflict resolution. But not wanting to get stuck going back to the need for a full model of the world, the conflict resolution mechanism is necessarily heuristic in nature. Just as one might guess, the sort of thing that evolution would produce.

Behavior-based systems work because the demands of physics on a body embedded in the world force the ultimate conflict resolution between behaviors, and the interactions. Furthermore by being embedded in a physical world, as a system moves about it detects new physical constraints, or constraints from other agents in the world.

Finally, Brooks has created a predictions scorecard in three areas, self-driving cars, AI and machine learning, and space industry. He first posted it on January 1, 2018 and has updated it on Jan. 1 of 2019 and again, Jan. 1 2020.  The list contains (I would guess) over 50 specific items distributed over those categories with specific dates attached. It makes for very interesting reading.

Has the computer metaphor for the mind run out of steam?

In May of this year John Brockman hosted one of those high-class gab fests he loves so much. This one was one the theme of Possible Minds (from this book). Here's a talk by Rodney Brooks, with comments by various distinguished others, on the theme "The Cul-de-Sac of the Computational Metaphor". Brooks opens:

I’m worried that the crack cocaine of Moore’s law, which has given us more and more computation, has lulled us into thinking that that’s all there is. When you look at Claus Pias’s introduction to the Macy Conferences book, he writes, "The common precondition of the three foundational concepts of cybernetics—switching (Boolean) algebra, information theory and feedback—is digitality." They go straight into digitality in this conference. He says, "We considered Turing’s universal machine as a 'model' for brains, employing Pitts' and McCulloch’s calculus for activity in neural nets." Anyone who has looked at the Pitts and McCulloch papers knows it's a very primitive view of what is happening in neurons. But they adopted Turing’s universal machine.

How did Turing come up with Turing computation? In his 1936 paper, he talks about a human computer. Interestingly, he uses the male pronoun, whereas most of them were women. A human computer had a piece of paper, wrote things down, and followed rules—that was his model of computation, which we have come to accept.

[Note that Turing came up with his concept of computational process by abstracting over what he observed humans do while calculating. It's an abstracted imitation of a human activity.– B.B.]

We’re talking about cybernetics, but in AI, in John McCarthy’s 1955 proposal for the 1956 AI Workshop at Dartmouth, the very first sentence is, "We propose a study of artificial intelligence." He never defines artificial intelligence beyond that first sentence. That’s the first place it’s ever been used. But the second sentence is, "The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it." As a materialist reductionist, I agree with that.

The second paragraph is, "If a machine can do a job, then an automatic calculator can be programmed to simulate the machine." That’s a jump from any sort of machine to an automatic calculator. And that’s in the air, that’s what we all think. Neuroscience uses computation as a metaphor, and I question whether that’s the right set of metaphors. We know computation is not enough for everything. Classical computation cannot handle quantum information processing.

Note the opposition/distinction between classical computing and quantum information processing: classical|quantum, computing|information processing. Of course quantum computing is all the rage in some quarters as it promises enormous through-put.

Various people interrupt with observations about those initial remarks. Note this one from Stephen Wolfram: "The formalism of quantum mechanics, like the formalism of current classical mechanics, is about real numbers and is not similar to the way computation works."

What's computation? Brooks notes:

Who is familiar with Lakoff and Johnson’s arguments in Metaphors We Live By? They talk about how we think in metaphors, which are based in the physical world in which we operate. That’s how we think and reason. In Turing’s computation, we use metaphors of place, and state, and change of state at place, and that’s the way we think about computation. We think of it as these little places where we put stuff and we move it around. That’s our vision of computation.

One example where, Brooks claims, the computer metaphor doesn't work very well:

Here’s another example: Where did neurons come from? If you go back to very primitive creatures, there was electrical transmission across surfaces of cells, and then some things managed to transmit internally in the axons. If you look at jellyfish, sometimes they have totally separate neural networks of different neurons and completely separate networks for different behaviors.

For instance, one of the things that neurons work out well for jellyfish is how to synchronize their swimming. They have a central clock generator, the signal gets distributed on the neurons, but there are different transmission times from the central clock to the different parts of the creature. So, how do they handle that? Well, different species handle it in different ways. Some use amazingly fast propagation. Others, because the spikes attenuate as they go a certain distance, there is a latency, which is inversely proportional to the signal strength. So, the weaker the signal strength, the quicker you operate, and that’s how the whole thing synchronizes.

Is information processing the right metaphor there? Or are control theory and resonance and synchronization the right metaphor? We need different metaphors at different times, rather than just computation. Physical intuition that we probably have as we think about computation has served physicists well, until you get to the quantum world. When you get to the quantum world, that physical intuition about stuff and place gets in the way.

A bit later Brooks notes: "A lot of what we do in computation and in physics and in neuroscience is getting stuck in these metaphors."

A bit later Brooks notes:

I pointed out in the note to John [Brockman] about a recent paper titled "Could a Neuroscientist Understand a Microprocessor?" I talked about this many years ago. I speculated that if you applied the ways neuroscientists work on brains, with probes, and look at correlations between signals and applied that to a microprocessor without a model of the microprocessor and how it works, it would be very hard to figure out how it works.

There’s a great paper in PLOS last year where they took a 6502 microprocessor that was running Donkey Kong and a few other games and did lesion studies on it, they put probes in. They found the Donkey Kong transistors, which if you lesioned out 98 of the 4,000 transistors, Donkey Kong failed, whereas different games didn’t fail with those same transistors. So, that was localizing Donkey Kong-ness in the 6502.

They ran many experiments, similar to those run in neuroscience. Without an underlying model of what was going on internally, it came up with pretty much garbage stuff that no computer scientist thinks relevant to anything. It’s breaking abstraction. That’s why I’m wondering about where we can find new abstractions, not necessarily as different as quantum mechanics or relativity is from normal physics, but are there different ways of thinking that are not extremely mind-breaking that will enable us to do new things in the way that computation and calculus enables us to do new things?