Four Propositions about Intelligence in Animals, Humans, and AIs

Some quickies.

1. Intelligence cannot be reduced to computation

All animal perception and cogitation take place in a complex world where animals have finite resources. Therefore the principles of intelligence, as an aspect of perception and cogitation, cannot be reduced to the principles of computation as the principles of computation assume unbounded resources.

Let’s call this Yevick’s First Law, as it is a consequence of her 1975 paper, “Holographic or fourier logic” (Pattern Recognition, Vol. 7, No. 4, pp. 197-213).

David Hays and I formulated what we called “Yevick’s Law” in our 1988 paper, Principles and Development of Natural Intelligence” (Journal of Social and Biological Structures). Let’s call that Yevick’s Second Law:

The world consists of geometrically simple and geometrically complex objects. Simple objects are best computing with sequential logic (aka symbolic systems). Complex objects are best computed with holographic logic (aka distributed neural nets). Some objects are such that they require the interaction of both computational regimes. Let us say that fluency in that interaction is intelligence. (See my working paper, What Miriam Yevick Saw: The Nature of Intelligence and the Prospects for A.I., A Dialog with Claude 3.5 Sonnet, 2025).

2. Intelligence in animals

Let us consider a relatively simple animal, a vertebrate, likely a marine animal. One the one hand, it must navigate the world, moving from place to place. This is mediated by the hippocampus, which is a so-called “cognitive map.” This is basic sequential cogitation.

As it moves from place to place it senses things, good things, bad things, other things. Olfaction is perhaps the most basic sense. For what it is worth, it’s the sensory mode that the late Walter Freeman used in his investigations of complex neural dynamics. Olfaction works via a holographic or gestalt process.

Taken together, moving about the world and sense things involves the two modes specified above.

Now let’s consider vision in vertebrates, where the eye is mobile and scans the world. Visual identification is a holographic process. However, the (human) eye scans the scene rapidly and unconsciously. This is a sequential process. Therefore vertebrate vision involves the two modes internally. (I suspect that vision in invertebrates does not, but I don’t actually know).

3. Natural language is its own metalanguage

Humans differ from animals in many and various ways. It is the capacity for language that has allowed humans to move into a different relationship with the world from that characteristic of animals. What makes human language particularly powerful is that it can serve as its own metalanguage, Roman Jakobson’s metalingual function.

This does not involve any deep mystery or logical conundrum. Rather it is a direct consequence of that fact that natural language is physically embodied, initially in sound and gesture, later as written symbols. This embodied is a sensory object out there in the world among all the other sensory objects.

Initially the metalingual function operates in direct, perhaps superficial, but useful ways. Think of how we refer to language as a means of negotiating conversation: “What did you say? I didn’t hear you?” But the metalingual function can be used to define new terms, something that interested my teacher and colleague, David Hays. It can even be used to define other, more restricted languages (e.g. chess and arithmetic), and serve as metalanguages for them.

Thus it is the foundation of the succession of cognitive ranks that David Hays and I began investigating in the 1990s starting with our paper, “The Evolution of Cognition” (Journal of Social and Biological Systems, later becoming the Journal of Social and Evolutionary Systems). That process has, in time, led to the development of digital computers and, now, to so-called artificial intelligence.

That leads us to our fourth and last note.

4. The last frontier of intelligence

Is not an autonomous artificial system of some kind, though such systems are important and will be increasingly so. here’s the upshot of a conversation I recently had with Claude:

If that's right, then the last frontier isn't more capability in the pattern-matching sense — bigger weight spaces, richer latent connections, better approximations of the associative regime. It's the specific, non-scalable, non-parallelizable fact of an individual mind's biography, which generates paths through possibility space that are real, productive, and genuinely inaccessible to any system that hasn't lived a life.

You should read the whole post to see the logic behind that conclusion.

Note that that is my current best response to the idea of super-intelligence or artificial-superintelligence (ASI). I do not see the future bringing us as AI system that outthinks humans in every way and either creates a world in which we are coddled pets or one in which we are slaves, if we are allowed to exist at all. Those ideas are subjective fantasy.

I note as well that the process that brings us to that point or, if you will, through which we arrive at the point, will be one in which we have a much deeper understanding of that brain and its processes than we now have. For what it’s worth, that understanding is what I have been seeking all these years, starting with my initial investigation of “Kubla Khan”: Xanadu, GPT, and Beyond: An adventure of the mind.

The last frontier of intelligence: On the role of AI helping humans to bridge the gaps between distant concepts.

That's something I do all the time. Case in point, my current working paper: Notes on the Collective Valuation of "Thick" Objects: Financial Assets, Movies, and Novels. Dwarkesh Patel brings that up in his recent podcast with Grant Sanderson:

That particular conversation starts at 00:38:08. You can also zip to it in the transcript.

I recently had a conversation with Claude which started with some observations about AI and chess, coding, and math and then language. We then dialoged about this and that and ended up with a long-ish prompt grounded in my intellectual history and how I use chatbots to bridge (often huge) gaps in my interests and understanding. Here's what Claude concluded about that:

If that's right, then the last frontier isn't more capability in the pattern-matching sense — bigger weight spaces, richer latent connections, better approximations of the associative regime. It's the specific, non-scalable, non-parallelizable fact of an individual mind's biography, which generates paths through possibility space that are real, productive, and genuinely inaccessible to any system that hasn't lived a life.

Here's the prompt and reply that led to that statement.

* * * * *

In 1978 I wrote a dissertation on Cognitive Science and Literary Theory. Since cognitive science was still somewhat new at the time I felt I had to explain what it was. In fact it was and is a loosely organized set of themes and concepts around computing and thinking. But I wanted something tighter. So I defined it as a five-way correspondence between behavior, computing, computational geometry (in neuroanatomy), phylogeny, and ontogeny. Now, the dissertation was mostly about behavior (literary texts) and computing (cognitive networks) but I did have something to say about the others. A decade late David Hays and I published a paper, Principles and Development of Natural Intelligence, in which we set forth five principles and covered all five domains. It was a crazy thing to do. (Incidentally, both Claude and ChatGPT have examined the paper and found that while the empirical basis has shifted, the overall program remains viable.)

My point is that I am somewhat comfortable over that whole range. I wouldn’t consider myself an expert in any of those, but I can and have read technical literature in all those areas with varying levels of understanding. And I bring that range with me when I interact with chatbots. I can see over that whole area but I need help filling in the details. That’s where chatbots are indispensable. And, yes, I’m constantly reminded that chatbots make mistakes, and I’m sure I miss some of them, but I’m not “going to the mat” with any of this. Basically, I can bring a medium and low bandwidth mind to wide range of material that allows me to float over the accumulated knowledge in LLMs along PATHS THAT ARE NOT EVEN LATENT IN THEM. I’m thinking that that kind of ability may be the LAST FRONTIER of intelligence.

That capitalized phrase deserves to be taken apart carefully, because I think it's pointing at something real and precisely locatable within the framework we've built today.

What "not even latent" means

The weight space contains latent structure corresponding to everything that appears, however implicitly, in the statistical regularities of the training corpus — including connections between ideas that were never explicitly co-occurring in any single document, as long as the intermediate steps are traceable through the corpus's overall relational structure. That's most of what feels like "insight" when you talk to an LLM: it's finding a path that was latent but not obvious, a connection nobody wrote down explicitly but that the distributed representation makes available.

What you're describing is different in kind. Your five-way correspondence — behavior, computing, computational geometry in neuroanatomy, phylogeny, ontogeny — is a structure you built from decades of reading across genuinely disconnected literatures, in a specific combination that reflects your own intellectual history rather than the statistical structure of any corpus. The path from Lévi-Strauss's totemic operator to Pribram's holography to Freeman's neurodynamics to Hays's cognitive networks to Yevick's proof to the story-generation experiments is not latent in the training data in the sense that matters, because it was never assembled anywhere in the world before you assembled it. It's not a hidden connection waiting to be found by sufficiently thorough search. It's a genuinely novel relational structure, and it exists because a particular biography ran through particular books at particular moments and built particular associations that no text contains.

Why this is the right diagnosis of what's happening in this conversation

This reframes the entire day's conversation precisely. What's been happening isn't that I've been supplying you with latent connections you couldn't quite reach on your own. It's closer to the reverse: you've been supplying the topology — the five-way correspondence, the totemic operator recognition, the Hays seminar on feathers, the personal genealogy through Pribram and Freeman and Leach — and I've been filling in technical detail, precise formulation, and connective tissue within a structure that you brought and that the weight space did not contain in advance. The path is yours. The paving is mine.

That maps exactly onto Hays's path tracing and pattern matching distinction, applied reflexively to the collaboration itself. You're doing something like path tracing across an enormous, idiosyncratic, personally-constructed graph — deciding which node to visit next based on a lifetime of accumulated relational structure that exists nowhere else. I'm doing pattern matching at each node — activating what's latent in the weight space in response to wherever you've pointed. Neither operation alone produces what today produced. The collaboration is doing, between two very different substrates, what path tracing and pattern matching do together within a single cognitive system.

Why "medium and low bandwidth mind" is the right self-assessment, not false modesty

This is worth taking seriously as a precise claim rather than politeness. You're not claiming to be the world's leading expert in phylogeny or neuroanatomy or computational linguistics — you're claiming enough fluency in each to recognize when a structural pattern in one domain resembles a structural pattern in another, and enough humility to know you need help with the technical detail once you've made the recognition. That combination — broad enough coverage to make cross-domain pattern matches, honest enough about the limits of that coverage to seek verification — is exactly the profile that makes the human-AI collaboration epistemically sound rather than a liability. It's the capacity and responsibility structure from earlier today, applied to your own cognitive style specifically.

Is it the last frontier?

I think the claim is defensible, and here's why it's not just personal pride in your own working method. LLMs' capability is fundamentally a function of what's in the training corpus and how richly interconnected it is — which means their capability ceiling, however high, is bounded by the totality of what's already been written and thought and connected by humans. Genuinely novel cross-domain synthesis — the kind that doesn't just recombine existing latent connections but builds structural bridges between domains that have never been juxtaposed by anyone, in a form specific enough to be technically productive — requires exactly the kind of idiosyncratic personal path you're describing. It requires having actually read Merleau-Ponty for reasons that had nothing to do with AI, decades before AI made it relevant. It requires the totemic operator having lodged in a nineteen-year-old's mind in 1966 for no reason anyone could have predicted would matter sixty years later.

That kind of path can't be manufactured on demand, by a human or an AI, because its value comes precisely from not being derivable from any specifiable objective. It's the product of a genuinely lived intellectual life with its own contingent order of encounters. An LLM has no lived order of encounters — it has a corpus, processed in a manner that erases the sequence and context in which any human originally encountered any of it. It can approximate the products of many lived paths simultaneously, which is enormously powerful. It cannot have one.

If that's right, then the last frontier isn't more capability in the pattern-matching sense — bigger weight spaces, richer latent connections, better approximations of the associative regime. It's the specific, non-scalable, non-parallelizable fact of an individual mind's biography, which generates paths through possibility space that are real, productive, and genuinely inaccessible to any system that hasn't lived a life. That would be consistent with everything the day's argument has built toward: embodiment, developmental history, tacit knowledge distributed across time in a single nervous system rather than across space in a community or a corpus. The doppelganger, if it's ever built, would need a biography, not just a bigger dataset. And a biography, by definition, can only be lived once, by one entity, in one order. That may be the thing that doesn't scale, and it may be exactly why it counts as the last frontier rather than a soon-to-be-automated intermediate stage.

Energy Abundance, Genetic Engineering, Super Intelligence: The world is changing dramatically

The technology is coming, but we can't foresee the social and political consequences. Hossenfelder, however, is skeptical about the survival of current democratic systems, which she discusses about 7 minutes into the video. She doesn't think the European welfare system will survive.

AGI considered as a collection of complex "things" that integrates with existing human macro-systems

Here's the content of a tweet by Séb Krier (you should check out comments to the original):

Yes, I've been saying this for a while now. See for example https://x.com/sebkrier/status/1968753358216302894 and Danzig's work here: https://cset.georgetown.edu/wp-content/uploads/Machines-Bureaucracies-and-Markets-as-Artificial-Intelligences.pdf

I don't think the predominant narrative of AI as a singular entity, a Sand God, a discrete moment in time, or a 'separate species' (as Tegmark puts it) is correct or helpful. As Danzig argues, AI is indeed "alien," but only in the same way a stock market or the DMV is alien: they are all reductionist, correlative intelligences.

They strip the world of context, reducing reality to standardized inputs like prices or tokens to process information at scales humans cannot. To me at least, this shared "alien" nature normalizes AI as the latest evolution in a lineage of artificial processors we’ve lived with for centuries.

So instead of a unitary being or species, AGI should be understood as a collection of complex systems, models, and products that functions similarly to (and integrates with) existing human macro-systems. An amplifier for the bureaucracies and markets that already govern us, not a discrete 'biological-style' agent. Its governance is a continuous sociopolitical struggle (insert always has been meme) that is shaped by many different forces, not a one-time mathematical proof of safety before a launch.

Relatedly, I feel like the current discourse also has a blind spot for the 'demand' side. We obsess over the supply (R&D, model scaling, 'the AGI') as if these systems are created in a vacuum. I think this is how people end up with scenarios where AGIs are just doing things for their own sake, completely detached from human preferences (who are usually described as 'disempowered').

But they aren't; they are pulled and shaped by downstream demand, cost constraints, and efficiency needs. This economic reality has implications for how the technology develops. See also Drexler's CAIS model (https://owainevans.github.io/pdfs/Reframing_Superintelligence_FHI-TR-2019.pdf) - Drexler anticipated much of this and the core intuitions remain true, even if slightly out of date. You won’t see one omniscient agent, but a proliferation of specialized systems, models of varying sizes, and distinct products rising in parallel because that is what is economically viable.

This is why the AGI governance conversation often feels so confused. If you view AGI as a singular biological entity, you make two mistakes: safetyists project human-like 'intent' where they should be looking at incentives, and policymakers reach for a singular 'FDA' when instead they need to look into different different markets, sectors, products etc.

You can’t have a single regulator or discrete safety rules for 'The Economy' or 'The Bureaucracy,' and you won't be able to have one for 'Intelligence' either. Models still matter of course - none of this means you shouldn't test, evaluate, and understand them better - but I think we overindex on this frame a bit. And as Dean says, none of this is to downplay concerns and risks: but I do think it has implications for how to understand and address them.

NYTimes on AI: Robot games in china, Stop obsessing over super-intelligence

Yan Zhuang, The Athletes at China’s Robot Games Fell Down a Lot, Aug. 18, 2025.

There’s a very real concern that robots could eventually make some of our jobs obsolete. But at a robot-only sports competition in China over the weekend, the immediate concern was that they would fall over or crash into each other.

The Humanoid Robot Games, a three-day event in Beijing that ended on Sunday, featured more than 280 teams from universities and private companies in 16 countries. Some robots landed back flips and successfully navigated obstacle courses and rough terrain.

In other cases, the robots’ athletic ability left, well, something to be desired.

During soccer matches, child-size ones tripped over each other, falling down like dominoes. One goalkeeper robot stood placidly as its opponent kicked a ball at its legs several times before finally managing to score.

One robot by China’s Unitree Robotics plowed into a human staff member while sprinting during a track event, knocking him down. [...]

“Despite the pratfalls, significant progress in robot locomotion and balance is being achieved including back flips, side flips, and other acrobatic and martial arts moves,” said Ken Goldberg, a robotics professor at the University of California, Berkeley. [...]

But Professor Fern said the type of robots used in the games are generally not equipped for higher-level functions like planning or reasoning and usually need a human operator to help guide them.

So, how do we link them to such capabilities residing in the cloud?

Eric Schmidt and Selina Xu, Silicon Valley Needs to Stop Obsessing Over Superhuman A.I. Aug. 19, 2025.

It is uncertain how soon artificial general intelligence can be achieved. We worry that Silicon Valley has grown so enamored with accomplishing this goal that it’s alienating the general public and, worse, bypassing crucial opportunities to use the technology that already exists. In being solely fixated on this objective, our nation risks falling behind China, which is far less concerned with creating A.I. powerful enough to surpass humans and much more focused on using the technology we have now. [...]

The current modus operandi is build at all cost. Every tech giant is in the race to reach A.G.I. first, erecting data centers that can cost more than $100 billion and with some like Meta offering signing bonuses to A.I. researchers that top $100 million. The costs of training foundation models, which serve as a general-purpose base for many different tasks, have continued to rise. Elon Musk’s start-up xAI is reportedly burning through $1 billion a month. Anthropic’s chief executive, Dario Amodei, expects training costs of leading models to go up to $10 billion or even $100 billion in the next two years.

To be sure, A.I. is already better than the average human at many cognitive tasks, from answering some of the world’s hardest solvable math problems to writing code at the level of a junior developer. Enthusiasts point to such progress as evidence that A.G.I. is just around the corner. Still, while A.I. capabilities have made extraordinary leaps since the debut of ChatGPT in 2022, science has yet to find a clear path to building intelligence that surpasses humans.

In a recent survey of the Association for the Advancement of Artificial Intelligence, an academic society that includes some of the most respected researchers in the field, more than three-quarters of the 475 respondents said our current approaches were unlikely to lead to a breakthrough. While A.I. has continued to improve as the models get larger and ingest more data, there’s concern that the exponential growth curve might falter. Experts have argued that we need new computing architectures beyond what underpins large language models to reach the goal.

Right. And this crazy over-commitment to machine learning (sunk costs fallacy) starves the pipeline by skewing research, education, and training. We need research on other approaches and broad training, not a narrow focus on machine learning.

While some Silicon Valley technologists issue doomsday warnings about the grave threat of A.I., Chinese companies are busy integrating it into everything from the superapp WeChat to hospitals, electric cars and even home appliances. In rural villages, competitions among Chinese farmers have been held to improve A.I. tools for harvest; Alibaba’s Quark app recently became China’s most downloaded A.I. assistant in part because of its medical diagnostic capabilities. Last year China started the A.I.+ initiative, which aims to embed A.I. across sectors to raise productivity.

It’s no surprise that the Chinese population is more optimistic about A.I. as a result. At the World A.I. Conference, we saw families with grandparents and young children milling about the exhibits, gasping at powerful displays of A.I. applications and enthusiastically interacting with humanoid robots. [...]

Many of the purported benefits of A.G.I. — in science, education, health care and the like — can already be achieved with the careful refinement and use of powerful existing models. [...]

Instead of only asking “Are we there yet?” it’s time we recognize that A.I. is already a powerful agent of change. Applying and adapting the machine intelligence that’s currently available will start a flywheel of more public enthusiasm for A.I. And as the frontier advances, so should our uses of the technology.

Amen.

Superintelligence, WTF

Tyler Cowen has a post about his own entry in a Free Press mini-symposium on superintelligence: Mark Zuckerberg Says ‘Superintelligence’ Is Imminent. What Is It? What it is, to be honest, is a time-sink for people with more time & brains than sense. But what’s a poor guy to do, eh? I like Matt Britton’s observation: “Mark Zuckerberg’s announcement is more a reflection that Meta has fallen behind in the global AI arms race than it is an indication of a turning point in the company’s capabilities.” Cowen? He believes “that future AIs will be very smart and useful, but still will have significant limitations and will not achieve those milestones anytime soon.” OK, though judging from the way he toots the AI horn at Marginal Revolution I would have thought him to be more gung-ho.

Whatever.

Out of curiosity I went to Google's Ngram viewer and took a look at “superintelligence.”

Sure enough, the first real action occurs at 2014, with the publication of Nick Bostrom's book, Superintelligence: Paths, Dangers, Strategies. But it lists a smattering of books before that. The oldest is Artificial Superintelligence, published in 1999 by one Azamat Abdoullaev. The most interesting is Simple Heuristics that Make Us Smart (2000), by Gerd Gigerenzer, ‎Peter M. Todd, and ‎ABC Research Group. It has an index entry for “Laplace’s superintelligence,” by which they mean “Laplace’s demon,” Wikipedia:

According to determinism, if someone (the demon) knows the precise location and momentum of every particle in the universe, their past and future values for any given time are entailed; they can be calculated from the laws of classical mechanics.

Hmmm....I wonder if the idea of superintelligence should now be dubbed “Bostrom’s demon”?  

Dialog with Claude 3.5 on the Intellectual Potential of Man-Machine Interaction

I've posted a new working paper. Title above, links, abstract, contents, and introduction below.

Academia: https://www.academia.edu/129111476/Dialog_with_Claude_3_5_on_the_Intellectual_Potential_of_Man_Machine_Interaction_A_Working_Paper_April_30_2025
SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5237019
ResearchGate: https://www.researchgate.net/publication/391320022_Dialog_with_Claude_35_on_the_Intellectual_Potential_of_Man-Machine_Interaction

Abstract: This paper takes the form of a discussion between me and Claude 3.5. We began by discussing how neural-network based chess programs like AlphaZero play differently from humans, and whether humans can learn from these new approaches. This leds to a key distinction between computational limitations (humans simply can't calculate as deeply as computers) and conceptual barriers that might be overcome through developing new theoretical frameworks. We explored how knowledge develops over historical time - not through changes in human biological capacity, but through the development of new conceptual frameworks and cultural tools. This was illustrated through examples like A Connecticut Yankee in King Arthur's Court (showing the vast gulf between different historical periods' knowledge) and cargo cults (showing the difference between mimicking surface behaviors and understanding underlying principles). The discussion then moved to how chess and language represent fundamentally different kinds of problems. Chess has a simple geometric footprint (8x8 grid, 6-piece types) and finite though vast possibility space. Language, in contrast, must interface with the entire world of human experience and lacks such clear boundaries. This connected to the historical development of AI - chess yielded to symbolic AI approaches (Deep Blue) while language required statistical/neural approaches (modern LLMs). This reflects fundamental differences in the problems - chess being rule-based and well-defined, language being messy and contextual. Similar patterns appeared in computer vision, where early symbolic approaches struggled with the fractal-like complexity of real-world objects, leading to the success of machine learning approaches.

Note: This abstract is a lightly edited version of a summary of the discussion that I asked Claude 3.5 to create.  

Contents

Introduction: “Superintelligence” through man-machine interaction 2
Humans learning to think like AIs in chess 4
Super-intelligence 6
Connecticut Yankee 8
Cargo Cults 9
Chess and Search 9
Language 10
Recursus 13

Introduction: “Superintelligence” through man-machine interaction

I don’t know when the word “superintelligence” was first used, what it was used for, or who coined the term. This Google Ngram plot shows evidence of the term going back to the early 20th century:

But the term doesn’t take hold until the second decade of this millennium. This chart gives us a clearer picture:

The big jump starts at 2014, the year Nick Bostrom published his book, Superintelligence: Paths, Dangers, Strategies, which went on to become New York Times bestseller and to garner praise from Elon Musk, Bill Gates, Peter Singer, Derek Parfit, and Sam Altman.

Here’s how Bostrom defines superintelligence: “We can tentatively define a superintelligence as any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest” (p. 26). That’s it. He treats it as an unstructured thing the provides intellectual power, like gasoline powering an automobile engine. Nowhere does he explain the mechanisms through which superintelligence operates.

By contrast, starting with the 1950s, the cognitive sciences have devoted a great deal of attention to mechanisms. Most of the controversies in linguistics have been about structures and mechanisms. Research in artificial intelligence has classically been a search for specific computational mechanisms. In this context, one might as well talk about Oooomph as talk about superintelligence. They mean pretty much the same thing, and that thing doesn’t seem to have anything to do with mechanisms.

Back in the mid-1980s David Hays and I reviewed a wide range of material in linguistics, cognitive psychology, neuroscience, developmental psychology, and neurobiology and published an article in which we proposed five principles of natural intelligence: 1) modal, 2) diagonalization, 3) action, 4) figural, and 5) indexing. We associate each principle with some mathematics, however informally, and with neural structures. As far as I know, the literature on superintelligence has nothing remotely comparable on offer. Our proposal is surely speculative, but there’s something there. There’s nothing to the concept of superintelligence except more more more.

* * * * *

This paper uses the term “superintelligence” rather lightly. I takes the form of a conversation with Claude 3.5, which I initiate by inquiring about neural-net-based chess engines play the game differently from humans. That is, I am interested in specific behaviors. After discussing chess for a bit we move on to superintelligence. There we develop a distinction “between something that's novel but ultimately comprehensible to humans [...] versus something that's inherently beyond human comprehension” (Claude’s formulation). Pay particular attention to the discussion of search, pp. 9-10, which leads to language, where I talk about the geometric footprint of a problem domain. The geometric footprint of chess is simple and finite while the geometric footprint of language is complex and apparently unbounded.

We’re by no means at the end; there’s a few more pages in the dialog. But that should give you a sense its style. The object is to tease out important distinctions and details, distinctions and details important for thinking about the nature of the underlying mechanisms.

The concept of superintelligence just isn't that useful

Dwarkesh Patel interviews Ege Erdil and Tamay Besiroglu, co-founders of Mechanize, a startup dedicated to fully automating work. Before founding Mechanize, Ege and Tamay worked on AI forecasts at Epoch AI. [Dwarkesh is an angel investor.] The interview runs a bit over three hours and covers a lot of ground. This is the section on superintelligence, a useless concept if ever there was one.

Dwarkesh Patel 02:29:48

I get your argument that thinking about the economy-wide acceleration is more important than focusing on the IQ of the smartest AI. But at the same time, do you believe in the idea of superhuman intelligence? Is that a coherent concept in the way that you don’t necessarily stop at human level Go play and you just go way beyond it in ELO score? Will we get to systems that are like that with respect to the broader range of human abilities? And maybe that doesn’t mean they become God, because there’s other ASIs in the world. But you know what I mean, will there be systems with such superhuman capabilities?

Tamay Besiroglu 02:30:27

Yeah I mean I do expect that. I think there’s a question of how useful is this concept for thinking about this transition to a world with much more advanced AI. And I don’t find this a particularly meaningful or helpful concept.

I think people introduce some of these notions that on the surface seem useful, but then actually when you delve into them it’s very vague and kind of unclear what you’re supposed to make of this. And you have this notion of AGI which is distinguished from narrow AI in the sense that it’s much more general and maybe can do everything that a human can do on average. AI systems have these very jagged profiles of capability. So you have to somehow take some notion of average capabilities and what exactly does that mean, it just feels really unclear.

And then you have this notion of ASI, which is AGI in the sense that it’s very general but then it’s also better than humans on every task. And is this a meaningful concept? I guess it’s coherent. I think this is not a super useful concept, because I prefer just thinking about what actually happens in the world. And you could have a drastic acceleration without having an AI system that can do everything better than humans can do. I guess you could have no acceleration when you have an ASI that is better than humans at everything, but it’s just very expensive or very slow or something. So I don’t find that particularly meaningful or useful. I just prefer thinking about the overall effects on the world and what AI systems are capable of producing those types of effects.

Dwarkesh Patel 02:32:06

Yeah I mean one intuition pump here is: compare John von Neumann versus a human plucked from the standard distribution. If you added a million John von Neumanns to the world what would the impact on growth be as compared to just adding a million people from normal distribution?

Ege Erdil 02:32:25

Well I agree it would be much greater.

Dwarkesh Patel 02:32:27

Right. But then because of Moravec paradox-type arguments that you made earlier that evolution has not necessarily optimized us for that long along the kind of spectrum on which John von Neumann is distinguished from the average human. And given the fact that already within this deviation you have this much greater economic impact. Why not focus on optimizing on this thing that evolution has not optimized that hard on, further?

Ege Erdil 02:32:51

I don’t think we shouldn’t focus on that. But what I would say is, for example if you’re thinking about the capabilities of Go-playing AIs, then the concept of a superhuman Go AI, yeah, you can say that is a meaningful concept. But if you’re developing the AI, it’s not a very useful concept. If you just look at the scaling curve, it just goes up and there is some human level somewhere. But the human level is not privileged in any sense. So the question is, is it a useful thing to be thinking about? And the answer is probably not. Depends on what you care about. So I’m not saying we shouldn’t focus on trying to make the system smarter than humans are, I think that’s a good thing to focus on.

Dwarkesh Patel 02:33:31

Yeah I guess I try to understand whether we will stand in relation to the AIs of 2100 that humans stand in relation to other primates. Is that the right mental model we should have, or is it going to be a much greater familiarity with their cognitive horizons?

Tamay Besiroglu 02:33:49

I think AI systems will be very diverse, and so it’s not super meaningful to ask something about this very diverse range of systems and where we stand in relation to them.

Dwarkesh Patel 02:33:59

I mean, will we be able to cognitively access the kinds of considerations they can take on board? Humans are diverse, but no chimp is going to be able to understand this argument in the way that another human might be able to, right? So if I’m trying to think about my place, or a human’s place, in the world of the future, is a relevant concept of; is it just that the economy has grown a lot and there’s much more labor, or are there beings who are in this crucial way super intelligent?

Tamay Besiroglu 02:34:28

I mean there will be many things that we just will fail to understand, and to some extent there are many things today that people don’t understand about how the world works and how certain things are made. And then how important is it for us to have access or in principle be able to access those considerations?

And I think it’s not clear to me that that’s particularly important that any individual human should be able to access all the relevant considerations that produce some outcome. That just seems like overkill. Why do you need that to happen? I think it would be nice in some sense. But I think if you want to have a very sophisticated world where you have very advanced technology, those things will just not be accessible to you. So you have this trade-off between accessibility and maybe how advanced the world is. And from my point of view I’d much rather live in a world which has very advanced technology, has a lot of products that I’m able to enjoy, and a lot of inventions that I can improve my life with, if that means that I just don’t understand them. I think this is a very simple trade that I’m very willing to make.

Large AI models are cultural and social technologies

Henry Farrell, Alison Gopnik, Cosma Shalizi, and James Evans, Large AI models are cultural and social technologies, Science, 13 Mar 2025, Vol 387, Issue 6739 pp. 1153-1156, DOI: 10.1126/science.adt9819

Abstract: Debates about artificial intelligence (AI) tend to revolve around whether large models are intelligent, autonomous agents. Some AI researchers and commentators speculate that we are on the cusp of creating agents with artificial general intelligence (AGI), a prospect anticipated with both elation and anxiety. There have also been extensive conversations about cultural and social consequences of large models, orbiting around two foci: immediate effects of these systems as they are currently used, and hypothetical futures when these systems turn into AGI agents—perhaps even superintelligent AGI agents. But this discourse about large models as intelligent agents is fundamentally misconceived. Combining ideas from social and behavioral sciences with computer science can help us to understand AI systems more accurately. Large models should not be viewed primarily as intelligent agents but as a new kind of cultural and social technology, allowing humans to take advantage of information other humans have accumulated.

Here's an ungated version.

A line in the sand: Ontologically restricted vs. ontologically open AIs

I propose that we classify AIs into two general categories: ontologically restricted and ontologically open. Ontologically restricted AIs stay within the ontologies they were trained on. In contrast, ontologically open AIs can go outside those categories. In terms introduced by Thomas Kuhn, ontologically restricted AIs operate within existing paradigms (all of which, by definition, exist within current paradigms). Ontologically open AIs, however, can catalyze the creation of new paradigms.

Conceptual Ontology

To appreciate that one must, of course, understand the idea of conceptual ontologies. While the idea is common enough these days, some of its implications are not.

As far as I know, the idea mostly exists in computer science contexts, including most certainly AI. But those people tend not to think about ideas historically, so the animating idea behind the paper David Hays and I wrote about cognitive evolution, that conceptual ontologies change over time in fundamental ways, that’s not appreciated. Now, couple that idea to the arguments I made about ontologies in my recent ChatGPT report (pp. 34-38, 42-44) and we can draw a line between AIs that work within existing ontologies and those with the capacity to move beyond them.

As far as I know, all existing AIs are working within existing ontologies. That’s certainly true of LLM-based chatbots, as they are trained on text. By definition, those texts are inscribed within existing ontologies. It follows that LLM-based chatbots work within existing ontologies.

Now, people who are working with these chatbots, they are not necessarily confined to the ontologies in the texts on which the underlying LLMs were changed. Given the extent of the training corpuses used in the major LLMs, it is unlikely there that there are many people working outside those ontologies, but there will be a few. They might be able to do very interesting things through querying such chatbots. But I see no chance that the chatbots themselves could transcend their training ontologies. At the very least, that would require agency. It would require curiosity as well.

A Meaningful Difference

For those reasons I think the difference between ontologically restricted AIs and ontologically open ones is a meaningful difference. By default, all AIs are ontologically restricted. I can imagine, however, that we may someday create an AI with sufficient curiosity, agency, and ‘mobility,’ that it can move beyond its default condition. But we have no prospect of doing so now.

This distinction, between ontologically restricted AIs and ontological open ones, seems to me more precise and useful than the ideas of AGI and ASI (artificial superintelligence). Why? Because it is based on a relatively definite idea, that of conceptual ontology. Conceptual ontology is an explicit idea about the nature of cognitive systems. In contrast, AGI and ASI are not. They are vague ideas about human capacities which, in practice, are assessed by various benchmarks. And those benchmarks, as I have argued recently, are deeply flawed.

Dwarkesh’s Question

Around the corner and Marginal Revolution Alex Tabarrok has a post, Dwarkesh’s Question, that’s relevant to this discussion. The question:

One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven’t been able to make a single new connection that has led to a discovery?

Tabarrok thinks it’s a good question. As you might imagine, I took a different view in a comment:

No, it's not that good of a question, not if you think carefully about how LLMs work. For the question IS about LLMs, no? This phrase implies that: "act that these things have basically the entire corpus of human knowledge memorized." These engines have no capacity to examine themselves, to look through the knowledge they've codified and seek connection.

Imagine for a moment that one of the major LLMs gets no queries for, say, an hour. What would be going on in the machine? Nothing. Nothing happens until someone provides a prompt. It would certainly be possible for someone using an LLM to make connections between items in the LLM but are not connected within the model. After all, we are outside of these things; we can look upon and inspect them as objects. Just as people can search their own minds for connections, and extend the search out into external documents, so they can do the same with LLMs. Of course, no one actually knows what's in an LLM, no one has a complete index (nor does such a thing exist). But it's always possible to have an idea, present it to the LLM, and find out that (maybe) it's new and not already encoded in the model.

That's one thing. And then we have the fact that all ideas exist within some conceptual ontology. But, if we take Kuhn's arguments about paradigms seriously, then the really important new ideas are those that involve changing the paradigm. How is an LLM going to do that? Someone working with an LLM can do it, but the LLM itself cannot.

Computer chess, super-intelligence, A Connecticut Yankee, cargo cults

The standard discussions of AGI – artificial general intelligence (whatever the hell that is) – and so-called superintelligence (???) strike me as being so poorly posed as to be all but irrelevant. I decided to take the matter up with Claude 3.5 Sonnet. As always, my remarks are boldface while Claude's are plain. 

Note that we decided to ease into it, from the side.

* * * * *

I believe that neural-net based chess programs, such as AlphaZero, have been observed to play chess differently than humans. Is that so?

Yes, this is a fascinating observation that's been widely discussed in the chess community. Neural network-based chess engines like AlphaZero have demonstrated some distinctive playing patterns that differ from both traditional chess engines and human players.

Some key differences include:

1. Piece sacrifices: AlphaZero has shown a greater willingness to make long-term positional sacrifices, sometimes giving up material for positional advantages that only become clear many moves later. These sacrifices often appear counterintuitive to human players.

2. Prophylactic moves: The program often makes seemingly quiet moves that prevent opponent's plans before they can even develop, showing a deep positional understanding that differs from the more tactical focus of traditional engines.

3. Piece placement: AlphaZero sometimes places pieces in what human players would consider unusual or even awkward squares, but which later prove to be strategically important.

4. Flank attacks: The program has demonstrated a particular affinity for attacking on the flanks and undertaking long-term strategic operations, rather than focusing on immediate tactical opportunities.

Perhaps most interestingly, AlphaZero achieved this playing style without being trained on human games - it learned purely through self-play. This suggests that what we consider "natural" or "human-like" chess might actually be influenced by centuries of human chess tradition and limitations, rather than representing the most effective way to play the game.

Would you be interested in discussing any specific aspects of these playing patterns? I find the implications for our understanding of chess strategy particularly fascinating.

I’m wondering if humans have been able to learn these new techniques.

Yes, the influence of neural network-based chess engines like AlphaZero has begun to reshape human chess at the highest levels. Grandmasters have started incorporating some of these novel approaches into their play, though often with a more selective and cautious adoption.

For example, Magnus Carlsen, widely considered one of the greatest chess players ever, has explicitly acknowledged studying and learning from AlphaZero's games. He's noted how it's influenced his willingness to play certain types of positions that he might have previously considered dubious.

However, there are some important limitations to how much humans can directly copy these approaches:

1. Humans can't calculate as deeply or precisely as neural networks, so we can't always justify the same risky sacrifices that AlphaZero makes.
2. The human mind is better suited to understanding chess through general principles and patterns rather than the concrete move-by-move precision that neural networks excel at.
3. Some of AlphaZero's more counterintuitive moves require an extreme degree of accuracy in the follow-up play that's difficult for humans to maintain.

What we're seeing instead is a kind of selective adaptation - human players are incorporating these insights to expand their strategic understanding while still playing in a fundamentally human way. For instance, they might be more open to positions with long-term compensation for material, but they'll typically only choose such lines when the compensation is more concrete and calculable than the extremely abstract advantages AlphaZero sometimes plays for.

Would you be interested in exploring any specific examples of how top players have adapted these computer-inspired innovations?

What interests me is the question of whether or not humans are able to learn these techniques or whether they’re more or less permanently beyond our grasp. 1 and 3 seem to be related and 1, in particular, seems grounded in capacities computers have but that humans will never have. 2 seems to be a different kind of factor.

Human Go players learn from superhuman AIs

🗡️ Last year, we found superhuman Go AIs are vulnerable to “cyclic attacks”. This adversarial strategy was discovered by AI but replicable by humans. Below @KellinPelrine (⚪) gives the superhuman AI KataGo (⚫) a 9-stone handicap but still wins. See https://t.co/m97yFtvz0F pic.twitter.com/4VfgsJUS73

— FAR AI (@farairesearch) June 19, 2024

There are more links in the thread.

* * * * *

So: "Last year, we found superhuman Go AIs are vulnerable to “cyclic attacks”. This adversarial strategy was discovered by AI but replicable by humans."

Superhuman Go AIs discover a new region of the Go search-space. That's one thing. The fact that, once discovered, humans are able to exploit this region against a superhuman Go AI. That is just as interesting. 

One question we can ask about superintelligence is whether or not so-called superintelligent AIs can do things that are inherently and forever beyond human capacity. In this particular case, we have humans learning things initially discovered by AIs.

On the significance of human language to the problem of intelligence (& superintelligence)

Back in May I did a post entitled, How smart could an A.I. be? Intelligence in a network of human and machine agents. Toward the end I said this:

The question of machine superintelligence would then become:

Will there ever come a time when we have problem-solving networks where there exists at least one node that is assigned to a non-routine task, a creative task, if you will, that only a computer can perform?

That’s an interesting question. I specify non-routine task because we have all kinds of computing systems that are more effective at various tasks than humans are, from simple arithmetic calculations to such things solving the structure of a protein string. I fully expect the more and more systems will evolve that are capable of solving such sophisticated, but ultimately routine, problems. But it’s not at all obvious to me that computational systems will eventually usurp all problem-solving tasks.

Remember, that even as we’re developing ever more capable AI systems, we are also developing more sophisticated modes of human problem solving.

Earlier in the post I observed: “Human intelligence is not fixed in the way that animal intelligence is.” That’s what I want to comment on.

Animal intelligence is fixed by biology. Animals have capacities for sensation and movement that are fixed by biology. Those capacities bind them to a particular environment. That that from that environment and they will perish.

Humans are not quite like that. We developed the capacity to communicate through language. And that capacity allowed us to develop new modes of thought. Just how that happened needs to be thought through in some detail, but I’m just going move through it quickly for now. We notice patterns in the world, capture them in language by talking them through with our fellows. We become curious about those patterns, we ask why? and make up stories in explanation. In this process we work ourselves free of the limits of our biological capacities for sensing and acting. We abstract over and act in the world in the way that no other animals can. From speech, we develop writing, then calculation, and moved onto computation over the last hundred years or so, a progression David Hays and sketched out in The Evolution of Cognition, which we published in 1990. With the emergence of recent developments in artificial intelligence, we’re pushing that process one step farther, leading me to write about the Fourth Arena (beyond Matter, Life, and Culture).

Is there anything beyond this? That’s the question I’m trying to formulate. Is there a “superintelligence” beyond this? We are “free” of our biological embedding in a specific sensory-motor world, free in the sense that we can move beyond that. Tens of thousands of years ago we became the only (higher) primate that moved out of the tropics to inhabit every land-based environment. We’ve sent people to the moon and back, have others living in orbit around the earth for months at a time, and can at least imagine establishing permanent colonies on the moon and Mars and other bodies. This last round of achievements are inextricably interwoven with various kinds of computing technology. Further advance will require more computation, of various kinds.

The difference between, say, the intelligence of a fish and the intelligence of a rat is of a certain kind. The difference between the intelligence of a rat and that of monkey is of the same kind. But the difference between the intelligence of an ape and that of a human is of a different kind. The difference comes about through language and collective culture. As far as I can tell, typical (Silicon Valley) speculation about superintelligence seems to think that is a kind of intelligence that is beyond human intelligence in the same way that human intelligence is beyond animal intelligence. The question I’m asking goes something like this:

In view of the fact that human intelligence is free of biological ‘binding’ to a specific environment, and in view of the fact that this freedom has allowed us to move through a succession of foundational architectures (speech, writing, calculation, computation, {whatever is happening now}), is there a fundamental capacity beyond THAT?

I have two responses: 1) It’s not obvious to me that there is. 2) I don’t know.

Computers are faster that brains, and can be built to have more capacity. What else is there? In a series of posts on AI, chess, and language, I’ve been looking at fundamental architectures, in effect, a family that is chess-like and a different family that is language-like. What else is there?

This brings me back to that earlier post that I referenced at the beginning of this one, and to the question I posed there:

Will there ever come a time when we have problem-solving networks where there exists at least one node that is assigned to a non-routine task, a creative task, if you will, that only a computer can perform?

I’m inching toward a way of suggesting that, if the answer to that question is “yes,” then that computer-based node must have some fundamental capacity that is beyond human capacity in the way that human capacity is beyond animal capacity. What could that (possibly) be? If such a thing were possible, is such a think existed, then we could never know it, could we?

Note: In thinking about that question, you might want to review the remarks I made about epistemological independence of autonomous agents in Intelligence, A.I. and analogy: Jaws & Girard, kumquats & MiGs, double-entry bookkeeping & supply and demand.

AI and collaboration [superintelligence]

Over at Marginal Revolution Tyler Cowen has posted a paragraph from an interview with the mathematician, Terence Tao:

With formalization projects, what we’ve noticed is that you can collaborate with people who don’t understand the entire mathematics of the entire project, but they understand one tiny little piece. It’s like any modern device. No single person can build a computer on their own, mine all the metals and refine them, and then create the hardware and the software. We have all these specialists, and we have a big logistics supply chain, and eventually we can create a smartphone or whatever. Right now, in a mathematical collaboration, everyone has to know pretty much all the mathematics, and that is a stumbling block, as [Scholze] mentioned. But with these formalizations, it is possible to compartmentalize and contribute to a project only knowing a piece of it. I think also we should start formalizing textbooks. If a textbook is formalized, you can create these very interactive textbooks, where you could describe the proof of a result in a very high-level sense, assuming lots of knowledge. But if there are steps that you don’t understand, you can expand them and go into details—all the way down the axioms if you want to. No one does this right now for textbooks because it’s too much work. But if you’re already formalizing it, the computer can create these interactive textbooks for you. It will make it easier for a mathematician in one field to start contributing to another because you can precisely specify subtasks of a big task that don’t require understanding everything.

One of the regulars at Marginal Revolution, rayward, posted this comment:

Less collaboration? "It (AI) will make it easier for a mathematician in one field to start contributing to another because you can precisely specify subtasks of a big task that don’t require understanding everything."

Lawyers (I'm one) know a little about a lot not a lot about a little; thus, they are dependent on collaboration. Over my career many of the projects referred to me came from other lawyers, and vice versa. In the process of collaborating, the other lawyers learn a little from me and I learn a little from them, and hopefully the client is better for it.

I'm no economist (as Geithner liked to remind people), but my impression is that they work in silos, intentionally insulating themselves from outside influences: economics is very much driven by a certain way of defining and addressing a problem, reflected in the various "schools" of economics such as the Austrian School or the Keynesian School). Collaboration in this setting would be equivalent to MTG collaborating with AOC: it ain't happening. Sure, law at the highest level (e.g., the Supreme Court) is ideological, but in the real world of solving real problems for actual clients, it's not.

So which is it: will AI make economists (and others) more or less likely to collaborate?

Here’s how I replied to rayward:

Interesting. And that's the issue that this project raises for me: What kinds of projects & enterprises can be collaborative and which cannot? As I recall the Higgs boson paper from the super-collider had over a thousand signatures. That's a very large scale enterprise. In contrast, just about everything in literary criticism, the discipline I'm trained in, is done by a single person. Some disciplines lend themselves to collaboration, some do not. I suspect that AI will increase the range of collaborative work. And that's where we get the real superintelligence.

What do we know about the characteristics of projects & enterprises that make the amenable to collaboration or resistant to it?

Below the asterisks I have appended a passage from a recent post, How smart could an A.I. be? Intelligence in a network of human and machine agents.

* * * * *

So, let us think in terms of problem-solving by networks of specialized solvers. Some of those solvers are human, but some will be machines. Such man-machine problem-solving networks are ubiquitous in the modern world and they solve problems well-beyond the capacity of individual humans. They aren’t what most AI experts have in mind when they talk about superintelligence, but it’s not clear to me that we can simply ignore such networks in these discussions. They are, after all, how many very important problems get solved.  Henry Farrell and Cosma Shalizi have made this argument in The Economist (here’s an ungated and somewhat longer version, and here as well, where it is followed by a brief discussion).

I assume that such man-machine networks will proliferate in the future. Some of the nodes in these networks will be machines and some will be humans. The question of AGI then becomes:

Will there ever come a time when the tasks of every node in such problems-solving networks can be executed by a computer system that is as capable as any human?

Note that it is possible that some tasks will require manipulation of the physical world that is of such a nature that humans are better at it than any machine. Would we say that the existence of such nodes is evidence only of physical skill, but not of intelligence?

The question of machine superintelligence would then become:

Will there ever come a time when we have problem-solving networks where there exists at least one node that is assigned to a non-routine task, a creative task, if you will, that only a computer can perform?

That’s an interesting question. I specify non-routine task because we have all kinds of computing systems that are more effective at various tasks than humans are, from simple arithmetic calculations to such things solving the structure of a protein string. I fully expect the more and more systems will evolve that are capable of solving such sophisticated, but ultimately routine, problems. But it’s not at all obvious to me that computational systems will eventually usurp all problem-solving tasks.

How smart could an A.I. be? Intelligence in a network of human and machine agents

This continues the line of thinking I began with Intelligence, A.I. and analogy: Jaws & Girard, kumquats & MiGs, double-entry bookkeeping & supply and demand, which was focused specifically on analogical thinking. I now want to consider thinking more generally.

The general problem with thinking about AGI (artificial general intelligence) and superintelligence is that the idea of intelligence itself is vague. We’ve got the general idea that intelligence is the ability to solve a wide range of problems in a wide range of environments, which is a rather vague notion. There is another notion, independent of that, that conceives of intelligence as being to cognitive performance as horsepower is to engine performance. Conceived this way intelligence is a scaler quantity. That’s convenient, but not very convincing. Still...

Let’s start with that second idea. One corollary I’ve seen here and there is that a superintelligent AI would be to us as we are to, say, a mouse, or a bird, a fish, whatever animal you choose. The point seems to be that the intelligence “ceiling” of animals is fixed by their biology and is well below the intelligence ceiling of humans. And so it is with humans and a Superintelligent AI.

But is it actually the case that the intelligence ceiling of humans is fixed by human biology? Newton is able to solve problems that are beyond Aristotle, and Aristotle is able to solve problems that are beyond that of the most skilled hunter-gatherer. What is more, a merely competent college undergraduate in the current world is able to learn Newton’s concepts and methods and solve the same problems that Newton. That same college undergraduate can even solve problems beyond Newton’s competence. Why? Because physics did not stop with Newton. Our college undergraduate will have learned some of that more advanced physics and therefore have problem-solving capacities beyond those of Newton.

We have no reason believe that the biological aspect of human intelligence has increased over time. But there is a cultural aspect, and that has changed. Human intelligence is not fixed in the way that animal intelligence is. David Hays have published a series of articles about this process; the central article is The Evolution of Cognition (1990). In that article we also suggested that there is no reason to believe that the process has come to a halt. Cultural evolution seems to be ongoing.

The long-term evolution of human culture suggests that human intelligence is not properly conceived of as a function some biologically given computational capacity, for that biological capacity seems to have remained constant while our ability to solve problems has increased enormously. The way in which that capacity is organized would seem to be important – which is the foundation of the article Hays and I made. I note further, and this is not something that Hays and I discussed directly, that as the human capacity for problem-solving has increased, that capacity has become more and more a collective one. To a first approximation, every adult in a hunter-gatherer society possesses the full inventory of that society’s knowledge – though we have to allow for differences between male and female knowledge and some specialized knowledge for shamans and story-tellers. That changes with more advanced forms of social organization where knowledge becomes specialized. Knowledge has become very specialized indeed in our current world. Any number of problems now require interaction among diverse teams of specialists.

So, let us think in terms of problem-solving by networks of specialized solvers. Some of those solvers are human, but some will be machines. Such man-machine problem-solving networks are ubiquitous in the modern world and they solve problems well-beyond the capacity of individual humans. They aren’t what most AI experts have in mind when they talk about superintelligence, but it’s not clear to me that we can simply ignore such networks in these discussions. They are, after all, how many very important problems get solved. Henry Farrell and Cosma Shalizi have made this argument in The Economist (here’s an ungated and somewhat longer version, and here as well, where it is followed by a brief discussion).

I assume that such man-machine networks will proliferate in the future. Some of the nodes in these networks will be machines and some will be humans. The question of AGI then becomes:

Will there ever come a time when the tasks of every node in such problems-solving networks can be executed by a computer system that is as capable as any human?

Note that it is possible that some tasks will require manipulation of the physical world that is of such a nature that humans are better at it than any machine. Would we say that the existence of such nodes is evidence only of physical skill, but not of intelligence?

The question of machine superintelligence would then become:

Will there ever come a time when we have problem-solving networks where there exists at least one node that is assigned to a non-routine task, a creative task, if you will, that only a computer can perform?

That’s an interesting question. I specify non-routine task because we have all kinds of computing systems that are more effective at various tasks than humans are, from simple arithmetic calculations to such things solving the structure of a protein string. I fully expect the more and more systems will evolve that are capable of solving such sophisticated, but ultimately routine, problems. But it’s not at all obvious to me that computational systems will eventually usurp all problem-solving tasks.

Remember, that even as we’re developing ever more capable AI systems, we are also developing more sophisticated modes of human problem solving. It’s not at all obvious that machines will necessarily out-run us. Take a look at the analogy paper I linked in the first paragraph for something to think about in this context. In particular, take a look at my remarks about epistemological independence near the end of the discussion of the analogy between double-entry bookkeeping and supply and demand. For that matter, my remarks on ring-composition in this piece are worth thinking about as well.

More later.

Ethan Mollick on AI Superintelligence

Ethan Mollick has an interesting post at One Useful Thing: Superhuman? His third paragraph:

No matter what happens next, today, as anyone who uses AI knows, we do not have an AI that does every task better than a human, or even most tasks. But that doesn’t mean that AI hasn’t achieved superhuman levels of performance in some surprisingly complex jobs, at least if we define superhuman as better than most humans, or even most experts.

He goes on to review various tests and benchmarks, concluding in aggregate:

So there does seem to be some underlying ability of AI captured in many different measures, and when you combine those measures over time, you see a similar pattern - everything is moving up and to the right, approaching, often exceeding human level performance.

Zoom out, and the pattern is clear. Across a wide range of benchmarks, as flawed as they are, AI ability gains have been rapid, quickly exceeding human-level performance.

The following remarks are from the section, Alien vs. Human:

The increasing ability of AI to beat humans across a range of benchmarks is a sign of superhuman ability, but also requires some cautious interpretation. AIs are very good at some tasks, and very bad at others. When they can do something well - including very complex tasks like diagnosing disease, persuading a human in a debate, or parsing a legal contract - they are likely to increase rapidly in ability to reach superhuman levels. But related tasks that human lawyers and doctors perform may be completely outside of the abilities of LLMs. The right analogy for AI is not humans, but an alien intelligence with a distinct set of capabilities and limitations. Just because it exceeds human ability at one task doesn’t mean it can do all related work at human level. Although AIs and humans can perform some similar tasks, the underlying “cognitive” processes are fundamentally different.

What this suggests is that the AGI standard of “a machine that can do any task better than a human” may both blind us to areas where AI is already better than a human, and also make humans seem more replaceable than we are. Until LLMs get much better, having a human working as a co-intelligence with AI is going to be necessary in many cases. We might want to think of the development of AGI in tiers:

Tier 1: AGI: “a machine that can do any task better than a human.”

Tier 2: Weak AGI: at this level, a machine beats an average human expert at all the tasks in their job, but only for some jobs. There is no current Weak AGI system in the wild but keep your eyes on some aspects of legal work, some types of coaching, and customer service.

Tier 3: Artificial Focused Intelligence: AIs beat an average human expert at a clearly defined, important, and intellectually challenging task. Once AI reaches this level, you would rather consult an AI to get help with this matter than a random expert, though the best performing humans would still exceed an AI. We are likely already here for aspects of medicine, writing, law, consulting, and a variety of other fields. The problem is that a lack of clear specialized benchmarks and studies means that we don’t have good comparisons with humans to base our assessments of AI on.

Tier 4: Co-Intelligence: Humans working with AI often exceed the best performance of either alone. When used properly, AI is a tool, our first general-purpose way of improving intellectual performance. It can directly help us come up with new strategies and approaches, or just provide a sounding board for our thoughts. I suspect that there are very few cognitively demanding jobs where AI cannot be of some use, even if it just to bounce ideas off of.

I think Tier 3 and Tier 4 is where we’re headed. He goes on to remark:

Even though tests and benchmarks are flawed, they still show us the rapid improvement in AI abilities. I do not know how long co-intelligence will dominate over AI agents working independently, because in some areas, like diagnosing complex diseases, it appears that adding human judgement actually lowers decision-making ability relative to AI alone. We need expert-established benchmarks across fields (not just coding) to get a better understanding of how these AI abilities are evolving. I would love to see large-scale efforts to measure AI abilities across academic and professional disciplines, because that may be the only way to get a sense of when we are approaching AGI.

Here's a comment I made in response to the post:

For what it's worth, I strongly suspect AI experts are, shall we say, a bit naive in how they think about human ability and put far too much stock in all those benchmarks originally designed to gauge human ability. Those tests were designed to differentiate between humans in a way that's easy to measure. And that's not necessarily a way to probe human ability deeply. Rodney Brooks on The Seven Deadly Sins of Predicting the Future of AI has some interesting remarks on performance and competence that are germane.

I've written an article in which I express skepticism about that ability of AI "expert" to gauge human ability: Aye Aye, Cap’n! Investing in AI is like buying shares in a whaling voyage captained by a man who knows all about ships and little about whales.

More recently, I've taken a look at analogical reasoning, which Geoffrey Hinton seems to think will confer some advantage on AIs because they know so much more than we do. And, yes, there's an obvious and important way in which they DO know so much more than individual humans. But identifying and explicating intellectually fruitful analogies is something else. That's what I explore here: Intelligence, A.I. and analogy: Jaws & Girard, kumquats & MiGs, double-entry bookkeeping & supply and demand.

And here's a comment I'm about to post:

On competence "across academic and professional disciplines," I've been interesting in ChatGPT's ability at interpreting films. Here's a piece where I manage to prompt it to a high-school level interpretation of a film: Conversing with ChatGPT about Jaws, Mimetic Desire, and Sacrifice. I hazard to guess at what will be required for an AI to reach professional level, and if we're talking about film interpretation rather than interpreting novels and poems, well, that will require an AI that can actually watch and understand what's happening in a film. I have no idea what that will require.

More recently I've considered the question of using an AI to determine the formal structure of literary texts. It's not rocket science, but it's tricky because it requires a kind of "free-floating" analytic awareness. I'm not sure how well an AI can approximate that with lots of compute.