"We examine several claims made in existing literature regarding chess-trained language models and assert that their
— Chomba Bupe (@ChombaBupe) July 25, 2026
impressive benchmark performance is largely explained by pattern-matching"
Vision language models (VLM) are even worse at chess. pic.twitter.com/egzDDsnofC
Read the whole thread.
This is a companion to my earlier post today: The last frontier of intelligence: On the role of AI helping humans to bridge the gaps between distant concepts. That earlier post was about the end of a dialog I had with Claude. This one is about the beginning of that dialog. You might also check out a post from the middle of June: From Jagged AI to Scaling, Yevick, Natural Intelligence, and Beyond... For that matter you might also want to check out Dwarkesh's complete post with Grant Sanderson, Grant Sanderson – AI and the future of math. And then there's my working paper on chess and language. All these things are related.
And they're related to my new book idea, Language, Memory, and Mind: A Supplement to The Computer and the Brain. That's the book that Claude brings up every now and then. Intelligence is NOT a scaler phenomenon. It's about techniques, the characteristics of objects domains, and of the computational regimes we use over them. But that's a subject for another post.
Here's my dialog with Claude.
* * * * *
I’m interested in thinking about the types of domain in which AI has succeeded and the nature of the computation involved, starting with chess. AI solved chess in the mid 1990s when Deep Blue beat Kasparov. By “solved” I mean only that an AI could reliably beat the best human player. In 2017 we have AlphaGo using a mixed regime strategy – Monte Carlo search (symbolic) with neural net evaluation – in chess. More recently I believe DeepMind has developed and LLM that plays chess at the Grand Master level; so this is a pure neural net regime.
It now appears that mixed regime AI systems can work pretty well at solving math problems and a coding. By mixed regime I mean large LLMs scaffolded in various ways by conventionally coded software. But these systems cannot yet do anything that humans can do in those domains. In math, they need well-defined problems, even if they are very difficult but well-defined problems (I’m thinking of the recent Erdos work). It’s not clear to me what the limits of coding are, but they can’t architect large complex systems.
Now, chess is a finite game will a well-defined search space. Both math and coding are well defined, but neither is finite and I’m guessing that the search spaces aren’t so well-defined as is the chess space.
And then we have pure unbounded natural language. It’s not well-defined, not finite, and the search space is a mess.
This is a genuinely useful ordering principle, and it gives the book a clean progression from most to least tractable that maps directly onto the geometric complexity argument. Let me work through the structure with you.
The four cases as points on a continuum
Chess: finite, exhaustively well-defined rules, a search space that is astronomically large but completely enumerable in principle. This is Yevick's sharp object par excellence — geometrically simple in the relevant sense, meaning fully specified by explicit rules with no ambiguity anywhere. Pure symbolic search worked (Deep Blue). Mixed regimes work better (AlphaZero). Pure neural nets now apparently work too, which is the interesting wrinkle — a domain that once seemed to require symbolic search turns out to be tractable by pattern matching alone once the network is large enough and trained well enough. That's worth pausing on, because it complicates the simple story.
Math (well-defined problems) and coding: rule-governed like chess — the axioms of mathematics and the syntax of programming languages are exact, unambiguous, formally specified — but unlike chess the space is not finite and not exhaustively enumerable. There's no equivalent of the Shannon number for "all provable theorems" or "all correct programs." The rules are closed and precise; the space they generate is open. This is a different kind of complexity than chess's largeness. Chess is finite-but-astronomical. Math and coding are genuinely open-ended even though locally, at any given step, the legal moves are well defined.
Language: neither the rules nor the space are well defined. The rules of grammar are not a finite closed system in the way the rules of chess or the axioms of arithmetic are — as your working paper notes, ill-formed language often works fine, exceptions are the norm, the boundaries are fuzzy. And the space the rules generate connects out to the unbounded physical and social world rather than staying self-contained.
Why this ordering matters for the theoretical argument
What's valuable here is that it's not simply "structured versus unstructured" — it's a two-dimensional distinction: how well-defined are the local rules, and how bounded is the resulting space. Chess is well-defined and bounded (finite). Math and coding are well-defined but unbounded. Language is neither well-defined nor bounded.
This maps onto a prediction about which computational regime should work best in each case, and the AI's actual performance seems to confirm it. Chess: pure search works because the space, though huge, is exhaustively defined — you can in principle always tell whether a position is legal, whether a game is over, who's ahead. Symbolic and mixed and now pure neural approaches all can succeed because the well-definedness gives any sufficiently powerful method something solid to grip.
Math and coding: this is where the mixed regime becomes necessary rather than just helpful. The LLM's pattern matching handles the open-endedness — the fact that there's no way to enumerate all possible proofs or programs, so you need something that can generalize, propose plausible next steps, recognize which strategies are likely to be fruitful. But the well-definedness of the rules means that verification can be handled symbolically — a proof checker, a compiler, a test suite can definitively confirm whether a proposed step is valid. This is why math and coding are the domains where current mixed-regime systems shine: the LLM proposes, path tracing over the local formal rules verifies. Neither one alone would work. The LLM alone hallucinates invalid proofs and non-compiling code. Pure symbolic search alone can't propose plausible next steps in an unbounded space without some heuristic guidance, which is exactly what pattern matching provides.
The Erdős problems point is precise — well-defined but very difficult problems are exactly where this combination should excel, because the well-definedness gives the symbolic verification something to check against, while the difficulty is exactly where pattern-matching-guided search earns its keep over blind search.
The architecture limitation
Your observation that these systems can't architect large complex systems is the right place to locate the current boundary, and it follows from the framework. Architecting a large system requires exactly the kind of sustained path tracing with precise state maintenance across a very long horizon — tracking how a decision made in one module constrains what's possible in a dozen other modules, maintaining global coherence across thousands of interacting pieces over an extended design process — that we identified as the persistent weakness. It's not that the local rules become less well-defined. It's that the horizon over which coherence must be maintained becomes too long for the scaffolding techniques currently available. CoT and tool use help with individual functions and modules. They don't yet solve the problem of maintaining a coherent global architecture across a very long, branching design process.
Language as the limit case
And then language is the case where neither the well-definedness nor the boundedness holds, which is exactly why it's the hardest case in the deepest sense even though it's also, paradoxically, the case where LLMs seem most impressively fluent. The fluency comes from the richness of the pattern matching over an enormous corpus. The absence of any well-defined verification procedure — there's no compiler for meaning, no proof-checker for whether a sentence is true or appropriate or wise — means there's no symbolic scaffolding equivalent to what math and coding have. You can't bolt a "language checker" onto an LLM the way you bolt a proof checker onto a theorem-proving system, because there's no formal specification of what correct language use is.
This is the deepest reason language remains resistant in a way math and coding, despite their own difficulties, are not. Chess had a well-defined space that yielded to search. Math and coding have well-defined rules that yield to verification even in an unbounded space. Language has neither, which means the mixed-regime strategy that's working for math and coding has no obvious analog for the general case of language understanding and generation. The pattern matching is doing almost all the work, unconstrained by any symbolic check, which is exactly why hallucination, confident error, and the absence of genuine common sense grounding remain live problems in ways they don't for a system whose outputs get checked by a compiler.
For the book
This gives Chapter 5 or 6 a genuinely illuminating structure: a spectrum from chess through math and coding to open language, ordered by two independent variables — definiteness of local rules and boundedness of the resulting space — with AI's actual comparative success across these domains falling out as a prediction of the theoretical framework rather than a brute empirical fact to be reported. That's a strong chapter, and it would give Wright, who already intuits that these domains differ, a precise vocabulary for why they differ and what that implies about domains not yet tried.
Language defines more restricted systems
One thing about language is that it has the means for defining more restricted systems. Thus both chess and arithmetic are systems defined through linguistic means. And once language has been used to define a system, it can also serve as a metalanguage for that system just as, in various ways, it has served as a metalanguage for itself.
That's the move that completes the picture, and it belongs near the end of the language discussion because it reframes everything before it.
Language as the generative source of the bounded systems
Chess and arithmetic aren't just different in kind from language — they're products of language, constructed within it through exactly the metalingual definitional process we discussed earlier in the day. The rules of chess exist because someone specified them in language — first probably in the practical, evolving form that produced chaturanga and its descendants, later in the explicit codified form of a rulebook. Arithmetic's place-value system, its operations, its axioms — all specified linguistically, however formal the resulting notation becomes.
Surprise! There’s been a change of plans. The last time I’d posted about Cowen’s monograph on marginalism I figured I had one more (longish) blog post, one about the fourth and final chapter, “Why Marginalism Will Dwindle, and What Will Replace It?” But the more I thought about it, the longer and more convoluted it got. So I’ve decided to simplify things by writing three posts, each substantial, but focused, instead of a long rambling affair like the one I did on biology. So, I ‘m writing one post about large language models (this post), which Cowen brings up at the end of the chapter. Then I’m writing one about high dimensional models in economics, which Cowen introduces early in the chapter. My final post will be a general response to Cowen’s ideas about where this is all headed.
In this post I want to do three things: 1) First I’ll talk about the surprise nature of the success achieved by GPT-3 and then ChatGPT. 2) Then I will present three passages from Cowen’s text and comment on them. 3) Finally, I want to give a brief rundown of tradition of statistical work that stands behind LLMs.
Surprise!
OpenAI released GPT-3 in 2020 to a limited audience of insiders, who recognized that it represented a breakthrough. This level of performance came as a surprise. No one predicted it. GPT-3 was scaled up from GPT-2, which was in turn scaled up from GPT-1, but no one was making explicit predictions about the level of performance to be achieved at each step. These were experiments: “Let’s try it and see what happens.” That’s fine. That’s a good way to make progress, to try things out and see what happens. But don’t mistake a lucky trial for genuine knowledge.
Cowen mentioned GPT-3 on Marginal Revolution on July 19, and then published a Bloomberg column on it on July 21, which he excerpted in Marginal Revolution the next day: “...think of GPT-3 as giving computers a facility with words that they have had with numbers for a long time, and with images since about 2012.” I published a working paper in August, GPT-3: Waterloo or Rubicon? Here be Dragons, in which I both acknowledged about the breakthrough and cautioned about becoming too satisfied with the technology that occasioned the breakthrough.
Two and a half years later, in November of 2022, OpenAI released ChatGPT to the general public. It spread like wildfire. Now the proverbial everyone witnessed what only a small group had witnessed in the summer of 2020. The machine speaks. Sorta’. But more convincingly than any machine had spoken before and in a way that had unimaginable implications for the future.
A threshold HAS been crossed, but it is not, so far as I can see, a threshold in our understanding, either of AI or anything else. It is a threshold in performance along a continuous line of scientific understanding and engineering design and construction, something I have documented in some detail in a recent working paper, The Origins of LLMs. As far as I can tell, there has been no paradigm shift, in Thomas Kuhn’s sense, no rank shift, in terms of cognitive rank theory. There were no fundamentally new ideas in the world by, say, late July of 2020 as a consequence consolidating GPT-3 and making it available in limited release.
“What about the scaling hypothesis,” you might ask. “Isn’t that new?” Ilya Sutskever first explored the idea in 2014. Rich Sutton’s famous 2019 essay, The Bitter Lesson, generated broad discussion about the issue. Then OpenAI published a paper in 2020 that cemented matters, “Scaling Laws for Neural Language Models.”
Given the nature of computing, scaling up is not trivial. Hundreds if not thousands of technical details need to be worked out as the size of the training corpus increases by factors of 10 or more, time after time, and as more and more GPUs are ganged together to assemble the computing power needed. The scaling hypothesis gave researchers a reason to expect improved performance with scaling, but without having to make fundamental breakthroughs in understanding, not of machine learning, artificial neural nets, and certainly not about language and cognition. Consequently our sense of possibility has expanded enormously. Our knowledge and deep understanding have remained the same and the scaling hypothesis made it easy to believe that that was just fine.
Passages from Cowen’s Text
Unfortunately Cowen seems to have bought this story. Not only that, but he doesn’t even acknowledge that there is considerable current debate about whether or not LLMs will be sufficient to achieve AGI (artificial general intelligence) when they are scaled up enough. The most visible opponent of this idea is Gary Marcus, a student of Steven Pinker, who argues that we need to incorporate insights and technology from “old school” symbolic computing (sometimes known as GOFAI, good old-fashioned AI). Marcus is certainly not alone, there are many others. But I don’t want to reprise that debate. I just want to mention that it exists and that Cowen completely ignores it.
What I would like to do in this section is quote some passages from his text and comment on them.
The Marginal Revolution: Rise and Decline, and the Pending AI Revolution, pp. 106-107:
Suffice to say, LLM construction has for the most part ignored linguists and philosophers, and that also means ignoring their intuitions. LLM construction also ignored a lot of people in the AI field who insisted neural nets were a dead end. Instead, in a relatively short number of years humans invented new ways of modeling language and reasoning through language. That research program has proven wildly successful, as we have much better models of language and reasoning than almost anyone had been expecting.
That first sentence is true, sorta’. It is also misleading. As I have documented in that working paper, The Origins of LLMs, this technology is based on a continuous line of statistical thinking that extends back to the 1950s (I take a brief look at this in the next section) . It is the syntacticians, semanticists, and cognitive scientists who been ignored. The second sentence is a bit of an exaggeration. AlexNet put neural nets firmly back on the agenda in 2012.
The big problem is Cowen’s use of “model” in the last two sentences. Large language models are not causal models like those economists use. They don’t tell us anything about how language and thought work. They are algorithmic models. They are about turning input into output; just how that is done is a mystery. Until we understand the internal operations of LLMs they tell us almost nothing about language and reasoning. They give a boost to the idea that some kind of statistical process is involved, but that’s it.
This situation is deeply paradoxical. These algorithmic models perform much better than the computer models created during the “classical” era of cognitive science, the 1960s and 1970s, models that were based on linguistic theory. We knew how those models worked. We don’t know how these models work. We have purchased performance at the cost of ignorance – a formulation I have from the late Martin Kay.
New working paper. Title above, abstract, contents, and introduction below:
Academia.edu: https://www.academia.edu/166054951/On_Method_Computational_Compressibility_in_Complex_Natural_and_Cultural_Phenomena
ResearchGate: https://www.researchgate.net/publication/404263330_On_Method_Computational_Compressibility_in_Complex_Natural_and_Cultural_Phenomena
Abstract: Various machine learning techniques have been used to develop models of complex systems from empirical data. Through discussions with Claude, this paper examines several examples, including: weather, protein folding, chess, language, asset pricing, ticket sales for movies, the 19th century English-language novel. These models differ from one another in various ways, but all are fundamentally descriptive in character. Explanations must necessarily reside with their respective disciplines. In some cases we already have fundamental accounts of the phenomena, while in other cases we do not. With respect to economics in particular, it is clear that such models reveal phenomena for which no explanations are currently available, presenting a challenge to economic theory.
Contents
Part I: Computational Compressibility, Implications for Economics, Description and Explanation 5
Part II: Weather, Protein Folding, Chess, and Language 16
Part III: Interim Summary: Compressibility Without Reducibility 26
Part IV: Pricing Theory, Movies, 19th Century Novels, and Cultural Evolution 28
Part V: To Infinity and Beyond! – Hollywood Redux, Blockbusters, the Spreadsheet, Economics Going Forward 38
Introduction: Describing Computationally Compressible Systems
This a transcription of a dialog I had with Claude 4.6 and 4.7 on April 21 - 23, 2026. While I started it with a specific case from Chapter 4 of Tyler Cowen’s recent monograph on marginalism ( The Marginal Revolution: Rise and Decline, and the Pending AI Revolution), now that the dialog has concluded with Chomsky’s distinction between descriptive and explanatory adequacy (Aspects of the Theory of Syntax), I realize that I’ve been thinking about the underlying issues for some time. While I read Aspects in about 1970, give or take a year, I didn’t think much about description as such until the 2000s, and then I was thinking about describing individual texts; but that’s not directly relevant to these cases in this paper. Then in the second decade of this century I began thinking about computational criticism, aka digital humanities, which typically involve some kind of statistical or machine learning investigation of a corpus of texts. In particular, I gave a great deal of attention to Macroanalysis (2013), where Matthew Jockers studied a corpus of roughly 3000 English-language novels published in the 19th century. That investigation culminated in a directed graph showing depicting relationships of close-similarity among the novels in the corpus. I decided that that graph, in effect, was fundamentally descriptive in character, depicting, in effect, the 19th century Anglophone Geist, or Spirit.
But Jockers’s graph wasn’t on my mind when I started my dialog with Claude. Rather, I was thinking about the distinction between computationally reducible and irreducible phenomena that Stephen Wolfram had introduced in his New Kind of Science (2001). As Claude notes in its summary of the dialog, “a reducible system admits shortcuts through its dynamics; an irreducible one must be simulated step by step.” My target was a paper about asset pricing that Cowen discussed in his monograph, which produced a model having 360,000 parameters but which defied intuitive understanding.
The weather is a canonical example of phenomenon that is computationally irreducible. Thus forecasting the weather generally involves running a simulation of the weather and stepping through it interval after interval. This requires enormous computing resources and takes time. But DeepMind has created a machine learning system, GraphCast, that abstracts over historical data in a way that allows more accurate forecasts with less compute. Thus the weather system is computationally irreducible, but it is also compressible.
I take that as my paradigm case of computational compressibility (pp. 16 ff.) and then move on to other examples: protein folding (another physical phenomenon, pp. 18 ff.), chess (human activity, pp. 22 ff.), and natural language (a different human activity, pp. 23 ff.), each of which is compressible using machine learning techniques. Each example sharpens and extends the idea of computational compressibility. At that point I asked Claude to summarize the discussion (pp. 26 ff)..
Then, and only then do I ask Claude to consider Cowen’s problematic example, AI Pricing Theory (pp. 28 ff.). In its analysis of the paper, Claude notice that it introduces something fundamentally new to the discussion, reflexivity. Asset pricing is done by a large group of actors over time who thus influence one another’s decisions. And that, in turn, brings up Arthur De Vany’s work on Hollywood Economics (pp. 30 ff.). De Vany discovered that box-office success cannot be predicted by such analytic variables as producer, screen writer, director, movie stars, or opening weekend box office. Rather the success of a film depends on a word-of-mouth cascade which cannot be predicted. That leads me, in turn, to suggest a thought experiment involve a hypothetical system capable to abstracting over entire films and developing a high-dimensional model which could be used to predict the success of individual films.
And that, in turn, led me to the work that Matthew Jockers had done on 19th century English-language novels (pp. 33 ff.), something that had not been on my mind when I began this dialog on April 21. Jockers used machine learning, albeit nothing so elaborate and computationally expensive as using a transformer to create an LLM – it only had roughly 600 parameters. What his model revealed, and what made it so fascinating to me, is that there is an inherent directionality to the production of novels over the course of a century. It’s not simply that later novels are systematically different from earlier ones, but that that difference has a direction in the 600-dimensional measurement space. What we’d really like to know, now, is a say to characterize that diction. The model shows us that there is a direction, but it doesn’t tell us what that direction is. Though the model is much simpler than that asset pricing model – it has three orders of magnitude fewer parameters – its significance is no more legible.
After that I have two discussions that are not based on existing models, but that do have implications for economists who want to study them. First, I consider the phenomenon of the blockbuster, arguing that it reveals audience preferences that had previously been unrecognized (pp. 41 ff.). Then I consider the spreadsheet (e.g. VisiCalc), which transformed the personal computer market from a small niche market into a large mainstream market (pp. 43 ff.). How do you create a model that allows you to predict markets that don’t even exist at the time you make your model? What kind of a problem is that? After that I took a brief look at Cowen’s argument in The Great Stagnation (pp. 45 ff.), where Claude remarked:
If the VisiCalc model is right, then what matters about ChatGPT and its successors is not primarily that they do existing things faster or cheaper—though they do—but whether they are constitutive technologies in the VisiCalc sense. Do they reorganize the space of possible wants, making new activities imaginable and practical that previously had no well-formed representation in anyone's preference space? With that I brought the exploration to a halt.
I then asked Claude to summarize the entire dialog, which I’ve placed immediately following these remarks (pp. XX ff), with a special emphasis on implications for economics (pp. 7 ff.). Then I introduce Chomsky’s distinction from the 1960s, description vs. explanation (pp. 9 ff.). Each of these cases involves a complex phenomenon that is irreducible, but can be compressed into a model that is descriptive in character. They have that in common. As for explanations, those must necessarily be specific to each phenomenon. Note that in some cases we have explanatory theories grounded in a fundamental understanding of the underlying system (weather, protein folding) while in others we do not (chess, asset pricing, cultural evolution).
Finally, I’ve added a coda from a different conversation with Claude (pp. 13 ff.), one I had with the AI that accompanied Cowen’s book. That conversation is about Hollywood Economics and Rational Ritual and argues that the factoring of intellectual space that we’ve inherited from the 19th century German university has outgrown its usefulness.
Note: Claude 4.5 drafted this article after a long series of dialogs over several days. This is a continuation of the thinking in my current article in 3 Quarks Daily, Chess and Language as Paradigmatic Cases for Artificial Intelligence.
See the new coda, from April 24, 2026
In the 1950s, artificial intelligence emerged from a productive confusion. We had just formalized computation itself—Turing and von Neumann had given us the fundamental principles of what computers could do. When we turned these powerful new machines toward intelligence, we naturally assumed the principles would be the same.
They aren't.
Computation vs. Intelligence
The principles of computation are domain-independent. A universal Turing machine can compute anything computable, whether that's arithmetic, chess moves, or protein folding. The Church-Turing thesis tells us that all models of computation are equivalent in what they can ultimately compute, given unlimited time and memory.
This universality is computation's glory—and intelligence's red herring.
Intelligence, as it actually exists in nature, operates under entirely different constraints. It must function in the physical world, with finite resources, solving problems that often don't have clean formal specifications. These aren't just practical limitations to be worked around; they're constitutive features that shape what intelligence is and how it must work.
Principle 1: Geometric Complexity Determines Computational Regime
The critical variable isn't how hard a problem is in some abstract computational sense, but the geometric complexity of the domain.
Consider chess versus visual object recognition. Chess is played on an 8×8 grid with a small set of piece types following rigid rules. The game tree is astronomically large—around 10^120 possible games—but it's finite and well-defined. You can represent board positions symbolically, enumerate legal moves, and search through possibilities systematically.
Vision operates in continuous three-dimensional space with effectively unbounded variation. Objects appear at different scales, orientations, and lighting conditions. There's no finite set of "legal configurations." You can't enumerate all possible images the way you can enumerate chess positions.
This difference in geometric complexity demands different computational approaches. Chess yields to systematic search through a definable space—what we might call sequential or symbolic processing. Vision requires something else: massively parallel processing that can handle continuous variation and incomplete information—holographic or neural processing.
In 1975, Miriam Yevick demonstrated this formally: the geometric complexity of objects in a domain determines the computational regime needed to identify them. Simple geometric objects can be handled by sequential symbolic systems. Complex geometric objects require holographic processing. This wasn't mere speculation—she made a formal mathematical argument about pattern recognition systems.
The field ignored her insight. We assumed all problems were fundamentally like chess—just harder. If symbolic AI could master chess, we thought, it would eventually master vision, language, and physical reasoning through better algorithms and more compute.
We were wrong. Vision didn't yield to symbolic AI no matter how much compute we threw at it. It required a regime shift to neural networks—systems whose architecture matches the geometric complexity of the visual world.
Principle 2: Intelligence Operates in Unbounded, Geometrically Complex Reality
Here's what makes intelligence different from computation in the abstract: intelligence evolved to work in the physical world, which is geometrically complex and open-ended. There's no finite game tree for "objects I might encounter" or "situations I might face."
This has profound implications. You can solve chess by exploring its game tree faster than humans can. But you can't solve vision or language understanding the same way because there's no complete tree to explore. The space isn't closed and enumerable—it's unbounded.
This is why Deep Blue beating Kasparov in 1997 didn't generalize the way we thought it would. Chess was solved by a room-sized supercomputer with custom hardware doing exactly what computers do best: blindingly fast systematic search. By 2025, a smartphone runs chess engines that would destroy both Deep Blue and Kasparov.
But that same smartphone can't run a GPT-4 level language model. Language still requires massive data centers. Why? Because language connects to the unbounded complexity of physical and social reality. No amount of faster chess-style search bridges that gap.
The field learned to beat humans at chess by doing what computers naturally excel at. Then we mistook this for a general template. We thought: "Intelligence is search through problem spaces. We just need bigger computers to search bigger spaces." But geometric complexity isn't about bigger—it's about different.
Principle 3: Embodiment as Formal Constraint
Embodiment isn't a philosophical talking point. It's a formal constraint on intelligence architecture.
When we say intelligence must be embodied, we mean: it must operate with finite computational resources in a geometrically complex physical world. This changes everything.
Abstract computation doesn't care about efficiency—a proof is valid whether it takes a second or a century. Physical computation must complete before the hardware fails. But biological intelligence faces a sharper constraint: it must acquire the energy it uses to compute. A deer's visual system can't require more calories than the deer can acquire. The computation must pay for itself.
This constraint shapes what kinds of solutions are viable. You can't exhaustively search unbounded spaces. You can't maintain perfect world models. You must make do with approximate, good-enough processing that operates in real time with available resources.
Crucially, this means different problems need different solutions—not just more or less compute, but fundamentally different architectures matched to the geometric complexity of the domain.
Why This Matters Now
Current AI has powerful neural networks that excel at pattern recognition in geometrically complex domains—vision, speech, even aspects of language. But the field still carries assumptions from the symbolic AI era:
These assumptions persist even though we've abandoned symbolic AI. We've swapped the implementation (symbols → neural networks) but kept the framework (more compute → general intelligence).
This is why we need to distinguish computation principles from intelligence principles. Turing and von Neumann gave us the former. For the latter, we need to recognize that geometric complexity, unbounded reality, and embodied constraints aren't bugs to be worked around—they're the constitutive features that determine what intelligence is and how it must work.
The principles of intelligence aren't the principles of computation. Understanding this distinction is the key to understanding both what current AI can do and what it cannot.
Coda: Rank 5 Cognition
Given that I’ve decided that Yevick’s 1975 paper is a convenient marker for Rank 5 cognition, it seems to follow that intelligence, in the sense discussed here, is a Rank 5 concept. So, the ranks shape up like this:
Rank 1: speech
Rank 2: writing
Rank 3: calculation
Rank 4: computation (flow of control)
Rank 5: intelligence (regime matching: computation in unbounded, geometrically complex, reality)
Late in his fourth chapter, Cowen introduces chess as an example of what contemporary AI can do. As you may know, chess has been a central interest of AI, so much so that John McCarthy, the mathematician who coined the term “artificial intelligence,” has written an article entitled, “Chess as the Drosophila of AI.” You may also know that chess has been a central interest of Cowen’s. He was a chess champion in his youth and he follows the game closely.
Recognition of the chess tree comes late
As it happens, chess presents us with one of those examples that Tyler finds so interesting in this book (The Marginal Revolution: Rise and Decline, and the Pending AI Revolution). It was during my freshman year at Johns Hopkins, I believe, that I read some Dover Publications book that was an omnibus presentation of information, cybernetics, and computing. I forget both the author and the exact title, but I remember two things. 1) In its presentation of computing, it talks about analog computing and digital computing. That was common until not long after personal computers arrived; after that, articles and books about “computers for dummies” stopped talking about analog computing and concentrated on digital (big mistake, IMO, but that’s a different story). 2) It talked about chess and made the point that, from an abstract point of view, chess was just like tic-tac-toe, an utterly trivial game. Both games are finite and the games can be arranged in a tree structure. But the chess tree is so large that not even the largest computer can list them all. I thought about it a little, got the point, and that has stuck with me ever since.
That was back in the mid-1960s. The largest computers at that time are tiny in comparison to the Brobdingnagian behemoths being used to train contemporary AIs, but they’re still too small for the chess tree.
The thing is, and here we’re getting to the Tylerian point, chess has been played by thoughtful people for centuries. Why was it only in the early 20th century that its tree-like structure was recognized? That’s so simple, and so basic.
So I put the question to Claude, who answered. And then we went on from there. I’ll present that whole dialog shortly, but I want to discuss something that I discovered while thinking about its answer.
Game theory (reading the historical record backwards)
First, I already knew more or less how it would answer, I was asking the question to get details. Ernst Zermelo explicated the tree structure in a paper published in 1913. The Wikipedia entry, “Zermelo’s theorem (game theory),” opens like this:
In game theory, Zermelo's theorem is a theorem about finite two-person games of perfect information in which the players move alternately and in which chance does not affect the decision making process. It says that if the game cannot end in a draw, then one of the two players must have a winning strategy (i.e. can force a win).
That clearly says that Zermelo’s theorem belongs to that branch of investigation known as game theory. But, thought I to myself, wasn’t game theory invented by John von Neumann in the middle of the 20th century? So I did an Ngram search on “game theory”:
The chart doesn’t lie. “Game theory” shows up in the middle of the century, not the beginning. So a did a bit more digging and found an article, “Zermelo and the Early History of Game Theory,” that begins with this sentence: “It is generally agreed that the first formal theorem in the theory of games was proved by E. Zermelo in an article on Chess appearing in German in 1913 (Zermelo (1913)).” So, Zermelo’s theorem was retrospectively assimilated into game theory once game theory had become a recognized field of inquiry. Culture works like that.
The same thing has happened to the diamonds-water paradox that Tyler presents early in the first chapter as THE paradigmatic example of marginalist reasoning (page 4 in the PDF). Tyler presents part of that story in the book. I dug out more of the story by querying the accompanying AI. I won’t go into the details, but it boils down to this:
And that textbook went on to become one of the most important introductory texts in economics.
And so it goes.
Note: At times Claude refers to my "Play book." That's the book I've been working on, and with ChatGPT as well: Play: How to Stay Human in the AI Revolution.
* * * * *
I now give you my dialog with Claude. Zermelo’s theorem is the start. From there:
I may return to those last three topics in a later post.
* * * * *
What are the prerequisites for Zermelo’s theorem?
I’ve got a question about chess, one that’s related to Chapter 3 of Tyler Cowen’s book, though he doesn’t discuss chess there. But he is concerned about why economics seems to come so late in intellectual history. So, why, given how long chess has been played, was it only in the early 20th century that the tree-structure of the game was recognized? On the one hand, since the chess tree is so large, it would be impossible to list the whole thing and determine its finite nature that way. But, abstractly considered, one might imagine getting a hint in that direction by trying to list the games. White has 20 possible moves to start (even if most of them are never played), and Black has 20 possible replies to each of those. At this point the chess tree is 400 branches wide, and it rapidly gets much worse. On second thought, no, that won’t work.
You need a method of reasoning. And that didn’t exist until….Well Zermelo’s proof dates from 1913. Just when the relevant math existed, I don’t know, but I doubt it would have been much before then. I’m sensing intuitively that we’re in the same ballpark as Cantor’s diagonal method, which was late 19th century.
Your intuition about the timing is sharp, and the Cantor connection is real — but the deepest reason runs even further into the foundational transformation of mathematics than just the diagonal argument. Let me develop this.
What the proof actually requires
Zermelo's 1913 paper is titled, revealingly, "On an Application of Set Theory (Mengenlehre) to the Theory of Chess Games." The title tells you everything: this is explicitly an application of Cantor's set theory. Zermelo needed three things that didn't exist before the late 19th century:
First, the concept of a complete strategy as a mathematical object — not a heuristic or a rule of thumb but a function that specifies a move for every possible position that could arise. This is a first-class set-theoretic object: a mapping from the (enormous but finite) set of all positions to the set of available moves. You cannot even state Zermelo's theorem without this concept. Before Cantor, "function" meant something like a formula or rule that you computed — the idea of a function as an arbitrary set-theoretic object, definable over an enormous domain without being explicitly written down, was simply not available.
Second, backward induction as a proof method on a finite tree of unbounded but determinate depth. You need to show that the set of positions from which White can force a win is well-defined, even though it's a subset of an incomprehensibly large structure. The argument works by showing that this set is recursively characterizable from the terminal positions backward — but "recursively characterizable over a structure you cannot enumerate" requires treating the whole tree as a completed mathematical object, not an unfolding process.
Third, Zermelo's own axiomatization of set theory (1908) — which came five years before the chess paper. The Axiom of Choice, in particular, is needed in the general case (for games of possibly unbounded length before the 50-move rule imposes finitude), and Zermelo had just invented it.
The Cantor connection is precise
You're right that it's the same ballpark, but it's more than approximate temporal proximity. The specific intellectual move Cantor pioneered that made Zermelo possible was the treatment of infinite (or astronomically large) structures as completed wholes — actual infinities, not potential infinities. Before Cantor, the dominant view (even among mathematicians, following Aristotle) was that infinity was always a process, never an object. You could always add one more, but you couldn't speak of "the set of all natural numbers" as a single thing you could reason about.
The chess tree is finite, but it's so large (roughly 10¹²⁰ possible games) that the conceptual move required is structurally identical to Cantor's: you cannot approach it by enumeration or by any incremental process. You have to treat the entire tree as a completed object and reason about its global properties by abstract methods. Cantor's diagonal argument is the purest example of this — you prove something about an uncountable structure by an argument that never touches most of its elements. Zermelo's chess proof does the same thing: it proves that a particular position is a forced win for White (or a draw) without ever specifying which moves constitute the winning strategy.
The Rank 4 structure
The cognitive rank analysis maps cleanly onto this. Rank 3 chess thinking is exactly what the great 19th-century practitioners developed: Morphy's tactical superiority, Steinitz's positional principles, opening theory, endgame theory. These are all algorithmic in the relevant sense — they are rules, patterns, procedures for navigating the game. This is sophisticated, even brilliant, but it's thinking with chess moves and positions as primitives.
The Rank 4 move is to ask: what kind of object is a chess game? Answer: a finite tree with two alternating agents, terminal nodes with assigned outcomes, and the question of whether a particular class of strategies (those guaranteeing a given outcome regardless of opponent play) is non-empty. You've taken the activity of playing chess — the Rank 3 algorithms and heuristics — and made the activity itself an object of mathematical modeling. That's precisely your formulation: the operations of the lower level become the objects of the next.
And notice what this means for the specific threshold you identified. The reason even imagining that the game is finite and tree-structured doesn't help you until you have the right tools is that the question you need to ask — "is there a complete strategy that guarantees outcome X?" — cannot be formulated without the concept of a complete strategy as a set-theoretic object. Knowing the tree is there doesn't help if you have no language for the thing you're trying to prove exists.
Cowen's theme
This fits Cowen's chapter 3 argument exactly. His diagnosis for why economics was so slow is that economic ideas require peering around a corner — you can't reach them by brute force, and you don't know where to look until the relevant conceptual apparatus exists. The chess case is if anything cleaner, because here the brute force failure is quantifiably obvious: 10¹²⁰ is not a number you can work through. The delay isn't from lack of effort or lack of institutional support or Christian resistance — the game had been played intensely for a thousand years by people who were clearly thinking hard about it. The delay was purely conceptual: the question couldn't even be asked in a mathematically tractable form until Cantor, Zermelo, and the late-19th-century foundations revolution made the requisite objects available. That's as clean an example as one could want of Cowen's "factoring a very large number" metaphor — once you have the factors (set theory, the function concept, backward induction), the result is almost immediate.
The 1913 date isn't merely contemporaneous with the Cantor-era mathematics — Zermelo himself was one of the creators of that mathematics, applying his own toolkit almost immediately to a new domain.
My examination of Tyler Cowen, The Marginal Revolution: Rise and Decline, and the Pending AI Revolution (2026), continues. While Tyler does discuss chess briefly at the end and in connection with artificial intelligence, this post has nothing to do with that discussion. That, I suppose, makes this post something of a digression. But it’s a digression in the style of Tristram Shandy, digress to progress. It’ll all make sense in the end.
Note that what I am reporting here comes at the end of a rather long discussion of chess, one that does have a direct connection of Tyler’s chapter three, where he’s interested in the fact that pinnacles of human achievement seemed to be reached at different times for different domains, some rather early, some quite late. While chess is rather old, traceable to the 6th century CE in India, the treelike structure of the game wasn’t explicated until recently, 1913 by Ernst Zermelo. I asked Claude about the nature of the mathematical equipment Zermelo needed to construct his proof. That led to a long discussion about the nature of the chess tree, chess as a vehicle for the expression of personality, how computers open up exploration of new regions of the chess tree and so forth. That whole discussion is part of the context for the discussion that follows. However, this discussion is pretty much independent of that one, so you don’t need that context to understand what Claude says here.
Note: I’m using my own account on Claude for this discussion, not the AI that Tyler made available with the book.
* * * * *
Chess as an abstraction over military conflict
Let’s take this discussion of chess in a different direction. I’m interested in the origins of the game. The game, after all, is a highly abstracted version of a war between two societies, each having its own king and queen. I could imagine a just-so story, perhaps a Star Trek episode, where we have two nations that are locked in a deadly conflict that continues year after year, decade after decade, until, their resources exhausted, the two nations call a truce and in the space of that truce, invent the game of chess. Then each nation chooses its champion, and the champions continue the conflict by playing chess. [When you consider the role that chess played in the Cold War between the USA and the USSR that origin story is all but a highly abstracted version of that historical conflict.] But what do we actually know about how the game originated?
Your just-so story is charming, and the Cold War parallel is genuinely apt — but the actual history is both more interesting and more complex, and it cuts against one specific element of your narrative in a revealing way.
What we actually know
The earliest precursor of modern chess is chaturanga, which flourished in India by the 6th century CE, and is the earliest known game to have two essential features found in all later chess variations — different pieces having different powers, and victory depending on the fate of one piece, the king.
The name itself is telling: chaturanga is a Sanskrit word meaning "four divisions of the military" — infantry, cavalry, elephants, and chariots — which map onto the pieces we now recognize as pawns, knights, bishops, and rooks. So the military origin of the game's structure is not speculative — it's explicit and preserved in the name.
The name came from a battle formation mentioned in the Indian epic Mahabharata. The game chaturanga was a battle-simulation game that rendered Indian military strategy of the time.
From India the game traveled westward: around the 7th century CE, chaturanga reached Persia and became known as "chatrang" or "shatranj." Many modern chess terms have Persian origins, such as "Shah" (King), and even "checkmate" from "Shah Mat," meaning "the king is dead" or "the king is helpless."
A form of chaturanga or shatranj made its way to Europe by way of Persia, the Byzantine Empire, and the expanding Arabian empire. Muslims brought chess to North Africa, Sicily, and Spain by the 10th century.
The rules then changed dramatically in late medieval Europe: the chaturanga piece called the elephant, which had been limited to a two-square diagonal jump in shatranj, became the bishop, more than doubling its range. With the new queen and bishop powers, the trench warfare of medieval chess was replaced by a game in which checkmate could be delivered in as few as two moves.
Where your just-so story cuts against the evidence
Your narrative imagines two symmetric opposing societies, each with a king and queen — the structure of modern chess with its two mirror-image armies. But chaturanga wasn't originally a two-player game in that sense, or at least not necessarily. The Cox-Forbes theory asserted that the four-handed game chaturaji was the original form of chaturanga — four players, four armies, a dice element. The clean two-army symmetry of modern chess is a later development, not the original structure.
And the queen — the piece that makes modern chess feel like a conflict between two courts — is a late European invention. The original piece in that position was the mantri (minister) or counselor, a weak piece that could only move one square diagonally. The vizier piece, which had limited movement in earlier versions, became the "queen" and gained much greater power only in the late 15th century in Europe. The transformation of a male counselor into a powerful queen is itself a fascinating cultural moment — it reflects something about late medieval European court culture, the figure of powerful queens like Isabella of Castile, that has nothing to do with the Indian military original.
The deeper point about abstraction
What's remarkable in the actual history is the direction of abstraction. The game began as an explicit military simulation — every piece named for a branch of the army, the board a battlefield, the objective the capture of the king. Over fifteen centuries of transmission across cultures — India to Persia to the Arab world to Europe — it progressively shed its military literalism and became more abstract. The elephant became a bishop (losing its literal referent entirely in the European context). The chariot became a rook (the Persian rukh, itself possibly meaning chariot or a mythical bird). The military counselor became a queen.
Each culture reshaped the pieces to reflect its own social imagery, but the underlying structure — the tree, the rules, the objective — remained surprisingly stable across this enormous cultural distance. The abstraction was always already present in the game's formal properties; what changed was the surface representation hung on that structure.
This is itself a nice illustration of the Rank 2 point from our earlier discussion. The game's transmissibility across such different cultures depended precisely on its formal specification being separable from any particular cultural content. You could strip away the Indian military imagery, replace it with Persian court imagery, then European feudal imagery, and the game remained the same game because the rules — the Rank 2 specification — were what was being transmitted. The cultural clothing was exchangeable; the formal skeleton was not.
The Cold War resonance
Your Cold War parallel is historically well-grounded, and not just as analogy. The 1972 Fischer-Spassky match in Reykjavik was explicitly framed by both sides as proxy warfare — American individualism versus Soviet collective training, capitalism versus communism, with the world watching. Fischer's behavior, the diplomatic incidents, the American government's last-minute intervention to keep Fischer in the match — it was understood by everyone involved as something more than a chess tournament. The game had come full circle: from explicit military simulation, through centuries of abstraction, back to serving as a ritualized substitute for geopolitical conflict. Your Star Trek episode would merely be making that latent structure manifest.
DeepBlue vs. Kasparov
And so the post-Cold War victory of Deep Blue over Kasparov becomes what? What’s the Straussian reading, as Tyler might say?
The surface reading is triumphalist: human chess genius defeated by American corporate technology, confirming the post-Cold War narrative of Western technological capitalism's dominance. IBM's stock rose. The press celebrated. It was framed as a milestone for artificial intelligence.
The Straussian reading is almost the opposite.
New working paper. Title above, links, abstract, contents and introduction below:
Academia.edu: https://www.academia.edu/164885566/Computation_Chess_and_Language_in_Artificial_Intelligence
SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6319062
ResearchGate: https://www.researchgate.net/publication/401355671_Computation_Chess_and_Language_in_Artificial_Intelligence
Abstract: This paper reexamines the foundations of artificial intelligence by contrasting chess and natural language as paradigmatic domains. Chess, long treated as a benchmark for intelligence, is finite, rule-governed, and geometrically well-defined. It lends itself naturally to symbolic search and evaluation. Natural language, by contrast, operates in an unbounded and geometrically complex reality. Its rules are open-ended, its objectives diffuse, and its domain inseparable from embodied experience. With chess as its premier case – McCarthy: “the Drosophila of AI,” – AI has been guided by a deeper assumption: that the first principles of intelligence reduce to the first principles of computation. Drawing on Miriam Yevick’s distinction between symbolic and neural computational regimes, I propose that intelligence must be understood as operating in a geometrically complex world under finite resource constraints. Embodiment is therefore a formal condition of intelligence, not an incidental feature. Recognizing the structural difference between bounded games and open-ended cognition clarifies both the historical trajectory of AI and the conceptual limits of current systems.
Contents
Introduction: Chess, Language, and Intelligence 3
Chess and Language as Paradigmatic Cases for Artificial Intelligence 5
Three Principles of Intelligence (That Aren't Principles of Computation) 12
Chronology of Chess, Language, and AI 15
Introduction: Chess, Language, and Intelligence
Chess has been a central concern of AI from the beginning. AI researchers didn’t become interested in natural language until the 1970s. Before that computational research on natural language was the domain of computational linguistics (CL), which started with machine translation (of texts from one natural language to another) as its primary problem. Thus we have two different disciplines, AI and CL.
In a sense, AI was fundamentally a philosophical exercise. It was an attempt to demonstrate, in effect, that we could understand the human mind in terms of computation. But rather than advance its philosophical objective through argument, it chose computational demonstration as its mode of expression. Chess became a central concern for two reasons: 1) On the one hand it was widely regarded as exhibiting the pinnacle of human reasoning ability. If we could create a computer program to play a championship game of chess, we could create a computer program that would be capable of cognitive or even perceptual task humans can do. 2) But also, the nature of chess made it well-suited for computational investigation.
The article that opens this working paper – Chess and Language as Paradigmatic Cases for Artificial Intelligence – concentrates on this and then goes on to make the point that language is utterly unlike chess in this respect. The chess domain is bounded and well-defined. Natural language is not; it is ill-defined and unbounded.
That’s as far as I got in the article, but I had been aiming for an argument that AI is still, in effect, mesmerized by the chess paradigm. I didn’t make it that far because language is so obviously different from chess that it is difficult to see how anyone would made that mistake.
What I have come to realize, only after I’d finished the article, is that it isn’t so much chess that has mesmerized AI. Rather it is computation itself. AI has been implicitly assuming that the First Principles of intelligence reduce to the First Principles of Computing. The first principles of computing can be found in the work of Alan Turing (the abstract idea of computing) and and others.
The first principles of intelligence are more stringent. As Claude put it a recent dialog:
First principle of intelligence: Must operate in unbounded, geometrically complex physical reality with finite resources.
Those two qualifications, an unbounded, geometrically complex reality, and finite computational resources, change the nature of the problem considerably. I note, in passing, that this allows us to assign formal significance to the concept of embodiment, for it is embodiment that commits intelligence to operating with finite resources in a geometrically complex universe.
Miriam Yevick’s 1975 paper, “Holographic or Fourier Logic,” is the crucial document, but it’s been forgotten. Using identification in the visual domain as her case, she showed that, where we are dealing with geometrically simple objects, sequential symbolic processing is the most efficient computational regime. But when we are dealing with geometrically complex objects, neural net processing is the most efficient computational regime. AI started out with symbolic processing in the 1950s and arrived at neural nets in the 2010s. But it hasn’t explicitly recognized that one must fit the mode of processing to the nature of the world. In that (perhaps a bit peculiar) sense, the researchers in the currently-dominant paradigm don’t know what they’re doing.
The second article in this working paper, Three Principles of Intelligence (That Aren't Principles of Computation), discusses this in more detail. I had it generated by Claude 4.5 after a long series of dialogs over several days.
The last article is a chronology of events in the history of chess and language in AI.
I've posted a new working paper. Title above, links, abstract, contents, and introduction below.
Academia: https://www.academia.edu/129111476/Dialog_with_Claude_3_5_on_the_Intellectual_Potential_of_Man_Machine_Interaction_A_Working_Paper_April_30_2025
SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5237019
ResearchGate: https://www.researchgate.net/publication/391320022_Dialog_with_Claude_35_on_the_Intellectual_Potential_of_Man-Machine_Interaction
Abstract: This paper takes the form of a discussion between me and Claude 3.5. We began by discussing how neural-network based chess programs like AlphaZero play differently from humans, and whether humans can learn from these new approaches. This leds to a key distinction between computational limitations (humans simply can't calculate as deeply as computers) and conceptual barriers that might be overcome through developing new theoretical frameworks. We explored how knowledge develops over historical time - not through changes in human biological capacity, but through the development of new conceptual frameworks and cultural tools. This was illustrated through examples like A Connecticut Yankee in King Arthur's Court (showing the vast gulf between different historical periods' knowledge) and cargo cults (showing the difference between mimicking surface behaviors and understanding underlying principles). The discussion then moved to how chess and language represent fundamentally different kinds of problems. Chess has a simple geometric footprint (8x8 grid, 6-piece types) and finite though vast possibility space. Language, in contrast, must interface with the entire world of human experience and lacks such clear boundaries. This connected to the historical development of AI - chess yielded to symbolic AI approaches (Deep Blue) while language required statistical/neural approaches (modern LLMs). This reflects fundamental differences in the problems - chess being rule-based and well-defined, language being messy and contextual. Similar patterns appeared in computer vision, where early symbolic approaches struggled with the fractal-like complexity of real-world objects, leading to the success of machine learning approaches.
Note: This abstract is a lightly edited version of a summary of the discussion that I asked Claude 3.5 to create.
Contents
Introduction: “Superintelligence” through man-machine interaction 2
Humans learning to think like AIs in chess 4
Super-intelligence 6
Connecticut Yankee 8
Cargo Cults 9
Chess and Search 9
Language 10
Recursus 13
Introduction: “Superintelligence” through man-machine interaction
I don’t know when the word “superintelligence” was first used, what it was used for, or who coined the term. This Google Ngram plot shows evidence of the term going back to the early 20th century:
But the term doesn’t take hold until the second decade of this millennium. This chart gives us a clearer picture:
The big jump starts at 2014, the year Nick Bostrom published his book, Superintelligence: Paths, Dangers, Strategies, which went on to become New York Times bestseller and to garner praise from Elon Musk, Bill Gates, Peter Singer, Derek Parfit, and Sam Altman.
Here’s how Bostrom defines superintelligence: “We can tentatively define a superintelligence as any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest” (p. 26). That’s it. He treats it as an unstructured thing the provides intellectual power, like gasoline powering an automobile engine. Nowhere does he explain the mechanisms through which superintelligence operates.
By contrast, starting with the 1950s, the cognitive sciences have devoted a great deal of attention to mechanisms. Most of the controversies in linguistics have been about structures and mechanisms. Research in artificial intelligence has classically been a search for specific computational mechanisms. In this context, one might as well talk about Oooomph as talk about superintelligence. They mean pretty much the same thing, and that thing doesn’t seem to have anything to do with mechanisms.
Back in the mid-1980s David Hays and I reviewed a wide range of material in linguistics, cognitive psychology, neuroscience, developmental psychology, and neurobiology and published an article in which we proposed five principles of natural intelligence: 1) modal, 2) diagonalization, 3) action, 4) figural, and 5) indexing. We associate each principle with some mathematics, however informally, and with neural structures. As far as I know, the literature on superintelligence has nothing remotely comparable on offer. Our proposal is surely speculative, but there’s something there. There’s nothing to the concept of superintelligence except more more more.
* * * * *
This paper uses the term “superintelligence” rather lightly. I takes the form of a conversation with Claude 3.5, which I initiate by inquiring about neural-net-based chess engines play the game differently from humans. That is, I am interested in specific behaviors. After discussing chess for a bit we move on to superintelligence. There we develop a distinction “between something that's novel but ultimately comprehensible to humans [...] versus something that's inherently beyond human comprehension” (Claude’s formulation). Pay particular attention to the discussion of search, pp. 9-10, which leads to language, where I talk about the geometric footprint of a problem domain. The geometric footprint of chess is simple and finite while the geometric footprint of language is complex and apparently unbounded.
We’re by no means at the end; there’s a few more pages in the dialog. But that should give you a sense its style. The object is to tease out important distinctions and details, distinctions and details important for thinking about the nature of the underlying mechanisms.
L. Schut, N. Tomašev, T. McGrath, D. Hassabis, U. Paquet, & B. Kim, Bridging the human–AI knowledge gap through concept discovery and transfer in AlphaZero, Proc. Natl. Acad. Sci. U.S.A. 122 (13) e2406675122, https://doi.org/10.1073/pnas.2406675122 (2025).
Significance
As AI systems become more capable, they may internally represent concepts outside the sphere of human knowledge. This work gives an end-to-end example of unearthing machine-unique knowledge in the domain of chess. We obtain machine-unique knowledge from an AI system (AlphaZero) by a method that finds novel yet teachable concepts and show that it can be transferred to human experts (grandmasters). In particular, the new knowledge is learned from internal mathematical representations without a priori knowing what it is or where to start. The produced knowledge from AlphaZero (new chess concepts) is then taught to four grandmasters in a setting where we can quantify their learning, showing that machine-guided discovery and teaching is possible at the highest human level.
Abstract
AI systems have attained superhuman performance across various domains. If the hidden knowledge encoded in these highly capable systems can be leveraged, human knowledge and performance can be advanced. Yet, this internal knowledge is difficult to extract. Due to the vast space of possible internal representations, searching for meaningful new conceptual knowledge can be like finding a needle in a haystack. Here, we introduce a method that extracts new chess concepts from AlphaZero, an AI system that mastered chess via self-play without human supervision. Our method excavates vectors that represent concepts from AlphaZero’s internal representations using convex optimization, and filters the concepts based on teachability (whether the concept is transferable to another AI agent) and novelty (whether the concept contains information not present in human chess games). These steps ensure that the discovered concepts are useful and meaningful. For the resulting set of concepts, prototypes (chess puzzle–solution pairs) are presented to experts for final validation. In a preliminary human study, four top chess grandmasters (all former or current world chess champions) were evaluated on their ability to solve concept prototype positions. All grandmasters showed improvement after the learning phase, suggesting that the concepts are at the frontier of human understanding. Despite the small scale, our result is a proof of concept demonstrating the possibility of leveraging knowledge from a highly capable AI system to advance the frontier of human knowledge; a development that could bear profound implications and shape how we interact with AI systems across many applications.
The standard discussions of AGI – artificial general intelligence (whatever the hell that is) – and so-called superintelligence (???) strike me as being so poorly posed as to be all but irrelevant. I decided to take the matter up with Claude 3.5 Sonnet. As always, my remarks are boldface while Claude's are plain.
Note that we decided to ease into it, from the side.
* * * * *
I believe that neural-net based chess programs, such as AlphaZero, have been observed to play chess differently than humans. Is that so?
Yes, this is a fascinating observation that's been widely discussed in the chess community. Neural network-based chess engines like AlphaZero have demonstrated some distinctive playing patterns that differ from both traditional chess engines and human players.
Some key differences include:
1. Piece sacrifices: AlphaZero has shown a greater willingness to make long-term positional sacrifices, sometimes giving up material for positional advantages that only become clear many moves later. These sacrifices often appear counterintuitive to human players.
2. Prophylactic moves: The program often makes seemingly quiet moves that prevent opponent's plans before they can even develop, showing a deep positional understanding that differs from the more tactical focus of traditional engines.
3. Piece placement: AlphaZero sometimes places pieces in what human players would consider unusual or even awkward squares, but which later prove to be strategically important.
4. Flank attacks: The program has demonstrated a particular affinity for attacking on the flanks and undertaking long-term strategic operations, rather than focusing on immediate tactical opportunities.
Perhaps most interestingly, AlphaZero achieved this playing style without being trained on human games - it learned purely through self-play. This suggests that what we consider "natural" or "human-like" chess might actually be influenced by centuries of human chess tradition and limitations, rather than representing the most effective way to play the game.
Would you be interested in discussing any specific aspects of these playing patterns? I find the implications for our understanding of chess strategy particularly fascinating.
I’m wondering if humans have been able to learn these new techniques.
Yes, the influence of neural network-based chess engines like AlphaZero has begun to reshape human chess at the highest levels. Grandmasters have started incorporating some of these novel approaches into their play, though often with a more selective and cautious adoption.
For example, Magnus Carlsen, widely considered one of the greatest chess players ever, has explicitly acknowledged studying and learning from AlphaZero's games. He's noted how it's influenced his willingness to play certain types of positions that he might have previously considered dubious.
However, there are some important limitations to how much humans can directly copy these approaches:
1. Humans can't calculate as deeply or precisely as neural networks, so we can't always justify the same risky sacrifices that AlphaZero makes.
2. The human mind is better suited to understanding chess through general principles and patterns rather than the concrete move-by-move precision that neural networks excel at.
3. Some of AlphaZero's more counterintuitive moves require an extreme degree of accuracy in the follow-up play that's difficult for humans to maintain.
What we're seeing instead is a kind of selective adaptation - human players are incorporating these insights to expand their strategic understanding while still playing in a fundamentally human way. For instance, they might be more open to positions with long-term compensation for material, but they'll typically only choose such lines when the compensation is more concrete and calculable than the extremely abstract advantages AlphaZero sometimes plays for.
Would you be interested in exploring any specific examples of how top players have adapted these computer-inspired innovations?
What interests me is the question of whether or not humans are able to learn these techniques or whether they’re more or less permanently beyond our grasp. 1 and 3 seem to be related and 1, in particular, seems grounded in capacities computers have but that humans will never have. 2 seems to be a different kind of factor.