Scaling won't get you to AGI

Pioneer of causal AI, Judea Pearl, argues that no amount of scaling will get LLMs to AGI.

He believes current large language models face fundamental mathematical limitations that can't be solved by making them bigger.

"There are certain limitations, mathematical limitation that… pic.twitter.com/xEpBKQReEj

— Big Brain AI (@realBigBrainAI) February 18, 2026

From the tweet:

When hospitals collect data on treatment effects, that raw data never reaches the LLMs.

Instead, the models consume doctors' written interpretations. Analyses shaped by people who already have a mental model of how disease and treatment work.

In other words, LLMs are learning from the map, not the territory.

Large Language Model Reasoning Failures

Peiyang Song, Pengrui Han, Noah Goodman, Large Language Model Reasoning Failures, arXiv:2602.06176v1 [cs.AI] https://doi.org/10.48550/arXiv.2602.06176

Abstract: Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, achieving impressive results across a wide range of tasks. Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios. To systematically understand and address these shortcomings, we present the first comprehensive survey dedicated to reasoning failures in LLMs. We introduce a novel categorization framework that distinguishes reasoning into embodied and non-embodied types, with the latter further subdivided into informal (intuitive) and formal (logical) reasoning. In parallel, we classify reasoning failures along a complementary axis into three types: fundamental failures intrinsic to LLM architectures that broadly affect downstream tasks; application-specific limitations that manifest in particular domains; and robustness issues characterized by inconsistent performance across minor variations. For each reasoning failure, we provide a clear definition, analyze existing studies, explore root causes, and present mitigation strategies. By unifying fragmented research efforts, our survey provides a structured perspective on systemic weaknesses in LLM reasoning, offering valuable insights and guiding future research towards building stronger, more reliable, and robust reasoning capabilities. We additionally release a comprehensive collection of research works on LLM reasoning failures, as a GitHub repository at this https URL, to provide an easy entry point to this area.

Intellectual creativity, humans-in-the-loop, and AI: Interlude, Claude on “xanadu” and the metaphysical structure of the world

After yesterday’s post about the Xanadu meme went up I started wondering just how an AI, a current AI, would even confront such a question. Why would an AI issue a simple one-word query to the web: “Xanadu”? And when it got 2,000,000 hits, what reason would it have to think that a large number? I decided to have a discussion with Claude Sonnet 4 about “xanadu.” That eventually led to an interesting discussion that ended with something I’ve been calling “the metaphysical structure of the world.” I introduced the term in a blog post from August of 2020, which I’d written in the wake of GPT-3 and then included it in my working paper, GPT-3: Waterloo or Rubicon? Here be Dragons. I’m beginning to think it might be useful. 

On the way there ChatGPT made the following observation

Current AI systems are remarkable at following up on investigations once they’re pointed in a direction. But that initial moment of recognition - “this might be interesting” - seems to require a different kind of agency. It’s not just pattern recognition; it’s pattern recognition combined with intrinsic motivation and aesthetic judgment about what’s worth pursuing.

Yes.

As always in these discussions, my prompts are in boldface.

* * * * *

“Xanadu”

What comes to mind when I say “Xanadu”?

Several things come to mind with “Xanadu”:

Literary: Most famously, it’s from Coleridge’s poem “Kubla Khan” - “In Xanadu did Kubla Khan / A stately pleasure-dome decree” - describing an exotic, dreamlike palace.

Historical: The actual Xanadu (Shangdu) was Kublai Khan’s summer capital in 13th-century Mongolia, which inspired Coleridge’s poem.

Pop culture: The 1980 musical film with Olivia Newton-John and Gene Kelly, along with its title song. There’s also Rush’s epic song “Xanadu” from their album A Farewell to Kings.

Cinema: In Citizen Kane, it’s the name of Charles Foster Kane’s sprawling, isolating estate.

The word has become synonymous with an idealized, fantastical place - somewhere luxurious but perhaps unattainable or illusory. It carries this sense of exotic grandeur mixed with a hint that such perfection might be more dream than reality.

What made you think of Xanadu? Were you thinking of one of these references in particular?

I was just curious. Back in January of 2006 I typed “Xanadu” into Google and got 2,000,000. That surprised me. “Xanadu” is not a common term. People don’t have much of an opportunity to use it. Yet there were 2 million web pages that mention it.

That’s a fascinating observation! You’ve hit on something really interesting about how language spreads in digital spaces versus everyday conversation.

“Xanadu” is indeed quite rare in daily speech - most people might go years without using it. But on the web, it has this remarkable reach because it’s such an evocative, culturally loaded word. Each of those cultural touchstones I mentioned creates its own cluster of web content: literary analysis sites discussing Coleridge, movie review sites covering the film, music sites with Rush lyrics, film studies pages about Citizen Kane.

Plus, “Xanadu” is exactly the kind of word that gets borrowed for business names, product names, gaming references, fantasy literature, and artistic projects. It carries this instant sense of mystery and grandeur that appeals to people creating content.

Your 2006 observation captures something about how the internet amplifies certain words that have cultural resonance far beyond their frequency in speech. Words that are “mythologically useful” - like Xanadu, Atlantis, Eden - probably punch way above their weight in search results because they’re so semantically rich.

It’s a perfect example of how digital archaeology can reveal hidden patterns in our collective cultural consciousness. Did you notice similar patterns with other evocative but uncommon words when you were exploring search results back then?

It didn’t take long to figure out that most of the hits did not link to pages mentioning Coleridge’s poem. After a bit I determined that a lot of the hits belonged in what I termed the “sybaritic” cluster, which I trace back to Citizen Kane. Lots of people saw that movie and it has a fairly substantial cultural presence. The movie itself quotes the first five lines of the poem in the newsreel sequence and, as you mentioned, there’s Kane’s mansion.

That’s a brilliant piece of cultural archaeology! Your “sybaritic cluster” concept really captures how Citizen Kane became the dominant cultural vector for “Xanadu” rather than the original Coleridge poem.

It makes perfect sense - while English literature students encounter the poem, Citizen Kane reached massive audiences and embedded “Xanadu” firmly in the American cultural lexicon as shorthand for extravagant, perhaps excessive luxury. The film’s Xanadu represents wealth turned grotesque and isolating, but the name itself got detached from that cautionary context.

So you probably found tons of luxury hotels, upscale restaurants, spa resorts, real estate developments, and high-end businesses that borrowed the name purely for its connotations of opulence. They’re not thinking about Mongolian summer palaces or Romantic poetry - they’re thinking about that sense of ultimate luxury that Kane’s estate represented.

This is such a neat example of how cultural transmission works. The literary source provides the raw material, but a hugely influential film recontextualizes it and creates the dominant semantic pathway. Most people using “Xanadu” for their luxury business probably have no idea they’re referencing Coleridge - they’re unconsciously channeling Welles.

Did you find that pattern held up as you dug deeper? Were there other cultural artifacts where you could trace similar semantic drift from high culture sources through popular media?

Ted Nelson’s Project Xanadu

I also identified a cluster I called the “cybernetic” cluster. For example, there’s a spot on Saturn’s moon, Titan, that I put in that cluster. Where do you think this cybernetic cluster comes from?

Ah, that has to trace back to Ted Nelson’s “Project Xanadu” - his pioneering hypertext system from the 1960s! Nelson chose the name explicitly referencing Coleridge’s poem, envisioning his system as a kind of “magic place of literary memory” where all human knowledge could be interconnected.

Poetry and the digital simulacrum of mind [poetry as digital touchstone]

I'm bumping this to the top of the queue on general principle.

* * * * *

Carmine Starnino, Poetry & Digital personhood, The New Criterion, April 2022.

Starnino starts out by talking about Racter, a well-known computer generator of poem simulacra from the 1980s, and then moves on to GPT-3: “...GPT-3 isn’t a better Racter. It’s a godlike Racter.” Then mixes in a little of this and that and observes:

The Turing Test, after all, has shown that readers have a weakness for rhetoric, grand gestures, and feelingful murk—all of which algorithms easily mimic. If this is what we mean when we say AI will one day rival human poets, then it will surely win, and indeed may already have.

Yes! Our willingness to read meaning to any quasi-intelligible lump of language makes it easy for computers to crank out simulacra of poetic profundity. That’s a low bar to cross.

Starnino goes on:

But there’s another kind of poetry AI will have to beat—poetry as an art of brilliant accuracies, of reality re-described in ways that bind sound to perception. And here AI’s deficiencies are brutally exposed. Because to compete at this imitation game, a machine has to show that, by micro-adjustments of effect, it can draw our senses to the highest pitch of expression. It will need to be able to match Les Murray’s depiction of beans as “minute green dolphins at suck,” or Peter van Toorn’s realization that flying dragonflies have “a great rattle of rice in their wings,” or Elizabeth Bishop’s noting a fish’s “coarse white flesh/ packed in like feathers,” or how, for Seamus Heaney, love was “like a tinsmith’s scoop/ sunk past its gleam/ in the meal-bin.” To play at this level, a machine has to imbue words with the most intimate associations and, turning inward, confess hardships, regret irrevocable choices, ponder its ultimate demise. It has to hit the same mark Robert Frost does at the end of his sonnet “Design,” when, watching a spider readying itself to eat a moth, he asks what the grim scene reveals about nature—and if there is any moral code to such predation. “What but design of darkness to appall?—/ If design govern in a thing so small.” The word “appall” here logs the shock perfectly. In its French root, the word means to make white. It also contains “pall”: a sheet, laid over a coffin, usually of white linen. Thus Frost’s diction hones our cognition, schooling us to see the world in a fresh way.

Yes.

Starnino goes on to remind us of Eliza:

Released in 1966 by the MIT professor Joseph Weizenbaum, ELIZA was the world’s first chatbot. Designed to impersonate a therapist, it would reflect back a user’s statements with open-ended questions and prepared responses (“My mother never loved me” would trigger “please go on” or “tell me more.”) Weizenbaum’s goal was to explore a computer’s capacity for conversation. Instead, he was alarmed by how completely users were taken in by ELIZA’s shallow repartee; his own secretary once insisted he leave the room so she could talk to the program in private. Credulity even extended to graduate students who had watched him build ELIZA from scratch. Sherry Turkle, a social scientist and Weizenbaum’s colleague, called it “the Eliza effect,” which she defined as “human complicity in a digital fantasy.” We can see this effect in the love-struck language Racter’s programmers used to describe the moment their creation came to life.

A bit later he notes:

It’s no coincidence that each time a new threshold is smashed, poetry is soon offered up as evidence of the breakthrough. The most profound exercise of full human consciousness, poetry has long been coveted as a benchmark for silicon-based minds, the ultimate proof of concept. Its principles were not only present at the founding of artificial intelligence as a field—the 1956 conference that set out to design machines able to “use language, form abstractions and concepts”—but every step in eroding the line between robots and people has been marked by a poetry generator. When the famed futurist Ray Kurzweil wanted to sell the public on the idea of a thinking machine in the late 1980s, he began by inventing a “Cybernetic Poet.” In fact, you can even argue that the pursuit of machine poetry has driven entire sectors of AI, helping push the limits of what language models can now do.

Hmmmm. I can’t help but think that, to the extent that that is true, my 1970s work using computational semantics to analyze a Shakespeare sonnet* deserves a re-reading, and that despite the fact that it is firmly entrenched in the era of symbolic computing.

Citing AI’s recent successes in both GO and Chess, Starnino offers and interesting argument. He notes that when Deep Blue beat Kasparov in 1997 “the IBM supercomputer appeared capable of counterintuitive thought with a baffling move that left Kasparov profoundly unnerved.” Similarly, when AlphaGo be Lee Sedol in Go in 2016 it did so with “a move that so stunned Sedol with its strangeness, he needed fifteen minutes to recover.” AI didn’t beat us in those games by learning to think about them in a human way. Rather:

It beat us because it learned to think in an entirely inhuman way. The scale of AI’s processing power—able to mull millions of strategies and pit itself against those strategies millions of times—found bizarre but superior solutions that centuries of flesh-and-blood play never considered, solutions so removed from normal reasoning as to be alien.

However, poetry

is inexorably linked to how humans think—a kind of undeluded self-questioning that, as T. S. Eliot wrote, helps us become “a little more aware of the deeper, unnamed feelings which form the substratum of our being.” It’s also tied to the need to think this way. A poem’s mental force derives from the set of intentions driving it, intentions that push poets into action. But when GPT-3 gets the call to write a poem, it doesn’t know it’s writing “poetry,” or what “writing” even is. That last part is anything but trivial. Style is a sentient act: you strive for it. My point is that a computer will never replicate what poets do unless it can also replicate why they do it.

There’s more at the link.

* William Benzon, Cognitive Networks and Literary Semantics, MLN 91: 1976, 952-982, https://www.academia.edu/235111/Cognitive_Networks_and_Literary_Semantics.

William Benzon, Lust in Action: An Abstraction, Language and Style 14, 1981, 251-270, https://www.academia.edu/7931834/Lust_in_Action_An_Abstraction.

Thoughts on the implications of GPT-3, two years ago and NOW [here be dragons, we're swimming, flying and talking with them]

When GPT-3 first came out, I registered my first reactions in a comment at Marginal Revolution, which I've appended immediately below the picture of Gojochan and Sparkychan. I'm currently completing a working paper about my interaction with ChatGPT. That will end with an appendix in which I repeat my remarks from two years ago and append some new ones. I've appended those after the comment to Marginal Revolution.

* * * * *

A bit revised from a comment I made at Marginal Revolution:

Yes, GPT-3 [may] be a game changer. But to get there from here we need to rethink a lot of things. And where that's going (that is, where I think it best should go) is more than I can do in a comment.

Right now, we're doing it wrong, headed in the wrong direction. AGI, a really good one, isn't going to be what we're imagining it to be, e.g. the Star Trek computer.

Think AI as platform, not feature (Andreessen). Obvious implication, the basic computer will be an AI-as-platform. Every human will get their own as an very young child. They're grow with it; it'll grow with them. The child will care for it as with a pet. Hence we have ethical obligations to them. As the child grows, so does the pet – the pet will likely have to migrate to other physical platforms from time to time.

Machine learning was the key breakthrough. Rodney Brooks' Gengis, with its subsumption architecture, was a key development as well, for it was directed at robots moving about in the world. FWIW Brooks has teamed up with Gary Marcus and they think we need to add some old school symbolic computing into the mix. I think they're right.

Machines, however, have a hard time learning the natural world as humans do. We're born primed to deal with that world with millions of years of evolutionary history behind us. Machines, alas, are a blank slate.

The native environment for computers is, of course, the computational environment. That's where to apply machine learning. Note that writing code is one of GPT-3's skills.

So, the AGI of the future, let's call it GPT-42, will be looking in two directions, toward the world of computers and toward the human world. It will be learning in both, but in different styles and to different ends. In its interaction with other artificial computational entities GPT-42 is in its native milieu. In its interaction with us, well, we'll necessarily be in the driver's seat.

Where are we with respect to the hockey stick growth curve? For the last 3/4 quarters of a century, since the end of WWII, we've been moving horizontally, along a plateau, developing tech. GPT-3 is one signal that we've reached the toe of the next curve. But to move up the curve, as I've said, we have to rethink the whole shebang.

We're IN the Singularity. Here be dragons.

[Superintelligent computers emerging out of the FOOM is bullshit.]

* * * * *

ADDENDUM: A friend of mine, David Porush, has reminded me that Neal Stephenson has written of such a tutor in The Diamond Age: Or, A Young Lady's Illustrated Primer (1995). I then remembered that I have played the role of such a tutor in real life, The Freedoniad: A Tale of Epic Adventure in which Two BFFs Travel the Universe and End up in Dunkirk, New York.

* * * * *

To the future and beyond!

I stand by those remarks from two years ago, but I want to comment on four things: 1) AI alignment, 2) the need for symbolic computing, 3) the need for new kinds of hardware, and 4) a future world in which humans and AIs interact freely.

Considerable effort has gone into tuning ChatGPT so that it won’t say things that are offensive (e.g. racial slurs) or give out dangerous information (e.g. how to hotwire cars). These efforts have not been entirely successful. This is one aspect of what is now being called “AI alignment.” In the extreme the field of AI alignment is oriented toward the possibility – which some see as a certainty – that in the future (somewhere between, say, 30 and 130 years) rogue AIs will wage a successful battle against humankind.[1] I don’t think that fear is very creditable, but, as the rollout of ChatGPT makes abundantly clear, AIs built on deep learning are unpredictable and even, in some measure, uncontrollable.

I think the problem is inherent in deep learning technology. Its job is to fit a model to, in the case of ChatGPT, an extremely large corpus of writing, much of the internet. That corpus, in turn, is ultimately about the world. The world is vast, irregular, and messy. That messiness is amplified by the messiness inherent in the human brain/mind, which did, after all, evolve to fit that world. Any AI engine capable of capturing a significant portion of the order inherent in our collective writing about the world has no choice but to encounter and incorporate some of the disorder and clutter into its model as well.

I regard such Foundation models[2], as they have come to be called, as wilderness preserves, digital wilderness. They contain what digital humanist Ted Underwood calls the latent space of culture. He says:

The immediate value of these models is often not to mimic individual language understanding, but to represent specific cultural practices (like styles or expository templates) so they can be studied and creatively remixed. This may be disappointing for disciplines that aspire to model general intelligence. But for historians and artists, cultural specificity is not disappointing. Intelligence only starts to interest us after it mixes with time to become a biased, limited pattern of collective life. Models of culture are exactly what we need.

In his penultimate paragraph Underwood notes:

I have suggested that approaching neural models as models of culture rather than intelligence or individual language use gives us even more reason to worry. But it also gives us more reason to hope. It is not entirely clear what we plan to gain by modeling intelligence, since we already have more than seven billion intelligences on the planet. By contrast, it’s easy to see how exploring spaces of possibility implied by the human past could support a more reflective and more adventurous approach to our future. I can imagine a world where generative models of culture are used grotesquely or locked down as IP for Netflix. But I can also imagine a world where fan communities use them to remix plot tropes and gender norms, making “mass culture” a more self-conscious, various, and participatory phenomenon than the twentieth century usually allowed it to become.

These digital wildness regions thus represent opportunities for discovery and elaboration. Alignment is simply one aspect of that process.

And by alignment I mean more than aligning the AI’s values with human values; I mean aligning its conceptual structure as well. That’s where “old school” symbolic computing enters the picture, especially language. Language  – not the mere word forms available in digital corpora, but word forms plus semantics and syntactic affordances –  is one of the chief ‘tools’ through which young humans are acculturated and through which human communities maintain their beliefs and practices. The full powers of language, as treated by classical symbolic systems, will be essential for “domesticating” the digital wilderness and developing it for human use.

However, this presents technical problems, problems I cannot go into here in any detail.[4] The basic issue is that symbolic computing involves one strategy for implementing, call it cogitation, in a physical system while the neural computing underlying deep learning requires a different physical implementation. These approaches are incompatible. While one can “bolt” a symbolic system onto a neural computing system, that strikes me as no more than an interim solution. It will get us started, indeed the work has already begun.[5]

What we want, though, is for the symbolic system to arise from the neural system, organically, as it does in humans.[6] This may well call for fundamentally new physical platforms for computing, platforms based on “neuromorphic” components that are “grown,” as Geoffrey Hinton has recently remarked.[7] That technology will give us a whole new world, one where humans, AIs and robots interact freely with one another, but will have communities of their own as well. We know that dogs co-evolved with humans over tens of thousand of years. These miraculous new devices will co-evolve with us over the coming decades and centuries.

Let us end with Miranda’s words from Shakespeare’s The Tempest:

“Oh wonder!
How many goodly creatures are there here!
How beauteous mankind is! Oh brave new world,
That has such [devices] in’t.”

* * * * *

[1] The virtual center of this belief is a website called LessWrong, which has extensive discussion of this issue going back well over a decade. Here it is, https://www.lesswrong.com/.

[2] Foundation models, Wikipedia, https://en.wikipedia.org/wiki/Foundation_models.

[3] Ted Underwood, Mapping the latent spaces of culture, The Stone and the Shell, Oct. 21, 2021, https://tedunderwood.com/2021/10/21/latent-spaces-of-culture/.

[4] I discuss this issue in this blog post, Physical constraints on computing, process and memory, Part 1 [LeCun], New Savanna, July 24, 2022, https://new-savanna.blogspot.com/2022/07/physical-constraints-on-computing.html.

[5] Consult the Wikipedia entry, Neuro-symbolic AI, for some pointers, https://en.wikipedia.org/wiki/Neuro-symbolic_AI.

[6] I discuss this in a recent working paper, Relational Nets Over Attractors, A Primer: Part 1, Design for a Mind, Version 2, Working Paper, July 13, 2022, pp. 76, https://www.academia.edu/81911617/Relational_Nets_Over_Attractors_A_Primer_Part_1_Design_for_a_Mind.

[7] Tiernan Ray, We will see a completely new type of computer, says AI pioneer Geoff Hinton, ZDNET, December 1, 2022, https://www.zdnet.com/article/we-will-see-a-completely-new-type-of-computer-says-ai-pioneer-geoff-hinton-mortal-computation/#ftag=COS-05-10aaa0j.

On limits to the ability of LLMs to approximate the mind’s structure

We assume that the mind has some structure. That structure is a function of 1) the structure of the world (which includes other humans) and 2) the brain’s ability to ‘model’ that structure. See my post, World, mind, and learnability: A note of the metaphysical structure of the cosmos.

The structure of the mind is not open to us through direct inspection. We need to approximate it by various indirect methods. Several academic disciplines are devoted to this job. Artificial intelligence approaches it by constructing computer programs that behave ‘intelligently.’ Deep learning is one such approach. Large Language Models are an approach that has recently received a great deal of attention.

Large Language Models

LLMs attempt to approximate the mind’s structure using a a procedure in which they attempt to guess the next word in a string they are currently examining. The model being developed is modified according to whether or not the guess is correct. In practice the ‘string’ in question is a concatenation of many many many strings that have been scraped from WWW.

Are there any theorems on what kinds of structures are discoverable through such a procedure? One current view posits that we can reach AGI – whatever that is – simply by scaling up. What does that view presuppose about the structure of the mind? Is it possible that there exists a mind – perhaps a super-intelligence of some kind – whose structure cannot be approximated by such a procedure?

Current LLMs are known to have difficulty with so-called common-sense reasoning. Some non-trivial component of common-sense reasoning consists of examples that are “close” to the physical world. Can this difficulty be overcome simply by scaling up? Why or why not? Note that if not, that suggests that there IS some kind of mental structure that cannot be discovered by the standard guess-the-next-word procedure. How can we characterize that structure?

I believe I believe that a 1975 paper by Miriam Yevick speaks to this issue, Holographic or fourier logic, and have appended a complete reference along with its abstract and summary. I also have a recent post setting forth her ideas: Miriam Yevick on why both symbols and networks are necessary for artificial minds.

If Yevick is correct, then mental structures and processes of the kind she calls holographic may not be very well approximated through the standard LLM training procedure. Beyond this I note the human minds are open to the external world and always “pursuing” it. I suggest that implies that they must necessarily “run ahead” of any model trained on texts, no matter how large the corpus.

Miriam Yevick 1975

I believe that a 1975 paper by Miriam Yevick speaks to this issue: Holographic or fourier logic, Pattern Recognition, Vol 7, #2, 1975, pp. 197-213. The abstract:

A tentative model of a system whose objects are patterns on transparencies and whose primitive operations are those of holography is presented. A formalism is developed in which a variety of operations is expressed in terms of two primitives: recording the hologram and filtering. Some elements of a holographic algebra of sets are given. Some distinctive concepts of a holographic logic are examined, such as holographic identity, equality, contaminent and “association”. It is argued that a logic in which objects are defined by their “associations” is more akin to visual apprehension than description in terms of sequential strings of symbols.

Concluding summary:

It has recently been conjectured that neural holograms enter as units in the thought process. If holographic processes do occur in the brain and are instrumental in thought, then the logical operations implicit in these processes could be considered as intuitive and enter as units in our mental and mathematical computations.

It has also been said that: "if we want the computer to have eyes, we shall first have to give him instruction in the facts of life".

We maintain in this paper that a language of thought in which holographic operations enter as primitives is essentially different from one in which the same operations are carried out sequentially and hence over a finite time span, as would be the case if they were computed by a neural network. Our assumption is that "holographic thought" utilizes the associative properties of holograms in "one shot". Similarly we maintain that apprehension proceeds from the very beginning via two modes, the aural and the optical; whereas the verbal string is natural to the first, the pattern as such is natural to the second: the essentially instantaneous nature of the optical process captures the apprehension as a global unit whose meaning is expressed in the first place in terms of "associations" with other such units.

We are hence led to search for a language of patterns based on the logic of holographic processes. In the first part of this paper we identify and express a number of derived holographic operations in terms of the two primitive operations of “recording the hologram” and “filtering.” We also derive some elements of a holographic algebra of patterns. In the second part some potentially distinctive aspects of a holographic logic are examined, such as holographic identity (directly related to randomness), equality, containment and "association". The notion of the Gödel Pattern is introduced as a bridge between such an associative, optical language and the usual sequential string of symbols of a formal language considered as "mere scratches on paper".

We speculate on the potential relation of the notion of holographic association to the, as yet unclarified, notion of "connotation" in logic. We also find that some of the concepts developed in this paper graze the boundaries of both the uncertainty principle and undecidability questions in self-referential systems. They may, perhaps, open up further insights into the connection, if any, between these two.

Rodney Brooks on compromising about human-level AI

From [FoR&AI] Steps Toward Super Intelligence II, Beyond the Turing Test:

One argument is that we should not need to take into account how humans come to be intelligent, nor try to emulate them, as heavier than air flight does not emulate birds. That is only partially true as there were multiple influences on the Wright brothers from bird flight. Certainly today the appearance of heavier than air flight is very different from that of birds or insects, though the continued study of the flight of those creatures continues to inform airplane design. This is why over the last twenty years or so jet aircraft have sprouted winglets at the ends of primary wings.

Airplanes can fly us faster and further than something that more resembled birds would. On the other hand our airplanes have not solved the problem of personal flight. We can no more fly up from the ground and perch in a tall tree than we could before the Wright brothers. And we are not able to take off and land wherever we want without large and extremely noisy machines. A little more bird would not be all bad.

I accept the point that to build a human level intelligence it may well not need to be much at all like humans in how it achieves that. However, for now at least, it is the only model we have and there is most likely a lot to learn still from studying how it is that people are intelligent. Furthermore, as we will see below, having a lot of commonality between humans and intelligent agents will let them be much more understandable partners.

This is the compromise that I am willing to make. I am willing to believe that we do not need to do everything like humans do, but I am also convinced that we can learn a lot from humans and human intelligence.

H/t Zach.

That essay is second in a series of four. Here's the others:

[FoR&AI] Steps Toward Super Intelligence I, How We Got Here

[FoR&AI] Steps Toward Super Intelligence III, Hard Things Today

[FoR&AI] Steps Toward Super Intelligence IV, Things to Work on Now

The machine in my mind, my mind on the machine: Will we ever build a machine to equal the human brain?

Some years ago, back in the Jurassic Era, I imagined that one day I would have access to a computer system I could use to “read” literary texts, such as a play by Shakespeare. As such a system would have been based on a simulation of the human mind, I would be able to trace its operations as it read through a text – perhaps not in real time, but it would store a trace of those actions and I could examine that trace after the fact. Alas, such a system has yet to materialize, nor do I expect to see such a marvel in my lifetime. As for whether or not such a system might one day exist, why bother speculating? There’s no principled way to do so.

Whyever did I believe such a thing? I was young, new to the research, and tremendously excited by it. And, really, no one knew what was possible back in those days. Speculation abounded, as it still does.

While I’ll return to that fantasy a bit later, this is not specifically about that fantasy. That fantasy is just one specific example of how I have been thinking about the relationship between the human mind and computational approximations to it. As far as I can recall I have never believed that one day it would be possible to construct a human mind in a machine. But I have long been interested in what the attempt to do so has to teach us about the mind. This post is a record of much of my thinking on the issue.

It’s a long way through. Sit back, get comfortable.

The early years

When I was nine years old I saw Forbidden Planet, which featured a Robot named Robbie. In the days and weeks afterward I drew pictures of Robbie. Whether or not I believed that such a thing would one day exist, I don’t remember.

Some years later I read an article, either in Mechanix Illustrated or Popular Mechanics – I read both assiduously – about how Russian technology was inferior to American. The article had a number of photographs, including one of a Sperry Univac computer – or maybe it was just Univac, but it was one of those brands that no longer exists – and another, rather grainy one, of a Russian computer that had been taken from a Russian magazine. The Russian photo looked like a slightly doctored version of the Sperry Univac photo. That’s how it was back in the days of electronic brains. 

I also remember that computers were often referred to as “electronic brains.” And then that phrase died out. Here’s a Google Ngram chart depicting that:

When I went to college at Johns Hopkins in the later 1960s one of the minor curiosities in the freshman dorms was an image of a naked woman ticked out in “X”’s and “O”’s on computer print-out paper. People, me among them, actually went to some guy’s dorm room to see the image and to see the deck of punch cards that, when run through the computer, would cause that image to be printed out. Who’d have thought, a picture of a naked woman – well sorta’, that particular picture wasn’t very exciting, it was the idea of the thing – coming out of a computer. These days, of course, pictures of naked women, men too, circulate through computers around the world.

Two years after that, my junior year, I took a course in computer programming, one of the first in the nation. As things worked out, I never did much programming, though some of my best friends make their living at the craft. I certainly read about computers, information theory, and cybernetics. The introductory accounts I read always mentioned analog computing as well as digital, but that ceased some years later when personal computers became widespread.

I saw 2001: A Space Odyssey when it came out in 1968. It featured a computer, HAL, that ran a spacecraft while on a mission to Jupiter. HAL decided that the human crew endangered the mission and set about destroying them. Did I think that an artificially intelligent computer like HAL would one day be possible? Nor do I recall what I thought about the computers in the original Star Trek television series. Those were simply creatures of fiction. I felt no pressure to form a view about whether or not they would really be possible.

Graduate school (computational semantics)

In 1973 I entered the Ph.D. program in the English Department at the State University of New York at Buffalo. A year later I was studying computational semantics with David Hays in the Linguistics Department. Hays had been a first-generation researcher in machine translation at the RAND Corporation in the 1950s and 1960s and left RAND to found SUNY’s Linguistics Department in 1969. I joined his research group and also took a part-time job preparing abstracts of the technical literature for the journal Hays edited, The American Journal of Computational Linguistics (now just Computational Linguistics). That job required that I read widely in computational linguistics, linguistics, cognitive science, and artificial intelligence.

I note, in passing, that computational linguistics and artificial intelligence were different enterprises at the time. They still are, with different interests and professional associations. By the late 1960s and 1970s, however, AI had become interested in language, so there was some overlap between the two communities.

In 1975 Hays was invited to review the literature in computational linguistics for the journal Computers and Humanities. Since I was up on the technical literature, Hays asked me to coauthor the article:

William Benzon and David Hays, “Computational Linguistics and the Humanist”, Computers and the Humanities, Vol. 10. 1976, pp. 265-274, https://www.academia.edu/1334653/Computational_Linguistics_and_the_Humanist.

I should note that all the literature we covered was within that has come to be known the symbolic approach to language and AI. Connectionism and neural networks did not exist at that time.

Our article had a section devoted to semantics and discourse in which we observed (p. 269):

In the formation of new concepts, two methods have to be distinguished. For a bird, a creature with wings, its body and the added wings are equally concrete or substantial. In other cases something substantial is molded by a pattern. Charity, an abstract concept, is defined by a pattern: Charity exists when, without thought of reward, a person does something nice for someone who needs it. To dissect the wings from the bird is one kind of analysis; to match the pattern of charity to a localized concept of giving is also an analysis, but quite different. Such pattern matching can be repeated recursively, for reward is an abstract concept used in the definition of charity. Understanding, we believe, is in one sense the recursive recognition of patterns in phenomena, until the phenomenon to be understood fits a single pattern.

Hays had explored that concept of abstract concepts in an article on concepts of alienation. One of his students, Brian Phillips, was finishing a computational dissertation in which he used that concept in analyzing stories about drownings. Another student, Mary White, was finishing a dissertation in which she analyzed the metaphysical concepts of a millenarian community. I was about to publish a paper in which I used the concept to analyze a Shakespeare sonnet, “The Expense of Spirit” (Cognitive Networks and Literary Semantics). That set the stage for the last section of our article.

Diagram from "Cognitive Networks and Literary Semantics"

 

We decided to go beyond the scope of a review article to speculate about what might one day be possible (p. 271):

Let us create a fantasy, a system with a semantics so rich that it can read all of Shakespeare and help in investigating the processes and structure that comprise poetic knowledge. We desire, in short, to reconstruct Shakespeare the poet in a computer. Call the system Prospero.

How would we go about building it? Prospero is certainly well beyond the state of the art. The computers we have are not large enough to do the job and the architecture makes them awkward for our purpose. But we are thinking about Prosper now, and inviting any who will to do the same, because the blueprints have to be made before the machine can be built.

A bit later (p. 272-273):

The state of the art will support initial efforts in any of these directions. Experience gained there will make the next step clearer. If the work is carefully planned, knowledge will grow in a useful way. [...]

We have no idea how long it will take to reach Prospero. Fifteen years ago one group of investigators claimed that practical automatic translation was just around the corner and another group was promising us a computer that could play a high-quality game of chess. We know more now than we did then and neither of those marvels is just around the current corner. Nor is Prospero. But there is a difference. To sell a translation made by machine, one must first have the machine. Humanistic scholars are not salesmen, and each generation has its own excitement. In some remote future may lie the excitement of using Prospero as a tool, but just at hand is the excitement of using Prospero as a distant beacon. We ourselves and our immediate successors have the opportunity to clarity the mechanisms of artistic creation and interpretation. We may well value that opportunity, which can come but once in intellectual history.

Ramble: Will computers ever be able to think like humans? Do I care?

Honestly, I don’t know. Something’s been going on in how I think about this kind of question, but I’m not quite sure what it is. It seems mostly intuitive and inarticulate. This is an attempt at articulation.

I’m sure that I’m at least irritated at the idea that computers will inevitably emerge. The level of irritation goes up when people start predicting when this will happen, especially when the prediction is, say, 30 to 50 years out. Late last year the folks at Open Philanthropy conducted an elaborate exercise in predicting when “human-level AI” would emerge. The original report is by Ajeya Cotra and is 1969 pages long; I’ve not read it. But I’ve read summaries: Scott Summers at Astral Codex Ten and Open Philanthropy’s Holden Karnofsky at Cold Takes). Yikes! It’s a complicated intellectual object with lots of order-of-magnitude estimates and charts. But as an effort at predicting the final – big fanfare here – emergence of artificial general intelligence (AGI). It strikes me as being somewhere between over-kill and pointless. No doubt the effort itself reinforces their belief in the coming of AGI, which is a rather vague notion.

And yet I find myself reluctant to say that some future computational system will never “think like a human being.” I remember when I first read John Searle’s famous Chinese Room argument about the impossibility of artificial intelligence. At the time, 1980 or so, I was still somewhat immersed in the computational semantics work I’d done with David Hays and was conversant with a wide range of AI literature. Searle’s argument left it untouched. Such a wonder would lack intentionality, Searle argued, and without intentionality there can be no meaning, no real thought. At the time “intentionality” struck me as being a word that was a stand-in for a lot of things we didn’t understand. And yet I didn’t think that, sure, someday computers will think. I just didn’t find Searle’s argument convincing. Not then, and not now, not really.

Of course, Searle isn’t the only one to argue against the machine. Hubert Dreyfus did so twenty years before Searle, and on much the same grounds, and others have done so after. I just don’t find the argumentation very interesting.

It seems to me that those arguing strongly against AI implicitly depend on the fact that we don’t have such systems yet. They also have a strong sense of the difference between artificial inanimate systems, like computers, and living beings, like us. Those arguing in favor are depending on the fact that we cannot know the future (no matter how much they try to predict it), which is when these things will happen. They also believe that while, yes, animate and inanimate systems are different, they are both physical systems. It’s the physicality that counts.

None of that strikes me as a substantial basis for strong claims for or against the possibility of machines thinking at the level of humans.

Meanwhile, the actual history of AI has been full of failed predictions and unexpected developments.

We just don’t know what’s going on.

Addendum, 3.29.22: Is the disagreement between two views constructed within the same basic conceptual framework (“paradigm”), or is the disagreement at the level of the underlying framework? 

See also, A general comment concerning arguments about computers and brains, February 16, 2022.

A general comment concerning arguments about computers and brains

Arguments about whether or not computers will ever match the powers of the human mind have been around for a long time. As far as I can recall the first such argument I gave serious attention to was John Searle’s Chinese Room argument. I found it unsatisfactory as it didn’t address any of the computational mechanisms in use. I still find that bothersome.

And yet I don’t believe that one day computers will match or exceed the general capacities of the human mind. Of course, in many domains, they already exceed our capacities. We’re not talking about them. We’re talking about something called “general intelligence.”

It seems to me that, in the end, the arguments against computer intelligence get much, if not in fact most, of their force from the fact that they are currently inferior to humans and there is no obvious immediate prospect of them catching up. On the other hand, the arguments in favor of computer intelligence (catching up to or exceeding human intelligence) get much of their force from the fact that we cannot know the future. We have a much deeper understanding of the requirements for sending humans to Mars than we do of creating the (mythical) artificial general intelligence (AGI).

This is not a very encouraging state of affairs.

Geoffrey Hinton says deep learning will do everything. I’m not sure what he means, but I offer some pointers. Version 2.

This is updated from a previous version to include a passage by Sydney Lamb.

* * * * *

Late last year Geoffrey Hinton had an interview with Karen Hao [1] in which he said “I do believe deep learning is going to be able to do everything,” with the qualification that “there’s going to have to be quite a few conceptual breakthroughs.” I’m trying to figure out whether or not, to what extent, in what way I (might) agree with him.

Neural Vectors, Symbols, Reasoning, and Understanding

Hinton believes that “What’s inside the brain is these big vectors of neural activity” and that one of the breakthroughs we need is “how you get big vectors of neural activity to implement things like reason.” That will certainly require a massive increase in scale. Thus while GPT-3 has 175 billion parameters, the brain has trillion, where Hinton treats each synapse as a parameter. 

Correspondingly Hinton rejects the idea that symbolic reasoning is primitive to the nervous system (my formulation), rather “we do internal operations on big vectors.” What about language? He doesn’t address the issue directly but he does say that “symbols just exist out there in the external world.” I do think that covers language, speech sounds, written words, gestural signs, those things are out there in the external world. But the brain uses “big vectors of neural activity” to process those. 

Hinton’s remark about symbols bears comparison with a remark by Sydney Lamb: “the linguistic system is a relational network and as such does not contain lexemes or any objects at all. Rather it is a system that can produce and receive such objects. Those objects are external to the system, not within it”[2]. Lamb has come to think of his approach as neurocognitive linguistics and, while his sense of the nervous system is somewhat different from Hinton’s, they agree on this issue and, in the current intellectual climate, that agreement is of some significance. For Lamb is a first generation researcher in machine translation  and so was working when most AI research was committed to symbolic systems. We’ll return to Lamb later as I think the notation he developed is a way to bring symbolic reasoning within range of Hinton’s “big vectors of neural activity”.

But now let’s return to Hinton with a passage from an article he co-authored with Yann LeCun and Yoshua Bengio [3]:

In the logic-inspired paradigm, an instance of a symbol is something for which the only property is that it is either identical or non-identical to other symbol instances. It has no internal structure that is relevant to its use; and to reason with symbols, they must be bound to the variables in judiciously chosen rules of inference. By contrast, neural networks just use big activity vectors, big weight matrices and scalar non-linearities to perform the type of fast ‘intuitive’ inference that underpins effortless commonsense reasoning.

I note, moreover, commonsense reasoning seems to be problematic for everyone.[4]

Let’s look at one more passage from the interview:

For things like GPT-3, which generates this wonderful text, it’s clear it must understand a lot to generate that text, but it’s not quite clear how much it understands.

I’m not sure that it is at all useful to say that GPT-3 understands anything. I think that, in using that term, Hinton is displaying what I’ve come to think of as the word illusion.[5] Briefly, GPT-3’s language model is constructed over a corpus consisting entirely of word forms, of signifiers without signifieds, to use an old terminology. Hinton knows that, of course, but, after all, he understands texts from seeing or hearing word forms alone, as do we all, and so, in effect, credits GPT-3 with somehow having induced meaning from a statistical distribution. The text it generates looks pretty good, no? Yes. And that is something we do need to understand, just what is GPT-3 doing and how does it do it? But this is not the place to enter into that.[6]

I think that GPT-3’s remarkable performance based on such ‘shallow’ material should prompt us into reconsidering just what humans are doing when we produce everyday ‘boilerplate’ text. Consider this passage from LeCun, Bengio, and Hinton, where they are referring to the use of an RNN:

This rather naive way of performing machine translation has quickly become competitive with the state-of-the-art, and this raises serious doubts about whether understanding a sentence requires anything like the internal symbolic expressions that are manipulated by using inference rules. It is more compatible with the view that everyday reasoning involves many simultaneous analogies that each contribute plausibility to a conclusion
.

In dealing with these utterly remarkable devices, we would be rein in our narcissistic investment in the routine use of our ‘higher’ cognitive and linguistic capacities as opposed to our mere sensory-motor competence. It’s all neural vectors. 

Note, however, that it is one thing to say that “we do internal operations on big vectors.” I agree with that. That’s not quite the same as saying we can do everything with deep learning. Deep learning is a collection of architectures, but I’m not sure such architectures are adequate for internalizing the vectors needed to effectively mimic human perceptual and cognitive behavior. The necessary conceptual breakthroughs will likely take us considerably beyond deep learning engines. With that qualification, let’s continue.

How the brain might be doing it

I find that, with the caveats I’ve mentioned, this is rather congenial. Which is to say that I can make sense of it in terms of issues I’ve thought through in my own work.

Some years ago David Hays and I wanted to come to terms with neuroscience and ended up reviewing a wide range of work and writing a paper entitled, “Principles and Development of Natural Intelligence.”[7] The principles are ordered such that principle N assumed N-1. We called the fifth and last principle indexing:

The indexing principle is about computational geometry, by which we mean the geometry, that is, the architecture (Pylyshyn, 1980) of computation rather than computing geometrical structures. While the other four principles can be construed as being principles of computation, only the indexing principle deals with computing in the sense it has had since the advent of the stored program digital computer. Indexed computation requires (1) an alphabet of symbols and (2) relations over places, where tokens of the alphabet exist at the various places in the system. The alphabet of symbols encodes the contents of the calculation while the relations over places, i.e. addresses, provide the means of manipulating alphabet tokens in carrying out the computation. [...] Within the context of natural intelligence, indexing is embodied in language. Linguists talk of duality of patterning (Hockett, 1960), the fact that language patterns both sounds and sense. The system which patterns sound is used to index the system which patterns sense.

In short, “indexing gives computational geometry, and language enables the system to operate on its own geometry.” This is where we get symbols and complex reasoning.

I should note that, while we talked of “an alphabet of symbols” and “relations over places” we were not asserting that that’s what was going on in the brain. That’s what’s actually going on in computers, but it applies only figuratively to the brain. The system that is using sound patterns to index patterns of sense is using one set of neural vectors (though we didn’t use that term) to index a different set of neural vectors.

How do we get deep learning to figure that out? I note that automatic image annotation is a step in that direction [8], but have nothing to say about that here.

Instead I want to mention some informal work I did some years ago on something I call attractor nets.[9] The general idea was to use Sydney Lamb’s relational networks, in which nodes are logical operators, as a tertium quid between the symbol-based semantic networks Hays and I had worked on in the 1970s and the attractor landscapes of Walter Freeman’s neurodynamics. I showed – informally, using diagrams – how using logical operators (AND, OR) over attractor basins in different neurofunctional areas could reconstruct symbolic systems represented as directed graphs. Each node in a symbolic graph corresponds to a basin of attraction, that is, an attractor. In the present context we can think of each neurofunctional area as corresponding to a collection of neural vectors and the attractors as objects represented by those vectors. An attractor net would then become a way of thinking about how complex reasoning could be accomplished with neural vectors.

In the attractor net notation word forms, or signifiers, are distinct from word meanings, of signifieds. Is that distinction important for complex reasoning? I believe it is, though I’m not interested in constructing an argument at this point. That, I believe, puts a limit on what one can expect of engines like GPT-3. That too requires an argument.

So, what about natural vs. artificial intelligence?

The notion of intelligence is somewhat problematic. As a practical matter I believe that a formulation by Robin Hanson is adequate: “’Intelligence’ just means an ability to do mental/calculation tasks, averaged over many tasks.”[10] As for the difference between artificial and natural, that comes down to four things:

1) a living system vs. an inanimate system,
2) a carbon-based organic electro-chemical substrate vs. a silicon-based electronic substrate,
3) real neurons (having on average 10K connections with others) vs. considerably simpler artificial neurons realized in program code, and
4) the neurofunctional architecture and innate capacities of a real brain vs. the system architecture of a digital computing system.

Make no mistake, those differences are considerable. But I think we now have in hand a body of concepts and models that is rich enough to support ever more sophisticated interaction between students of neuroscience and students of artificial intelligence. To the extent that our research and teaching institutions can support that interaction I expect to see progress accelerate in the future. I offer no predictions about what will come of this interaction.

Some related posts

William Benzon, Showdown at the AI Corral, or: What kinds of mental structures are constructible by current ML/neural-net methods? [& Miriam Yevick 1975], New Savanna, June 3, 2020, https://new-savanna.blogspot.com/2020/06/showdown-at-ai-corral-or-what-kinds-of.html.

William Benzon, What’s AI? – Part 2, on the contrasting natures of symbolic and statistical semantics [can GPT-3 do this?], New Savanna, July 17, 2020, https://new-savanna.blogspot.com/2019/11/whats-ai-part-2-on-contrasting-natures.html.

William Benzon, A quick note on the ‘neural code’ [AI meets neuroscience], New Savanna, April 20, 2021, https://new-savanna.blogspot.com/2021/04/a-quick-note-on-neural-code-ai-meets.html.

References

[1] Interview with Karen Hao, AI pioneer Geoff Hinton: “Deep learning is going to be able to do everything”, MIT Technology Review, Nov. 3, 2020. https://www.technologyreview.com/2020/11/03/1011616/ai-godfather-geoffrey-hinton-deep-learning-will-do-everything/. 

[2] Sydney Lamb, Linguistic Structure: A Plausible Theory, Language Under Discussion, 4(1) 2016, 1–37, https://doi.org/10.31885/lud.4.1.229.

[3] From Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, Deep Learning, Nature, 521 28 May 2015, 436-444, https://doi.org/10.1038/nature14539.

[4] As an example I offer a recent post in which I quiz GPT-3 about a Jerry Seinfeld bit: Analyze This! Screaming on the flat part of the roller coaster ride [Does GPT-3 get the joke?], May 7, 2021, https://new-savanna.blogspot.com/2021/05/analyze-this-screaming-on-flat-part-of.html.

[5] See my post, The Word Illusion, May 12, 2021, https://new-savanna.blogspot.com/2021/05/the-word-illusion.html.

[6] For some extended remarks, see my working paper, GPT-3: Waterloo or Rubicon? Here be Dragons, Working Paper, Version 3, August 20, 2020, 34 pp., https://www.academia.edu/43787279/GPT_3_Waterloo_or_Rubicon_Here_be_Dragons_Version_3.

[7] William Benzon and David Hays, Principles and Development of Natural Intelligence, Journal of Social and Biological Structures, Vol. 11, No. 8, July 1988, 293-322, https://www.academia.edu/235116/Principles_and_Development_of_Natural_Intelligence.

[8] Wikipedia, Automatic image annotation, https://en.wikipedia.org/wiki/Automatic_image_annotation.

[9] William Benzon, Attractor Nets, Series I: Notes Toward a New Theory of Mind, Logic, and Dynamics in Relational Networks, Working Paper, 52 pp., https://www.academia.edu/9012847/Attractor_Nets_Series_I_Notes_Toward_a_New_Theory_of_Mind_Logic_and_Dynamics_in_Relational_Networks.

William Benzon, Attractor Nets 2011: Diagrams for a New Theory of Mind, Working Paper, 55 pp., https://www.academia.edu/9012810/Attractor_Nets_2011_Diagrams_for_a_New_Theory_of_Mind.

William Benzon, From Associative Nets to the Fluid Mind, Working Paper. October 2013, 16 pp. https://www.academia.edu/9508938/From_Associative_Nets_to_the_Fluid_Mind.

[10] Robin Hanson, I Still Don’t Get Foom, Overcoming Bias, July 24, 2014, https://www.overcomingbias.com/2014/07/30855.html.

Cognitivism for the Critic, in Four & a Parable [basic conceptual materials for the study of mind in the 21st century]

I’m bumping this to the top of the queue because these matters are much on my mind these days. I recommend, in particular, that you read the section at the end on Simon’s Ant, for it contains a way to approach the common-sense issue that’s plagued AI since the 1970s. Simon’s parable can be taken to mean that the mind is in fact dependent upon the world. Common-sense reasoning exists in that dependency, if you will. A language model that is constructed over texts that are severed from the world – as all such models are – is thus cut off from the source of common-sense reasoning. No amount of text can make up for that loss (see the discussion of Miriam Yevick in Showdown at the AI Corral).

* * * * *

I published this in The Valve in March of 2010. It's a guide on how to get on board with the computational thinking that's the driver in the so-called "cognitive revolution" — which, BTW, has peaked and is now deep into routinization. Computational thinking is a style of thought, a way of looking at the world. Alas, the most familiar example of computation, arithmetic, is not a very good way to get a feel for the style, not as you need it to investigate literature, the arts, or even the human mind. This post is a brief guide to THAT style. 

Note that the first book I recommend is a comic about comics. Forget about MLA-authorized easings into literary cognitivism and similar things and forget about Turner and Lakoff, More than Cool Reason. Move them down on your list. Put McCloud first. Why? Because cognitivism is in fact about building things, about how the mind builds perceptual and conceptual structures. McCloud is about how comics are built. And, in one way or another, the other books give you a sense of construction as well. The Braitenberg constructs a mind, mechanism by mechanism. There's NO sense of mechanism in Turner and Lakoff. See also Cognitivism and the Critic 2: Symbol Processing.

* * * * *

It has long been obvious to me that the cognitive sciences are what happened when the computation and the computer hit the behavioral sciences as a source of models and metaphors. And that is what is missing from almost all of the work I’ve seen in cognitive approaches to literature. In this post I list and annotate four modest books that can help restore the sense of computation, and the constructive, that’s otherwise absent. I list them in order of suggested reading, starting with a comic book about comic books. After that we have a bonus section, a parable about computation based on passages from Simon about a drunken ant walking on the beach.

(1) Scott McCloud (1993). Understanding Comics. New York: HarperPerennial.

In some odd, but wonderful, ways this may be the best single introduction to the cognitive study of literature. It's not an academic book; there's no scholarly apparatus. But it yields a superb sense of what it is like to think about story-telling from a cognitive point of view. It takes the form of a comic book, words and images in panels cover every page - McCloud is a cartoonist. The pictorial form is what makes it so effective. So, McCloud has the reader thinking about visual objects and how they're constructed and how those constructions are organized into stories. It conveys a sense of design, engineering, and construction which is very important and which is missing in much of the current literary cognition literature. It gives the reader a whiff of mechanism without the pain involved in understanding the computational models of the cognitive sciences.

(2) Valentine Braitenberg (1999). Vehicles: Experiments in Synthetic Psychology. Cambridge, MA: MIT Press.

This is a cumulative series of thought experiments, 14 of them in the first 83 pages. Braitenberg asks us to imagine a simple (artificial) creature in a simple environment. Here’s how he begins to describe the first one: “Vehicle 1 is equipped with one sensor and one motor. The connection is a very simple one. The more there is of the quality to which the sensor is tuned, the faster the motor goes” (p. 3). He then works out the consequences of this very simple creature, how it moves about. In the second chapter he gives the vehicle two sensors and two motors and from that constructs primitive fear and aggression. And so it goes for the rest of these 14 chapters. In each chapter he adds a little bit to the vehicle from the previous chapter and explores the behavior consequences, e.g.: love (vehicle 3), concepts (#7), getting ideas (#10), egotism and optimism (#14). The last 50 pages contain biological notes on the vehicles, thus relating to the real nervous systems of real animals. Like the McCloud, it conveys a sense of design, engineering, and construction that is essential to the cognitive science.

(3) Herbert A Simon (1981). The Sciences of the Artificial, Second Edition. Cambridge, MA: MIT Press.

Trained in political science, Simon became one of the founding fathers of artificial intelligence, computing, and cognitive science during the 1950s and 60s. This is a relatively informal collection of essays that has been widely, and justly, influential. From the preface (xi): “Engineering, medicine, business, architecture, and painting are concerned, not with how things are but with how they might be—in short, with design. . . . These essays then attempt to explain how a science of the artificial is possible and to illustrate its nature. I have taken as my main examples the fields of economics (chapter 2), the psychology of cognition (chapters 3 and 4), and planning and engineering design (chapters 5 and 6).” Chapter 7 is entitled “The Architecture of Complexity” (originally published in 1962) and takes up the problem of biological evolution. The bonus section of this post is based on a thought experiment or parable from chapter 3, “The Psychology of Thinking: Embedding Artifice in Nature.”

(4) John von Neumann (1958). The Computer and the Brain. New Haven: Yale University Press.

Von Neumann was an mathematician who made contributions in many fields. But he is best known for his work in computing. This slender volume (82 pages) is the last project he worked on and is incomplete. Brain cancer took him before he could finish. It is about two ways a computing process can be embodied in physical matter, the analog and the digital, and addresses, among other things, the limitations these modes impose on the process. Though the book contains no math, it is quite abstract, its details at some remove from all the complex details about existing computers (then and now) or the messy wetware of the brain. That is to say, it is about the essential. Forget about the fact that computers are now quite different from those von Neumann knew, and forget about the fact that most of what we know about the brain was discovered since von Neumann’s death. In this book first class mind grappls with deep questions in simple, if abstract, terms. Reading it is a good work out.

Bonus: Simon’s Ant and Slocum’s Pilot

Think of this as a parable about computation, about how computational requirements depend on the problem to be solved. Stated that way, it is an obvious truism. But Simon’s thought experiment invites you to consider this truism where the “problem to be solved” is an environment external to the computer – it is thus reminiscent of Braitenberg’s primitive vehicles.

 

Think of it like this: the nervous system requires environmental support if it is to maintain its physical stability and coherence. Note that Simon was not at all interested in the physical requirements of the nervous system. Rather, he was interested in suggesting that we can get complex behavior from relatively simple devices, and simplicity translates into design requirements for a nervous system.

Simon asks us to imagine an ant moving about on a beach:

We watch an ant make his laborious way across a wind- and wave-molded beach. he moves ahead, angles to the right to ease his climb up a steep dunelet, detours around a pebble, stops for a moment to exchange information with a compatriot. Thus he makes his weaving, halting way back to his home. So as not to anthropomorphize about his purposes, I sketch the path on a piece of paper. It is a sequence of irregular, angular segments--not quite a random walk, for it has an underlying sense of direction, of aiming toward a goal.

After introducing a friend, to whom he shows the sketch and to whom he addresses a series of unanswered questions about the sketched path, Simon goes on to observe:

Viewed as a geometric figure, the ant’s path is irregular, complex, hard to describe. But its complexity is really a complexity in the surface of the beach, not a complexity in the ant. On that same beach another small creature with a home at the same place as the ant might well follow a very similar path.

I want to consider a variation on his parable. What would happen if we put the ant on an absolutely featureless surface and let it walk about? What kind of paths would it trace then? As that surface lacks any of the normal cues in the ant’s environment I would imagine the ant would either not move at all or move in a genuinely random or perhaps a rigidly stereotypic way (e.g. around and around in a circle). Or perhaps the ant would hallucinate.

Simon, of course, was not particularly interested in ants. He only told that story to make a point he wanted to apply to the human case. So let us continue on and consider the human case. What happens to us when we face a blank world, a world that does not support our intentionality? What happens is that the mind becomes unstable.

Early on in The Ghost Dance, a classic anthropological study of the origins of religion, Weston La Barre considers what happens under various conditions of deprivation. Consider this passage about Captain Joshua Slocum, who sailed around the world alone at the turn of the 20th Century:

Once in a South Atlantic gale, he double-reefed his mainsail and left a whole jib instead of laying-to, then set the vessel on course and went below, because of a severe illness. Looking out, he suddenly saw a tall bearded man, he thought at first a pirate, take over the wheel. this man gently refused Slocum’s request to take down the sails and instead reassured the sick man he would pilot the boat safely through the storm. Next day Slocum found his boat ninety-three miles further along on a true course. That night the same red-capped and bearded man, who said he was the pilot of Columbus’ Pinta, came again in a dream and told Slocum he would reappear whenever needed.

La Barre goes on to cite similar experiences happening to other explorers and to people living in isolation, whether by choice, as in the case of religious meditation, or force, as in the case of prisoners being brainwashed.

Imagine, now, that when you step away from the world, you fill the void with a work of imaginative literature. How does that work support the operations of the nervous system, giving them resistance and structure so they don’t flail into chaos?

* * * * *

I just googled "simon's ant" and got over 5 million hits. I didn't check more than 15 or so, but 3 or 4 of those were references to Simon's little story. People seem to find it a very useful tool for thinking, as well they should.

Further reading (more posts)

Speculative Engineering – From the introduction to Beethoven's Anvil, about my intellectual method.

Style Matters: Intellectual Style – In which I argue, in effect, that there is a prose style of thought and a diagrams style of thought and that such things are deep in individuals, and that "some of the pigheadedness that often crops up in discussions about humanities vs. science is grounded in stylistic preference that gets rationalized as epistemological belief."