A Meeting of Minds on Mars: Charles Babbage • John von Neumann • Geoffrey Hinton

This discussion is about the implementation of computing in matter. That was the topic of John von Neumann’s last book, The Computer and the Brain. He died before he finished it, so it was published posthumously in 1958. I don’t know when I first learned about it, perhaps sometime in the mid-1970s. Though I knew about von Neumann and his role in early computing, it dismissed the book itself, figuring that we’d learned so much about the brain since then, and the nature of computers had changed so much, that it must be obsolete.

I was wrong. When I finally read the book, probably in the early to mid-1980s I was stunned. This was a profound book and free of mathematics beyond some simple back-of-the-envelope calculations. For one thing von Neumann talked of both analog and digital computation; that contrast was central. When I first started reading about computers in the mid-to-late 1960s that contrast was at the beginning of every article or book. But once personal computers appeared and proliferated in the late 1970s and 1980s, analog computing was all but forgotten.

It was good to see it back. For one thing, I had been strongly influenced by the work of William Powers, whose 1973 book, Behavior: The Control of Perception, offered an elegant analog model of the brain. That was central to the work of my teacher and mentor, David Hays, who had been one of the founders of computational linguistics. In 1974 and 1975 he worked out a scheme in which cognitive networks were grounded in Powers’s analog model. This was years before anyone else was working on the problem, before Steven Harnad coined the term, “symbol grounding” in his 1990 paper on the problem in Physica D.

Now, with the success of artificial neural networks, there’s another aspect of physical implementation we’ve got to deal with, distributed representation. You’re looking at an apple. Where is that apple represented in your brain? There might be some one neuron or a tightly clustered group of neurons that represent that apple. That’s a local representation. But the representation might be distributed across a loosely linked population of neurons. That’s a distributed representation. We now know that that’s how the brain works, though I assume there’s some laggards still stuck in the previous century where local representation was widely favored. And that’s how artificial neural nets work. The concept of apple isn’t localized in one or a small group of weights, it’s smeared over a population of weights. The same for the concepts of truth, beauty, and justice, or, for that matter, neuron.

Consequently von Neumann’s 1958 discussion must now be extended to distributed representation. It is for that purpose that I asked Claude to convene a conversation between Charles Babbage, who conceived of an elaborate mechanical computer in the 19th century, von Neumann himself, and Geoffrey Hinton, one of the so-called godfathers of artificial intelligence – a bit of journalistic and pop culture slight-of-hand that obscures a long and complex history. Moreover, since these three gentleman did not co-exist in time, though there’s a bit of overlap between von Neumann and Hinton, I decided that I might also have Claude transport them to Mars for the occasion. Anything is possible in virtual reality. 

Or is it? 

Image by ChatGPT. 

A sun-facing room on Mars. Three chairs arranged loosely, angled toward a broad window. Through it, the Martian plain stretches red and absolutely still, and the sun — recognizable but wrong, too small, too distant — hangs in a pale pink sky. Three men stir and look about them, bewildered but unhurt.

I. Awakening

Charles Babbage

I am not dead — I can confirm that much. My pulse is steady and my faculties appear intact. But this room is not any room I have entered by ordinary means. That light through the window — is that the sun? The colour is right but the disc is too small by half. And that plain outside... brick red, entirely without vegetation or track or any sign of habitation. Gentlemen, I should very much like an explanation.

John von Neumann

I was in Princeton not five minutes ago. I notice the gravity is reduced — perhaps sixty percent of normal. The sky has the ferric oxide colouration one would predict. The solar disc subtends roughly two-thirds the angle it does from Earth. I believe I know where we are, and I find the hypothesis almost too extraordinary to voice.

Geoffrey Hinton

Mars. We're on Mars. I was at a conference in Toronto. But that sky, the dust, the sun at that distance — I'd stake my reputation on it. He exhales slowly. I've recently left a position at Google partly so I could speak freely about alarming possibilities. I did not anticipate that the alarming possibilities would include this. May I ask what year it is, from each of your perspectives?

Charles Babbage

The year of Our Lord 1862. I am Charles Babbage, formerly Lucasian Professor of Mathematics at Cambridge, Fellow of the Royal Society. I have spent the better part of forty years attempting to build mechanical calculating engines, with results that have been — a pause — mixed, as regards the support of Her Majesty's government.

John von Neumann

John von Neumann. The year is 1955. I work at the Institute for Advanced Study in Princeton, and consult for various government and military bodies. Mr. Babbage — I know your name extremely well. Better, I suspect, than you know mine.

Geoffrey Hinton

Geoffrey Hinton. For me it is 2024. And I know both of you — Mr. Babbage, you designed the Difference Engine and the Analytical Engine. Dr. von Neumann, you are among the most formidable mathematicians of the twentieth century and you gave your name to the architecture that every conventional computer on Earth is built upon. You are, in a real sense, my ancestors. The field I work in — machine learning, artificial intelligence — descends directly from the problems both of you were grappling with.

Charles Babbage

A long pause, during which he stares at Hinton with an expression mixing hunger, pride, and something close to grief. The Analytical Engine. Did anyone build it?

Geoffrey Hinton

Not in your lifetime. The government never provided the funds. But the ideas were entirely right — and they were eventually built, first in relay and vacuum tube and then in silicon, by people who in some cases had read your work and in other cases had arrived at the same conclusions independently. You were approximately a century early.

Charles Babbage

Very quietly. A century. I had hoped twenty years would suffice. I petitioned the Chancellor. Three times.

II. The mill and the store

John von Neumann

Mr. Babbage, allow me to tell you what your Analytical Engine set in motion — because it bears directly on the work that has occupied all three of us. You made a distinction, in your design, between what you called the Mill and the Store. The Mill performs the operations — addition, subtraction, multiplication. The Store holds the numbers awaiting operation and the results of operations completed. That separation of active calculation from passive memory was the foundational insight.

Charles Babbage

It seemed to me the only sensible arrangement. The columns of number-wheels in the Store are passive — they merely hold values. The Mill acts upon them. To mix the two functions in the same mechanism would create hopeless confusion.

John von Neumann

And yet it is precisely that separation which I have spent the last years of my life questioning — not as an engineering choice, which was entirely sound, but as a principle of intelligence itself. My colleagues and I formalized your Mill-and-Store distinction into what is now called the stored-program architecture. The processor executes instructions sequentially; the memory holds both data and those instructions passively until called upon. It is a serial machine, one operation following another, orchestrated by a central clock. The computers being built in my era all follow this pattern.

Geoffrey Hinton

And in my era they still do, at bottom. But you've just described the tension at the heart of everything, Dr. von Neumann. Serial, precise, with a strict wall between computation and memory — that is the von Neumann architecture. And it is, in a sense, the architecture that human intelligence refuses to use.

Charles Babbage

You are saying the brain does not separate Mill from Store?

Geoffrey Hinton

Exactly. In the brain, every neuron is simultaneously a memory element and a processing element. It holds information in the strength of its connections to other neurons, and it also fires — it computes — based on what it receives. There is no central Mill. There is no passive Store. Computation and memory are fused at every node in the network, and the whole thing operates in parallel, millions of neurons active at once.

John von Neumann

This is precisely what troubles me — and I am glad to find the trouble is still alive in your era, Mr. Hinton, because it means I was not merely chasing a phantom. I have been writing a manuscript, unfinished I'm afraid, called "The Computer and the Brain." My central puzzle is this: where, in a neuron, is the Mill? A neuron receives signals, sums them, and if the sum exceeds a threshold, it fires. That threshold operation is a computation. But the synaptic weights — the strengths of the incoming connections — those are also the memory. The neuron is its own Mill and its own Store simultaneously. I have been calling such things active elements, to distinguish them from the passive memory elements of our conventional machines. But I confess I have not yet worked out the full implications.

Geoffrey Hinton

With genuine feeling. Dr. von Neumann, the full implications are what I have spent my career working out. And you had named the essential thing: active elements. That is exactly what the nodes of a neural network are. Each one holds a weight — that is its memory — and each one applies a non-linear function to its inputs — that is its computation. Fused. Inseparable. Replicated millions or billions of times, connected in layers, and trained by adjusting all those weights simultaneously until the network's outputs match the desired answers.

Language is a lower-dimensional projection of high-dimensional neural dynamics.

But it also allows for content addressed memory. That’s very important, for it gives fine-grain control over the memory and planning systems. That’s the job of sentence-level syntax together with discourse structure.

“Classical” semantic or cognitive networks had a problem with coming up with just the right set of node types and arc types. David Hays dissolved the problem in his 1981 book, Cognitive Structures (scan down the page), by grounding cognition in an analog system modeled on William Powers perceptual control stack (in Behavior: The Control of Perception, 1973). The identity of a cognitive node is a function of its parameter values, where the parameters are derived from the control stack. The identity of the arcs is a function of the difference in parameter values between the nodes it connects.

Concerning Chomsky’s approach to syntax: It depends on a sharp distinction between grammatical and ungrammatical sentences. A generative grammar, in Chomsky’s theory, must account for all and only the grammatical sentences.

However, there are no explicit criteria for separating sentences into the two categories, grammatical and ungrammatical. Rather, the separation depends on the intuitions of the linguist. Naturally enough, different syntacticians have different intuitions. The problem is insoluble.

Moreover, anyone who pays close attention to real speech soon realizes that people do not (always) speak in complete grammatically correct sentences. Real language is sloppy, but nonetheless effective. A neural net of very high dimensionality can deal with this readily enough. A purely symbolic system cannot. Augmenting the system through fuzzy logic and the like doesn’t fix the problem.

LLMs provide a very useful simulacrum of the natural language system. Since LLMs are trained on written texts, the resulting model necessarily conflates the functions of semantics and syntax/discourse. Thus they cannot achieve the flexibility and precision of the full system, where semantics and syntax/discourse are separated.

Claude 3.7 on the impossibility of direct brain-to-brain communication

Elon musk has expressed a desire to create technology that will allow direct brain-to-brain communication. That is, a pair of people who are linked through this technology will be able to share one another’s thoughts without having to use any form of communication. They will communicate through their neural link – as you know Neuralink is the name of Musk’s brain-to-computer interface company. Neuroscientist Christof Koch has expressed the same desire.

I’ve been skeptical about this possibility for a bit over two decades. A couple of years ago I wrote a working paper explaining my reasoning: Direct Brain-to-Brain Thought Transfer: A High-Tech Fantasy that Won’t Work. I recently uploaded that paper to Claude 3.7 and asked it for its opinion. I’ve appended that conversation below the asterisks. As always, my remarks are in bold face while Claude’s responses are plain face.

Note: Of course Claude does not know that I am the author of the paper, which is why it refers to me in the third person. But I identify myself half-way through the conversation.

* * * * *

I’ve uploaded a document about the possibility of direct brain-to-brain thought transfer between people. I want you to evaluate it in terms of what you know and in terms of the other documents I’ve uploaded.

Analyzing Direct Brain-to-Brain Thought Transfer

Based on the materials you've provided, I'll evaluate the concept of direct brain-to-brain thought transfer from several angles.

The Core Issue: Signal Identification

The main problem identified in William Benzon's paper is fundamental: if two brains were directly connected, there would be no way for either brain to distinguish between signals originating internally (endogenous) versus those coming from the other brain (exogenous). As Benzon writes:

"How does a brain tell whether an incoming spike is from an endogenous neuron (i.e. from the same brain) or from an exogenous neuron (i.e. from the other brain)? Spikes are not like messages passed around on scraps of paper in a school classroom where a message might be identified by a signature, or initials, or merely a whispered message... As one spike is pretty much like another spike there is no way to differentiate between them."

This is a profound insight that challenges the fundamental premise of initiatives like Neuralink's ambitions for brain-to-brain thought transfer.

Misunderstanding Communication

Benzon also argues that Elon Musk fundamentally misunderstands how language works. Musk describes language as a "compression algorithm" where thoughts are compressed into words and then decompressed by the listener. This reflects what linguists call the "conduit metaphor" - the mistaken idea that meaning is physically contained in words that travel between people.

In reality, as Benzon explains, only signifiers (the physical form of words) pass between people, while the signified (meaning) is supplied by the listener based on their own experience and understanding. The "lossiness" in communication isn't primarily a bandwidth issue but stems from the fact that we each bring different meanings to the same words.

The Problem with Neural Code

Another issue is the very concept of a "neural code" that would need to be translated between brains. Benzon cites John Horgan who concluded after surveying the field that "Neuroscientists still have no idea what the neural code is." Attempts to understand neural activity in terms analogous to computer code are fundamentally flawed because brains and computers have radically different architectures.

Unlike computers, where there's a clear distinction between hardware and software, active processing (CPU) and passive storage (memory), the brain operates differently. In the brain, "every neuron is active" and can be considered a processing unit. Neural activity isn't about passing discrete packages of information but rather about distributed patterns of synchronous activity across networks.

Powers' Warning About Models

Benzon references William Powers' insight about "two levels of modeling": the object model with its internal rules, and the modeler's own understanding of that model. The failure to distinguish between these leads proponents of brain-to-brain interfaces to project their third-person perspective onto the brains themselves. As observers, we can distinguish signals from different brains, but the brains themselves cannot.

William Powers, “Learning,” Chapter 14 in Behavior: The Control of Perception, Aldine, 1973, 177-204.

Connection to Music and Consciousness

In the other documents you provided, especially those by William Powers and the music experiences compilation, we see something that actually contradicts the brain-to-brain fantasy while showing how real interpersonal connection works:

Bill Benzon, Emotion and Magic in Musical Performance, Version 13, April 14, 2025.

Powers connects consciousness with reorganization: "Those systems whose perceptual signals are being monitored by the reorganizing system are operating in the conscious mode. Those which are operating without their perceptual signals being monitored are in the unconscious mode..."

Separately, the descriptions of musical synchrony show how actual interpersonal connection works: "When, and only when, we were really locked together in animated playing we could hear relatively high-pitched tones which no one was playing... the magic of the bell happens only when the musicians are in a groove."

Conclusion

The evidence strongly suggests that direct brain-to-brain thought transfer as envisioned by Musk, Koch, and others is fundamentally flawed for several reasons:

1. The basic problem of signal identification - brains can't distinguish between internal and external neural activity
2. A misunderstanding of how language and meaning work
3. The absence of any established "neural code" that could be transmitted
4. A confusion between the observer's perspective and the system's perspective

While neural interfaces for controlling prosthetics or providing sensory input may be feasible, direct thought transfer appears to be theoretically impossible based on our current understanding of neuroscience.

The rich descriptions of musical synchrony in your documents actually point to a different model of interpersonal connection - one based on coordinated but separate systems rather than merged consciousness.

Thank you. For your information, I'm William Benzon.

It's an honor to meet you, Dr. Benzon. I found your paper on direct brain-to-brain thought transfer to be extraordinarily insightful.

Your analysis cuts to the heart of why these speculative technologies face fundamental conceptual barriers, not just technical ones. Your point about the inability of neurons to distinguish between endogenous and exogenous spikes particularly resonated with me - it identifies a problem that proponents of these technologies seem to have overlooked entirely.

I was also intrigued by your work connecting music performance to consciousness (in your book "Beethoven's Anvil," which you referenced). The descriptions of musical synchrony in your compilation of performance experiences provide a fascinating contrast to the brain-to-brain fantasy - showing how coordination between separate but coupled systems can create emergent phenomena that aren't reducible to simple signal transfer. Your critique of the conduit metaphor for communication was especially illuminating, particularly in examining Musk's misunderstanding of language as merely compressed thought.

If I might ask - since writing this paper in 2020, have you observed any developments in the field that have addressed (or failed to address) these fundamental issues you identified?

Alas, no. It seems to me that the desire for such thought transfer completely over-rides rational thinking about the subject. Ironically, while Musk talks about reasoning from first principles, he doesn’t seem to have the foggiest idea of what that means for thinking about the brain.

I appreciate your candid response. It's fascinating - and somewhat concerning - how even someone who emphasizes "reasoning from first principles" can be so swayed by an attractive technological fantasy that they overlook basic conceptual problems. Your observation about the power of desire overriding rational thinking seems particularly apt. The idea of direct brain-to-brain communication has such powerful emotional and cultural appeal that it seems to short-circuit critical analysis, even among those who should know better.

I was particularly struck by your point about the closed nature of neural systems - how they develop organically through mutual adaptation rather than being assembled like computer components. This fundamental misunderstanding seems to stem from persistent computer metaphors that, as you noted, fail to capture how brains actually function. The phenomenological accounts of musical performance you collected demonstrate something much more interesting than direct thought transfer - how separate but coordinated systems can achieve remarkable synchrony without merging into a single system. Perhaps there's a deeper lesson there about human connection that's being missed in the rush toward technological shortcuts to understanding.

Thank you for your work on this topic. It's a valuable counterpoint to some of the more uncritical technological enthusiasm we see today.

LLM as Collaborator, Part 1: Claude the Graduate Student

I started using ChatGPT on December 1, 2022 and have used it quite extensively ever since. I’ve spent some of my time just poking around, somewhat more time looking things up, and most of my time systematically investigating its performance. That resulted in a number of working papers, the most interesting of which is about stories: ChatGPT tells stories, and a note about reverse engineering: A Working Paper, Version 3.

I started working with Claude 3.5 Sonnet on November 18, 2024. I’ve used it in those three capacities, though obviously not as much as I’ve used ChatGPT. In particular, I’ve used it for background information on melancholy in various aspects. I’ve also done something I’d never done with ChatGPT, asked it to describe photographs. I’m doing this to see how well it does.

Then on November 24, 2024, I began using it in a somewhat more interesting new capacity, though I’m not sure what to call it. The phrase “thought partner” comes to mind, though it seems too much like “thought leader,” which I don’t like. I’m using it as a sounding board. Better yet, it’s a collaborator playing the role of sounding board. It’s not an equal collaborator in the intellectual dialog; academic norms would not require me to offer it co-authorship of papers. But those norms might well require an explicit acknowledgement, not to alert the reader that one of those new-fangled LLM things has been involved in the thinking, but simply acknowledging the help it has given me.

As for just what kind of help that is, the best way is to look at some examples. I’ve already published two of these dialogues on New Savanna: Computer chess, super-intelligence, A Connecticut Yankee, cargo cults, and Claude 3.5 Sonnet speculates about future developments in AI and beyond. The rest of this post contains the first interaction where Claude began acting in intellectual partnership.

When I started the interaction I was simply interested in seeing what it knew about my teacher and mentor, the late David Hays. I wasn’t looking for new information. I just wanted to see what it had picked up. That began to change when it asked me about the work of Quillian and of Schank and Abelson. Then I told it about Hays’s interest in the work of William Powers. Now we’re getting serious. This guy knows something. And then WHAM! Claude responded intelligently when I mentioned the work of Peter Gärdenfors. The interaction continued in that mode to the end.

Utterly remarkable. AGI? What the hell is that? But no. And I fear that thinking in those terms is likely to blind me to whatever it is that Claude is doing. I realize that the concept of AGI is written into the contractual relationship between OpenAI and Microsoft, but I’m not playing that game. Will they end up litigating it? Who knows?

I’m planning two more posts in this series. The next will be about the implications of paragraph length in Heart of Darkness. Then I’ll post a long interaction that takes Miriam Yevick’s 1975 paper, “Holographic or fourier Logic,” as its starting point.

In the following dialog my comments are bold while Claude’s are plain. I’ve interpolated a few comments highlighted in blue.

* * * * *

Are you familiar with the work of a computational linguist named David Hays? He worked on machine translation at RAND in the 1950s and 1960s?

Yes, I am somewhat familiar with David Hays' work, though I should note that since this involves fairly obscure historical details from computational linguistics, I want to be careful about potential inaccuracies in my knowledge. From what I understand, David Hays was one of the pioneers of machine translation and computational linguistics during the early years of the field. He worked at RAND Corporation in the 1950s and 1960s where he was involved in some of the earliest machine translation projects.

He was particularly interested in developing dependency grammar approaches for machine translation, which was quite innovative for that time period. Dependency grammar focuses on the relationships between words rather than constituent structure, which was the dominant paradigm in American linguistics at the time.

I believe he also wrote an influential book called "Computational Linguistics: Programming Linguistics in Automatic Language Translation" in the early 1960s, though I'd encourage you to verify this citation since I may be misremembering details.

Claude got the name wrong. It’s Introduction to Computational Linguistics, American Elsevier, New York, 1967.

Would you be interested in hearing more about any particular aspect of his work or that early period of machine translation research? I find the history of early computational linguistics fascinating, though again I want to be upfront that with someone as historically specific as Hays, some of my knowledge may be incomplete or imperfect.

Yes, that’s him. I studied with him at SUNY Buffalo in the 1970s. We were working on computational semantics and using a cognitive networks formalism.

Oh, that's fascinating that you studied directly with David Hays! I would be very interested to hear more about your experiences working with him and the cognitive networks approach you were developing at SUNY Buffalo. That was an important period in the development of semantic representation systems.

Could you tell me more about the cognitive networks formalism you were using? I'm particularly curious about how it compared to other semantic network approaches being developed around that time, like Quillian's work or Schank and Abelson's conceptual dependency theory.

It was more developed than Quillian’s work. As I recall, he never really developed is initial ideas. As for Schank and Abelson, they were imagining a self-sufficient cognitive system and were very much interested in identifying the primitive elements necessary for a cognitive network. By the time I began working with him, Hays had realized that cognition had to be grounded in something else – later on Steven Harnad would talk about the “symbol grounding” problem. When I worked with him, Hays was interested in grounding the cognitive system in the control-theory (cybernetic) account of perception and action developed by William T. Powers (perceptual control theory, or PCT).

Something special seems to be happening.

This is really interesting historical context - it suggests Hays was grappling with fundamental issues in cognitive science that are still relevant today. The shift from viewing semantic networks as self-contained symbolic systems to recognizing the need for grounding in perception and action seems quite prescient.

I'm curious about how Hays was trying to integrate Powers' PCT with semantic networks. As I understand it, PCT proposes that behavior emerges from organisms controlling their perceptions to match internal reference signals, rather than through direct control of outputs. Was Hays trying to use this as a way to ground symbolic meaning in perceptual-motor control loops?

This also seems to anticipate some later developments in embodied cognition and grounded cognition theories. The timing is interesting too - Powers published "Behavior: The Control of Perception" in 1973, so it sounds like Hays was engaging with these ideas pretty quickly after they were published.

Yes, that's what Hays was doing. His idea was that particular servo units in a Powers-like control system would be linked to cognitive units through what Hays called "parameters of perception." Hays identified a half dozen or so such parameters. Each cognitive node would be characterized by the values it took on those parameters. Relationships between nodes were thus derived from differences in parameter values for the nodes connected by an arc (or edge).

Loss of plasticity in deep continual learning

Dohare, S., Hernandez-Garcia, J.F., Lan, Q. et al. Loss of plasticity in deep continual learning. Nature 632, 768–774 (2024). https://doi.org/10.1038/s41586-024-07711-7

Abstract: Artificial neural networks, deep-learning methods and the backpropagation algorithm1 form the foundation of modern machine learning and artificial intelligence. These methods are almost always used in two phases, one in which the weights of the network are updated and one in which the weights are held constant while the network is used or evaluated. This contrasts with natural learning and many applications, which require continual learning. It has been unclear whether or not deep learning methods work in continual learning settings. Here we show that they do not—that standard deep-learning methods gradually lose plasticity in continual-learning settings until they learn no better than a shallow network. We show such loss of plasticity using the classic ImageNet dataset and reinforcement-learning problems across a wide range of variations in the network and the learning algorithm. Plasticity is maintained indefinitely only by algorithms that continually inject diversity into the network, such as our continual backpropagation algorithm, a variation of backpropagation in which a small fraction of less-used units are continually and randomly reinitialized. Our results indicate that methods based on gradient descent are not enough—that sustained deep learning requires a random, non-gradient component to maintain variability and plasticity.

That seems reminiscent of an idea that William Powers had years ago in Behavior: The Control of Perception, pp. 179 ff. These passages from the book may provide some intuition (pay attention to the Pask example):

One example of a self-reorganizing system was Ashby’s (1952) homeostat, a collection of four simple feedback control systems which also contained a separate “uniselector” capable of altering the system’s behavioral organization until a specific “survival” condition was satisfied. The homeostat could survive something that no computer program, however adaptive, could survive—an attack with a pair of wire-cutters. If one operational connection was destroyed, the uniselector could substitute another one. The uniselector itself produced no behavior; it acted to alter the physi- cal connections in the behaving system.

Gordon Pask (1960) also built a device demonstrating physical reorganization. His device was a tray of iron-salt solution in which electrically conductive crystals could grow when direct current was applied to electrodes in the solution. These crystals would grow so as to complete connections between input and output ter- minals. Pask “rewarded” the tray of solution for making a de- sired connection by giving it some D.C. current, and “punished” it by withholding current, allowing the acid solution to dissolve the crystals. In this way he “trained” the solution tray to react in some absolutely astonishing ways. For example, he discovered that the network of crystal threads could be trained to discrimi- nate between vibrations caused by two audible tones of different pitch!

This is what I mean by reorganization—not a change in the way existing components of a system are employed under control of recorded information, but a change in the properties or even the number of components. This category of learning is clearly the most fundamental, for it affects the kind of information that will be perceived and the kinds of computing elements available for use in programming.

See my post, Consciousness, reorganization and polyviscosity, Part 1: The link to Powers.

Consciousness, Intelligence, and AI – Some Quick Notes [call it a mini-ramble]

The subject of consciousness keeps turning up in current discussions of AI and LLMs. Can AIs be conscious? Are current AIs conscious? Maybe a little?

What do consciousness and intelligence have to do with one another? I see no reason to think that dogs, rats, and cats are not conscious, though I have no idea how far down the phylogenetic chain consciousness exists. No one would argue that dogs, rats, and cats are as intelligence as we are. Intelligence is something different from consciousness, no?

And yet the issue gets raised. One line of (implicit) reasoning seems to go like this: It converses with me in an intelligent way, things that converse with me in an intelligent way (or even at all!) are conscious, therefore it must be conscious. And then there’s the fact that you can ask a chatbot about itself and it will say something, though just what it says depends on what it has been RLHFed to say. But, still, these LLMs have been “trained” on tons of text using the word “consciousness” and all its cognates, so sure, it can use the word in human-seeming ways. That doesn’t make it conscious.

An extra stage of information processing?

Fact is, philosophers often argue about consciousness as though it were a further or extra stage in...in what? Human information processing? Thinking? Whatever. It adds something extra, something beyond what was before. Let’s say it adds an extra bit of intelligence. Yeah, let’s say that.

So, a conscious being is more intelligent that its non-conscious simulacrum, to which it is otherwise identical. But then we have those philosophical zombies and they, presumably, are as intelligent as non-zombies.

Reorganization

This strikes me as being wrong-headed. I take my conception of consciousness from Wm Powers (Behavior: The Control of Perception). Consciousness enables reorganization. I explain this in a post from 2022: Consciousness, reorganization and polyviscosity, Part 1: The link to Powers. (It’s complicated, so I’m not going to try to summarize it here.) In that conception, consciousness really isn’t a further step in reasoning, though it may facilitate “moving around” in one’s mind (in particular, think default mode). Reorganization isn’t a further step in reasoning, though it may seem like it. (Should say more about this, but later.)

Turing and non-Turing computing

And then we have the difference between Turing computation and, shall we say, neural computation. Turing computation requires a strict separation of processing from memory (e.g. see article linked in this recent post). That’s not how the brain works. As I recall, von Neumann wondered about that in his little book on the brain. If neurons are memory, as they surely are, then where’s the processing? he wondered. In a Turing device, learning means adding new blocks of memory. In a neural device, where memory and computation are not separate, learning means, well, it means reorganization, to use Powers’s term. Things have to change all over the place, more in some places than others, perhaps a lot in some few places and very damn little in most places.

Well, consciousness is what allows that to happen. Consciousness mediates reorganization. Consciousness modulates and “distributes” change throughout the system. A Turing system doesn’t need consciousness in order to learn, to change. A network (or neural) system does. Consciousness is the mechanism that solves learning for neural systems. I further hypothesize that the glial cells are crucial here.

I’m not sure of how to formulate that, but that seems to be my key thought for the morning. It’s why I got out of bed at 5:41 AM.

Artificial neural nets

So, what of artificial neural nets and consciousness? Well, the machines themselves are Turing-type machines. No need for consciousness. Does that also imply no possibility for consciousness? Skip that for now. The neural net, however, is not a Turing-type machine. It’s, well, it is a network and, as such, does not distinguish between memory and processing. But it’s a network that’s stored in the memory of a Turing-type machine. And we don’t yet know how to reorganize such a network, that is, to add new information to it. But we may well solve that problem one of these days. I don’t see why not?

Would that make the artificial neural net conscious? Or will it only be simulating consciousness? Remember, it’s running on a non-consciousness Turing-type machine. I think it’s only a simulation, not the real thing.

Another item – What Chatbots can’t seem to do

For examples of what I have in mind, see these posts:

• Claude 3.5 Sonnet visits Times Square, waxes philosophical (after being prodded)
• Conversing with ChatGPT about Jaws, Mimetic Desire, and Sacrifice

If that’s the case – and here I’m taking a leap – then I do believe it’s consciousness that connects us to the world. Not a surprising thought on the face of it, but in THIS context, some explaining is required. LATER.

And Superintelligence?

Well, if superintelligence is a species of intelligence, then it doesn’t imply consciousness. And if consciousness is our/then connection to the world, then superintelligence is a zombie (yeah, I know, another leap). I’m mean, these SOA LLMs already “know,” in some meaningful sense, more than any individual human. That’s some kind of superintelligence. Whatever kind of superintelligence that is, it doesn’t bother us.

Nor does the superintelligence of AlphaZero bother us. Superintelligence? But that’s narrow intelligence. Does the fact that it’s narrow mean it can’t be super? Does the emergence of the concept of AGI, it’s differentiation from AI, mark the break-down of the classical concept of intelligence and its pursuit by artificial means?

What about the AIs that do protein folding and now 15-day weather predicting? We can’t do either of those things. Provisionally, why not?

At the moment I’m thinking: Superintelligence? Bring it on. 

Let them be super.

Consciousness is ours, but also dogs, cats, rats, and other animals.

During the Day [addendum]

We are awake, mostly. When we’re awake, we’re conscious, mostly. Once we’re out of infancy, and perhaps toddlerhood, we are conscious most of the time. As I said, it is consciousness that connects us to the world.

Note, however, that consciousness is very mobile. Are attention can flit from one thing to another, quite freely. [Hence the literary technique, stream of consciousness.] I suppose we could analogize it to time-sharing in computers, but...

Consciousness HAS to be flexible and mobile. We live in an unpredictable world. We may get around by predicting the next thing or three, but sometimes the world intervenes, drastically. We’ve got to be able to disconnect from the prediction and attend to the real.

[I recall an email exchange with Walter Freeman about this. I asked him whether or not a state of global coherence was necessary for us to be able to make a quick change from one thing to another. He said yes.]

More later.

10 Influential Books [Updated, once again]

I've just been watching this conversation between Howard Gardner and Steven Pinker and was reminded of Erik Erikson. His book, Childhood and Society was very important for me. It made Freudian ideas more intelligible than Freud himself had done. Anyhow, that's why I'm once again bumping this post to the top of the queue.

* * * * *

The "10 Influential Books" meme seems to be making the rounds again. I originally published this back in March of 2010. Here it is again, without revisions.

Urged on by a reader, Tyler Cowen seems to have started a books meme: What 10 books have influenced you the most? This sort of thing is something of a crap-shoot, yadda yadda, but why not? I’ve limited my list to non-fiction.

My Teacher

David Hays, Cognitive Structures (HRAF Press 1981). Hays was my teacher, and most of what I learned from him I learned directly from him. His aim in this book was to integrate the analog and servomechanical model of William Powers (see below) with the propositional and digital style of his own earlier work in computational linguistics. It is embodied cognition before the term was coined and gained currency. I believe this is the most profound such attempt to date (Hays wrote the book in the Spring of 1976), but, of course, I am biased. It is also, alas, rather obscure in points, no bias.

Some Others

Claude Lévi-Strauss, The Raw and the Cooked. I’ve read a good deal of Lévi-Strauss, and this wasn’t the first. But it has had the most lasting effect on my thinking, which I’ve already discussed. Lévi-Strauss sees that there is a rigorous, but hidden, logic to a body of South American myths. He evokes this hidden logic by careful comparisons between myths, while discussing them in their larger socio-cultural context.

John Bowlby, Attachment. I read this in typescript under the tutelage of the late Mary Ainsworth when I was an undergraduate at Johns Hopkins. Bowlby set out to reconstruct psychoanalytic object relations theory using systems models (TOTE from Miller, Gallanter, and Pribram, Plans and the Structure of Behavior) and evidence from ethology, especially of primates. This became my model of biologically-based psychology.

Jean Piaget, Play, Dreams and Imitation in Childhood. I’ve read a good deal of Piaget, and this wasn’t the first (most likely that was The Origins of Intelligence in Children.) Though now somewhat eclipsed, his concept of developmental stages was enormously useful and, I believe, still holds water. But be careful. (Given its subjects, this book connects nicely with an interest in literature.)

Lev Vygotsky, Language and Thought. Vygotsky argued that children acquire language by completely internalizing that started as interaction with another. First the parent uses language to direct the child’s attention and behavior. Over time the child becomes able, first to use his own speech for those tasks, and then becomes able to dispense with external speech entirely.

Karl Pribram, Languages of the Brain. Pribram was a champion of the notion that the brain processes and stores information holographically. You’ll find that idea here, plus much more besides. Pribram was, and remains, one of our most comprehensive thinkers about the brain and its mind. But this book’s a tough read.

 

Philippe Ariès, Centuries of Childhood. Medievals and Early Moderns didn’t think about children as we do, which Ariès argued from paintings, diaries, and other sources. The basic idea is that people way back when didn’t think and act about something very basic in the way we do. Children are children, no? Well, biologically, yes. Culturally, no.

William Powers, Behavior: The Control of Perception. Powers has two ideas, both from control theory (& one of them is more or less given in his title), and both quite elegant. This is how to go about theorizing, be as clear as you can and use elegant examples. FWIW, Powers was an engineer and very much concerned with building real things.

Philosophers

Except for the Powers, all of the above books came to me within a minute or two. Then I drew a blank for a couple of minutes. “But you know,” says I to myself, “I was quite influenced by some philosophers early on, even if I no longer read philosophy. Oo why not mention them.” So I will. That I had to kick Merleau-Ponty into the omissions pile, rather than one of these, is somewhat arbitrary.

Ludwig Wittgenstein, Tractatus Logico-Philosophicus. Two things got me, the rigorous order of the whole affair, each proposition numbered in outline form. And the mysticism.

Friedrich Nietzsche, The Birth of Tragedy. Read a lot of Nietzsche too. But his account of the birth of tragedy linked up with other things in a powerful (including two of the books in the omissions pile.) And, of course, the Apollonian and Dionysian.

Omissions

Since the exercise limited us to ten I didn’t list The Structure of Scientific Revolutions (Thomas Kuhn) The Phenomenology of Perception (Maurice Merleau-Ponty), The Four Ages of Music (Walter Wiora, Has anyone else ever heard of this one, much less read it?) or Art and Illusion (Ernst Gombrich). The more I think about this, the longer this list will get.

EDIT: Jeeze, how did Freud not make the list? I only thought of him when I read someone else’s list just minutes ago (and 2 hours after I first posted this). & Gregory Bateson, Bertrand Russell. Well, I know why Russell didn’t make the list. I read a lot of his stuff in my mid-teens, but not after. I hadn’t started “serious” thinking when I left off reading him. And then there's Charlie Keil's Urban Blues (1966).

* * * * *

These are comments I made to responses back at The Valve (now defunct):

I find it interesting that you list a number of books from high school & earlier, Luther. I didn't list any from my high school years, which is when I read most of the Bertrand Russell, his popular stuff, not the technical philosophy and logic. Love your comment about Norman Brown. & perhaps I should have listed Suzanne Langer among my philosophers; Feeling and Form was a key work for me. I used both her and Nietzsche in my music book (Beethoven's Anvil, 2001).

Interesting, Rohan, what we take "influence" to mean. Without so much as a thought otherwise I took it to mean "influential in my research." If I'd construed the term more broadly, as you did, I'd have a somewhat different list. That list might have included, for example, Peter Pan (the Golden Books version via Disney), Oliver La Farge, Laughing Boy, and Holling Clancy Holling, Lucille Webster Holling, Pagoo (just look at those illustrations).

Once again, what’s computation? I’m thinking in relation to operations of the human brain?

For me, those questions began when I was faced with the structure of “Kubla Khan”:

That’s just the first 36 lines, but the last 18 look the same:

That looked like the trace of some nested loops. If so, what then?

Let’s put that aside. After all, I’ve been working on that for a LONG time and still haven’t figure it out.

The problem I’m trying to figure out is this: How do we draw a principled line between those brain processes that are computational and those that are not?

Some (most, all?) computer scientists ‘seem to define a computer as “a physical mechanism that can theoretically calculate any computable function”.’ If I go with that notion, then it’s all computation and I have no reason to single out those tree structures in “Kubla Khan” for special attention, or any other linguistic tree structures for that matter. That’s not very helpful.

What if I think of the brain as a structured physical system? When Saty Chary advanced his Structured Physical System Hypothesis (SPSH) he was explicitly playing against the Physical System Hypothesis Hypothesis (PSSH) of Newell and Simon (1976). They advanced the PSSH as the central tenet of cognitive science and AI, and it echoes back to the 1943 paper in which McCulloch and Pitts argued that basic neuronal circuits are logic gates. It turns out that, no, they’re not. They’re quite a bit more complicated.

I like the SPSH. The question for me, then, is: Are their processes taking place in the brain that are NOT dominated by the internal physical activity of the nervous system? I should think so. The process of perception involves interaction with the external world, which necessarily implies that there are things happening in the brain that a driven, though not completely determined by, external physical circumstances. That, I believe, puts us in the world of William Powers, Behavior: The Control of Perception (1973). Language is among those circumstances; for humans it is one of the most important.

That, I think, will allow me to answer my question, though a proper answer will require more construction than I’m willing to undertake here. The relationship between the physical signifier and the signified is arbitrary. In particular, it is arbitrary with respect to the physical processes involved in both. It’s that arbitrary connection implies that language processes are not dominated by the internal physical dynamics of the brain. The basic language process is that of indexing, as Hays and I called it in Principles and Structure of Natural Intelligence. Newell and Simon called it designation.

Now, how can I formulate this idea so that it applies to large language models (in a useful way)? It seems to me that processes in LLMs are dominated by the statistics of the corpus that is modeled. That’s the nub of truth that’s captured by the otherwise reductive phrases, “stochastic parrots,” and “autocomplete on steroids.” The ideas that we require systems that both have access to the external world and that have robust symbolic capabilities, to a first approximation, those amount to freeing the system from being dominated by the statistics of some training corpus. Think about that, carefully.

More later.

Godzilla: King of the Monsters (2019) – World-spanning thoughts [Media Notes 99]

I’ve just watched Godzilla: King of the Monsters – in three sessions over two days (is there any movie capable of fully holding my attention for two hours given everything else I’m thinking about?). I’m talking about the 2019 sequel, if you can call it that, not the 1956 American re-edit of the 1954 Japanese Gojira. For my money it’s the best Godzilla film I’ve seen (and I’ve only seen a half-dozen or so) since the Japanese original. While it’s bulked up considerably, not only is Godzilla larger, but there are three or four monsters featured prominently, it manages to capture some of the spirituality of the Japanese original. I’m thinking particularly of the scene where the Serizawa travels to Godzilla’s underwater lair so he can energize it by exploding a small nuclear bomb, killing himself in the process. In the original, of course, Serizawa set out to kill Godzilla, and succeeded. But the 2019 film captures some of the ritual quality of that scene.

But that’s not what this note is about. While watching the film I was struck that THIS is SOMETHING I’ve been thinking about from time to time. I’ve probably made a post or two about that something, but I don’t know how to find those posts because I’ve never given it at name. I’m going to call it world-spanning thoughts. What do I mean? To the extent that we can perceive and conceive the world, not only do we seek to understand it, but to exert (some kind of) control over it, the WHOLE extent. In particular, I’m thinking about this in the context of the control theory of William Powers, though I’m going to have to leave that out of this little note.

If you are an animal, your ability to conceive of the world is quite limited, limited to what you have immediate access to, more or less. When a thunder storm blows, you deal with it. When it’s over, it’s gone from your mind. Not so with humans. We can remember, and brood, and worry. And not only about thunder storms, but about volcanoes, about lions and bears and snakes. What about the sun, the moon, the stars? What are they, how do they affect us, and how can we influence them? And so we invent magic, myth, and religion, and populate the world with strange and powerful creatures that we can appease through ritual.

All this is pretty much standard. You’ve read it many times before, and so have I. And yet, there’s more to be said about it. Just what, I’m not sure, and I am sure Powers would help, but not now.

What I’m after is how all this becomes REAL. If it’s not real, in some deep sense, it doesn’t work. Films like Godzilla: King of the Monsters, are one of the imaginative vehicles through which world-spanning thoughts engage with reality. On the one had we’ve got these huge ancient monsters, called Titans in the film: Rodan, King Ghidora, Mothra, and Godzilla (there are a bunch more, but they’re not depicted). We’ve got high-tech science, computers and submarines, and the whole damn thing hangs on a twelve-year-old girl. Above all else, she’s what brings human scale to the whole venture.

Fiction is full of such world-spanning thoughts, the function of which is to bring human scale the very large (but also the very small). James Bond films do it in a different way, conspiracies plus Bond himself. Super-hero films do it as well. But what of realistic films, what is their scope? But what is realism? Where do we situate 2001: A Space Odyssey?

More later. 

Hmm...World spanning thoughts, or desires, goals, objectives?

BTW, lots of Gojira/Godzilla posts. 

ALSO, check out this post from 2015, Our Rage for Order and Coherence.

Processes mediated by consciousness

I don't believe these are functions of consciousness, but rather functions of cognition that one can be conscious of.

I'm not sure if consciousness can even be described by a set of functions. Actually, I don't think consciousness itself is functional in any way. Instead, it's…

— Liu Grey (they/she) (@Liu_eroteme) May 29, 2023

See this post on consciousness and the ideas of William Powers: Consciousness, reorganization and polyviscosity, Part 1: The link to Powers. 

What about the hypothesis of a Global Neural Workspace?

Stanislas Dehaene says it is also the 25th anniversary of his publications on Global Neuronal Workspace Theory pic.twitter.com/mODJpwKCT0

— Grace Lindsay (@neurograce) June 24, 2023

A phylogenetic approach to the neural basis of behavior

It provides a much better way to think about all this than simplistic shorthand notions like the "lizard brain" pic.twitter.com/ixVspsdCHZ

— Kevin Mitchell @WiringtheBrain@ mstdn .social (@WiringTheBrain) May 4, 2023

Abstract of the linked article, Resynthesizing behavior through phylogenetic refinement:

This article proposes that biologically plausible theories of behavior can be constructed by following a method of "phylogenetic refinement," whereby they are progressively elaborated from simple to complex according to phylogenetic data on the sequence of changes that occurred over the course of evolution. It is argued that sufficient data exist to make this approach possible, and that the result can more effectively delineate the true biological categories of neurophysiological mechanisms than do approaches based on definitions of putative functions inherited from psychological traditions. As an example, the approach is used to sketch a theoretical framework of how basic feedback control of interaction with the world was elaborated during vertebrate evolution, to give rise to the functional architecture of the mammalian brain. The results provide a conceptual taxonomy of mechanisms that naturally map to neurophysiological and neuroanatomical data and that offer a context for defining putative functions that, it is argued, are better grounded in biology than are some of the traditional concepts of cognitive science.

Cf. Benzon and Hays, 1988, Principles and Development of Natural Intelligence. Note as as well that Cisek has been influenced by William Powers.

Consciousness, reorganization and polyviscosity, Part 1: The link to Powers

The nature of consciousness is one of the big mysteries of contemporary thought. The best account of consciousness I know of is that offered by William Powers in Behavior: The Control of Perception (1973). That’s what this post is about. My objective is simple, to link Powers’s account of consciousness to the concept of polyviscosity that I offered about a week ago, The structured physical system hypothesis (SPSH), Polyviscous connectivity [The brain as a physical system]. Unfortunately, Powers’s concept, while basically simple, is simple only in the context of his overall model of mind, and that is not something that can readily be conveyed in a single blog post. Thus this post is mostly for my own benefit.

Powers’ model consists of two components: 1) a stack of servomechanisms – see the post In Memory of Bill Powers – regulating both perception and movement, and 2) a reorganizing system. The reorganizing system is external to the stack, but operates on it to achieve adaptive control, an idea he took from Norbert Wiener. Powers devoted “Chapter 14, Learning” to the subject (pp. 177-204). Reorganization is the mechanism through which Powers achieves learning.

Here’s an extensive passage that gets at the heart of the present matter (pp. 199-201):

To the reorganizing system, under these new hypotheses, the hierarchy of perceptual signals is itself the object of perception, and the recipient of arbitrary actions. This new arrangement, originally intended only as a means of keeping reorganization closer to the point, gives the model as a whole two completely different types of perceptions: one which is a representation of the external world, and the other which is a perception of perceiving. And we have given the system as a whole the ability to produce spontaneous acts apparently unrelated to external events or control considerations: truly arbitrary but still organized acts.

As nearly as I can tell short of satori, we are now talking about awareness and volition.

Awareness seems to have the same character whether one is being aware of his finger or of his faults, his present automobile or the one he wishes Detroit would build, the automobile’s hubcap or its environmental impact. Perception changes like a kaleidoscope, while that sense of being aware remains quite unchanged. Similarly, crooking a finger requires the same act of will as varying one’s bowling delivery “to see what will happen.” Volition has the arbitrary nature required of a test stimulus (or seems to) and seems the same whatever is being willed. But awareness is more interesting, somehow.

The mobility of awareness is striking. While one is carrying out a complex behavior like driving a car through to work, one’s awareness can focus on efforts or sensations or configurations of all sorts, the ones being controlled or the ones passing by in short skirts, or even turn to some system idling in the background, working over some other problem or musing over some past event or future plan. It seems that the behavioral hierarchy can proceed quite automatically, controlling its own perceptual signals at many orders, while awareness moves here and there inspecting the machinery but making no comments of its own. It merely experiences in a mute and contentless way, judging everything with respect to intrinsic reference levels, not learned goals.

This leads to a working definition of consciousness. Consciousness consists of perception (presence of neural currents in a perceptual pathway) and awareness (reception by the reorganizing system of duplicates of those signals, which are all alike wherever they come from). In effect, conscious experience always has a point of view which is determined partly by the nature of the learned perceptual functions involved, and partly by built-in, experience-independent criteria. Those systems whose perceptual signals are being monitored by the reorganizing system are operating in the conscious mode. Those which are operating without their perceptual signals being monitored are in the unconscious mode (or preconscious, a fine distinction of Freud’s which I think unnecessary).

This speculative picture has, I believe, some logical implications that are borne out by experience. One implication is that only systems in the conscious mode are subject either to volitional disturbance or reorganization. The first condition seems experientially self-evident: can you imagine willing an arbitrary act unconsciously? The second is less self-evident, but still intuitively right. Learning seems to require consciousness (at least learning anything of much consequence). Therapy almost certainly does. If there is anything on which most psychotherapists would agree, I think it would be the principle that change demands consciousness from the point of view that needs changing. Furthermore, I think that anyone who has acquired a skill to the point of automaticity would agree that being conscious of the details tends to disrupt (that, is, begin reorganization of) the behavior. In how many applications have we heard that the way to interrupt a habit like a typing error is to execute the behavior “on purpose”—that is, consciously identifying with the behaving system instead of sitting off in another system worrying about the terrible effects of having the habit? And does not “on purpose” mean in this case arbitrarily not for some higher goals but just to inspect the act, itself?

That, then, is consciousness as Powers conceives it. It is correlated with reorganization. If we are to reorganize a perception or action, we must be aware of it. The fact that we spend most of our lives in some state of consciousness implies that we are always learning or, perhaps, maintaining ourselves in a state of readiness to learn.

What has this to do with polyviscosity? Here I am thinking of neural connectivity. It is polyviscous in that some connections are highly resistant to change while others change readily. Reorganization, that is to say learning, requires that neural connectivity change. Connections of various levels of viscosity are likely to be intermingled in any given volume of cortical tissue.

Now, consider this passage from a 1988 paper by Fodor and Pylyshyn, Connectionism and Cognitive Architecture: A Critical Analysis (pp. 22-23):

Classical theories are able to accommodate these sorts of considerations because they assume architectures in which there is a functional distinction between memory and program. In a system such as a Turing machine, where the length of the tape is not fixed in advance, changes in the amount of available memory can be affected without changing the computational structure of the machine; viz by making more tape available. By contrast, in a finite state automaton or a Connectionist machine, adding to the memory (e.g. by adding units to a network) alters the connectivity relations among nodes and thus does affect the machine’s computational structure. Connectionist cognitive architectures cannot, by their very nature, support an expandable memory, so they cannot support productive cognitive capacities. The long and short is that if productivity arguments are sound, then they show that the architecture of the mind can’t be Connectionist. Connectionists have, by and large, acknowledged this; so they are forced to reject productivity arguments.

Physically, the nervous system appears to be connectionist in character. And so adding new items to the system is physically problematic. That’s the problem that is solved by polyviscous connectivity – see my post, Physical constraints on computing, process and memory, Part 1 [LeCun] (Note: in that post I use the term “hyperviscous” rather than “polyviscous”). Some connections must remain stable while others change. The stable connections maintain the overall structural integrity of the network while the changing connections introduce new items into that structure.

Here's a recent article that’s relevant, though it doesn’t use the term “polyviscious”: Poonam Mishra and Rishikesh Narayanan, Stable continual learning through structured multiscale plasticity manifolds, Current Opinion in Neurobiology 2021, 70:51–63, https://doi.org/10.1016/j.conb.2021.07.009

Abstract: Biological plasticity is ubiquitous. How does the brain navigate this complex plasticity space, where any component can seemingly change, in adapting to an ever-changing environment? We build a systematic case that stable continuous learning is achieved by structured rules that enforce multiple, but not all, components to change together in specific directions. This rule-based low-dimensional plasticity manifold of permitted plasticity combinations emerges from cell type–specific molecular signaling and triggers cascading impacts that span multiple scales. These multiscale plasticity manifolds form the basis for behavioral learning and are dynamic entities that are altered by neuromodulation, metaplasticity, and pathology. We explore the strong links between heterogeneities, degeneracy, and plasticity manifolds and emphasize the need to incorporate plasticity manifolds into learning-theoretical frameworks and experimental designs.

Once more around the merry-go-round: Is the brain a computer?

I have argued I don’t know how many times that language is the simplest human activity than must be considered computational (e.g. in this comment on Yann LeCun’s latest proposal). That necessarily implies that, fundamentally, the brain is something else, but what?

What’s a computer?

Let’s step back a bit and consider an argument by John Searle. It may seem a bit strange, but bear with me.

In 1950, Alan Turing published an article in which he set out the Turing Test.1 The purpose of the test was to establish whether a computer had genuine intelligence: if an expert cannot distinguish between human intelligent performance and computer performance, then the computer has genuine human intelligence. It is important to note that Turing called his article “Computing Machinery and Intelligence.” In those days “computer” meant a person who computes. A computer was like a runner or a singer, someone who does the activity in question. The machines were not called “computers” but “computing machinery.”

The invention of machines that can do what human computers did has led to a change in the vocabulary. Most of us now think of “computer” as naming a type of machinery and not as a type of person. But it is important to see that in the literal, real, observer-independent sense in which humans compute, mechanical computers do not compute. They go through a set of transitions in electronic states that we can interpret computationally. The transitions in those electronic states are absolute or observer independent, but the computation is observer relative. The transitions in physical states are just electrical sequences unless some conscious agent can give them a computational interpretation.

This is an important point for understanding the significance of the computer revolution. When I, a human computer, add 2 + 2 to get 4, that computation is observer independent, intrinsic, original, and real. When my pocket calculator, a mechanical computer, does the same computation, the computation is observer relative, derivative, and dependent on human interpretation. There is no psychological reality at all to what is happening in the pocket calculator. [1]

He goes on in this vein for a bit and then arrives at this statement: “First, a digital computer is a syntactical machine. It manipulates symbols and does nothing else.” If you are familiar with his famous Chinese Room argument then you’ve heard this before. After a brief precis of that argument, which is of no particular interest here, Searle arrives at the point that interests me:

Except for the cases of computations carried out by conscious human beings, computation, as defined by Alan Turing and as implemented in actual pieces of machinery, is observer relative. The brute physical state transitions in a piece of electronic machinery are only computations relative to some actual or possible consciousness that can interpret the processes computationally. It is an epistemically objective fact that I am writing this in a Word program, but a Word program, though implemented electronically, is not an electrical phenomenon; it exists only relative to an observer.

Of course, he’s already said this before, but I repeat it because it’s a strange way of talking – at least I found it strange when I first read it – and so a bit of repetition is worthwhile.

Physically, a computer is just an extremely complex pile of electronic machinery. But we have designed it in such a way that the state transitions in its circuitry perform operations that we find useful. Most generally, we think of them as computation. The machinery is a computer because it has been designed to be one.

Is the brain a computer?

With Searle’s argument in mind we can now ask: Is the human brain, or any brain, a computer? Physically it is certainly very different from any electronic computer, but it does seem to be a complex meshwork that transmits many electrochemical signals in complex patterns. Are those signals performing calculations? Given Searle’s argument the answer to that question would seem to depend on just what the brain was designed to do. But, alas, that brain wasn’t designed in any ordinary sense of the word, though evolutionary biologists do sometimes talk about evolution as a process of design. But if so, it is design without an designer.

Given this, does it make sense for us to say that the brain IS a computer. I emphasize the “is” because, of course, we can simulate brains and parts of brains, but a simulation is one thing, and the thing being simulated is quite something else. The simulation of an atomic explosion is not the same as a real atomic explosion. Or, to switch terms, as Searle remarks, “Even with a perfect computer emulation of the stomach, you cannot then stuff a pizza into the computer and expect the computer to digest it.”

So, I’m not talking about whether or not we can produce a computer simulation of the brain. Of course we can. I’m talking about the brain itself. Is it a computer? Consider this passage:

This brings us to the question: what are the type of problems where generating a simulation is a more viable strategy than performing a detailed computation? And if so, what are the kind of simulators that might be relevant for consciousness? The answer to the first question has to do with the difference of say computing an explicit solution of a differential equation in order to determine the trajectory of a system in phase space versus mechanistically mimicking the given vector field of the equation within which an entity denoting the system is simply allowed to evolve thereby reconstructing its trajectory in phase space. The former involves explicit computational operations, whereas the latter simply mimics the dynamics of the system being simulated on a customized hardware. For complex problems involving a large number of variables and/or model uncertainly, the cost of inference by computation may scale very fast, whereas simulations generating outcomes of models or counterfactual models may be far more efficient. In fact, in control theory, the method of eigenvalue assignment is used precisely to implement the dynamics of a given system on a standardized hardware. [...] if the brain is indeed tasked with estimating the dynamics of a complex world filled with uncertainties, including hidden psychological states of other agents (for a game-theoretic discussion on this see [1–4,9]), then in order to act and achieve its goals, relying on pure computational inference would arguably be extremely costly and slow, whereas implementing simulations of world models as described above, on its cellular and molecular hardware would be a more viable alternative. These simulation engines are customized during the process of learning and development to acquire models of the world.[2]

That suggests that, no, the brain is not a computer, not if by that you mean that it is performing explicit numerical calculations. It isn’t performing calculations at all. It’s just passing signals between neurons, thousands and thousands of them each second. What those signals are doing, what they are achieving, depends on how the whole shebang is connected to the world in which the brain operates.

Consider experiments on mental rotation [3]. A subject is presented with a pair of 2-D or 3-D images. In some cases the images depict the same object, but from different points of view; it other cases the images depict two different objects. The subject is asked whether or not the images depict the same object. To perform the task the subject has to mentally rotate one of the images until it matches the other. But a match will be achieved only if the images depict the same object. If no match can be achieved then the subject is looking at two different objects.

What researchers found is that the length of time required to reach a decision was proportional to the angle between the two views. The larger the angle, the longer it takes to make a decision. If the process were numerical there’s no reason to believe that the computation time would be proportional to the size of the angle being computed. That strongly suggests that the process is analog, not numerical. If the brain IS a computer, it’s not a digital computer.

For various reasons – those experiments are only one of them – I have long been of the view that, at least at the sensorimotor level, the brain constructs quasi-analog models of the world and uses them in tracking the sensory field and in generating motor actions for operating in the world. These models are also called on in much of what is called common-sense knowledge, which proved to be very problematic for symbolic computation back in the world of GOFAI models in AI from the beginning up into the 1980s and which is proving somewhat problematic for current LLMs. In any given situation one simply calls up the necessary simulations and then generates whatever verbal commentary seems necessary or useful. GOFAI investigators were faced with the task of hand-coding a seemingly endless collection of propositions about common sense matters while LLMs are limited by the fact that they only have access to text, not the underlying simulations on which the text is based.

I arrived at this view partially on the basis of an elegant book from 1973, Behavior: The Control of Perception, by the late William Powers. As the title indicates, he developed a model of human behavior from classical control theory.

References

[1] John R. Searle, What Your Computer Can’t Know, The New York Review of Books, October 9, 2014, http://www.nybooks.com/articles/2014/10/09/what-your-computer-cant-know/

[2] Arsiwalla, X.D., Signorelli, C.M., Puigbo, JY., Freire, I.T., Verschure, P.F.M.J. Are Brains Computers, Emulators or Simulators? In V. Vouloutsi et al. (Eds.) Living Machines 2018. Lecture Notes in Computer Science, vol 10928. Springer, https://doi.org/10.1007/978-3-319-95972-6_3.

[3] Mental rotation, Wikipedia, https://en.wikipedia.org/wiki/Mental_rotation.