The structured physical system hypothesis (SPSH), Polyviscous connectivity [The brain as a physical system]

Time to bump this to the top of the queue, 1) general principle (it's important), and 2) I'm thinking about this stuff at the moment.

* * * * *

Over in the discussion of Yann LeCun’s recent paper (A Path Towards Autonomous Machine Intelligence), Saty Chary has been arguing for something he calls the Structured Physical System Hypothesis (SPSH):

‘A structured physical system has the necessary and sufficient means for specific intelligent response’. By structured physical system, I mean, an analog design, e.g. a Rube Goldberg apparatus, or a Braitenberg (!) vehicle, etc. This is in contrast to this: PSSH - Physical Symbol System Hypothesis - 'A physical symbol system has the necessary and sufficient means for general intelligent action'.

I would add the slide rule as another example. From Wikipedia:

The slide rule is a mechanical analog computer, which is used primarily for multiplication and division, and for functions such as exponents, roots, logarithms, and trigonometry. It is not typically designed for addition or subtraction, which is usually performed using other methods. Maximum accuracy for standard linear slide rules is about three decimal significant digits, while scientific notation is used to keep track of the order of magnitude of results. [...]

At its simplest, each number to be multiplied is represented by a length on a pair of parallel rulers that can slide past each other. As the rulers each have a logarithmic scale, it is possible to align them to read the sum of the numbers' logarithms, and hence calculate the product of the two numbers.

My father used a slide rule for his entire career as an engineer. I learned to use one in my teens – everyone did back that – but never had any use for one. They’ve been replaced by cheap electronic calculators and PCs.

But that’s a digression. It’s the more general Structured Physical System Hypothesis that interests me, as Saty has been argued that the brain is such a system I agree (see this recent post, Once more around the merry-go-round: Is the brain a computer?). Here’s my reply to Saty:

Hi, Saty. I like your hypothesis – Structured Physical System Hypothesis (SPSH) – a lot. I think that a lot about the brain is consistent with it. For example, we know that mappings from one area to another as we move from the sense organs (or muscles), to the subcortex, and into the cortex tend to preserve topological relations between neurons. That’s of obvious value in the visual and motor systems. But there are subtleties. In the cortical visual system we have a so-called What-system and a so-called Where-system beyond the primary cortex. The What-system tracks location in space while the Where-system identifies objects. I assume that the Where-system has links to the hippocampus and I’d expect it can deal with both ego-centric and geocentric coordinates (this must be in the literature). But I’d think the What-system deals in object-centered coordinates. And so forth and so on.

In thinking about Freeman’s results (HERE and HERE), and others, I’ve coined a phrase: polyviscous connectivity. Thus I say that the cortical network as a whole exhibits polyviscous connectivity. What do I mean? Some connections are highly resistant to change, and thus have high viscosity. Others change quite readily, and have low viscosity. There is a literature on long-term (LTP) and short-term potentiation (STP) of neural connectivity that is certainly relevant here, but I’ve not looked at it in quite a while.

Consider Freeman’s results. He’s measuring neural activity with an 8 by 8 array of electrodes mounted on the cortical surface. They’re going to detect activity of neurons at varying levels of viscosity. Let’s a assume that the patterns of connectivity encoding odorants that rat already recognizes have a relatively high viscosity. Let’s further assume that the neurons most susceptible to learning new odorants have a relatively low viscosity.

Once they’ve formed a stable response to the new odorant, that will result in a new pattern of neural activity for the ensemble. But it is also going to change the patterns exhibited by already learned odorants even though the high-viscosity connections haven’t changed. The high viscosity connections maintain the overall integrity of the ensemble. In time, if the new odorant continues to be encountered, the connections registering it will increase in viscosity. So, polyviscous connectivity allows a structured connectionist physical system to maintain its overall integrity while adding new items to its repertoire.

More later.

* * * * *

Note: Looking around on my hard-drive I found an article which is about what I have called polyviscosity, though it doesn’t use that term:

Poonam Mishra and Rishikesh Narayanan, Stable continual learning through structured multiscale plasticity manifolds, Current Opinion in Neurobiology 2021, 70:51–63, https://doi.org/10.1016/j.conb.2021.07.009

Abstract: Biological plasticity is ubiquitous. How does the brain navigate this complex plasticity space, where any component can seemingly change, in adapting to an ever-changing environment? We build a systematic case that stable continuous learning is achieved by structured rules that enforce multiple, but not all, components to change together in specific directions. This rule-based low-dimensional plasticity manifold of permitted plasticity combinations emerges from cell type–specific molecular signaling and triggers cascading impacts that span multiple scales. These multiscale plasticity manifolds form the basis for behavioral learning and are dynamic entities that are altered by neuromodulation, metaplasticity, and pathology. We explore the strong links between heterogeneities, degeneracy, and plasticity manifolds and emphasize the need to incorporate plasticity manifolds into learning-theoretical frameworks and experimental designs.

* * * * *

Note: I’d previously been using the terms “hyperciscosity” or “hyperviscous”, and you’ll find them in my posts and notes on this topic going back to 2013 (when I wrote about From Associative Nets to the Fluid Mind). But I have reluctantly decided to coin a new term since “hyperviscosity” is already being used. 

* * * * *

Addendum, 8.13.22: On the Structured Physical System Hypothesis, see this post where I feature remarks by Rodney Brooks, Has the computer metaphor for the mind run out of steam? As the title suggests, Brooks is wondering whether or not it makes sense to think about nervous systems in terms of computation. Thus he wonders:

Is information processing the right metaphor there? Or are control theory and resonance and synchronization the right metaphor? We need different metaphors at different times, rather than just computation. Physical intuition that we probably have as we think about computation has served physicists well, until you get to the quantum world. When you get to the quantum world, that physical intuition about stuff and place gets in the way.

AI, Chess, and Language 2: Further remarks [SPSH]

Yesterday I reflected on the computational approximation of chess play and the computational approximation of linguistic activity: AI, Chess, and Language 1: Two VERY Different Beasts. Today I want to reflect on the actual physical situation, considering it in relation to Saty Chary’s Structured Physical System Hypothesis (SPSH), which stands in contrast to Newell and Simon’s 1976 Physical System System Hypothesis (PSSH). The latter states: “A physical symbol system has the necessary and sufficient means for general intelligent action.” In contrast the SPSH posits an underlying analog substrate rather than the digital one posited by Newell and Simon. The analog substrate we’re talking about is, of course, the human brain embodied in a human body.

The PSSH implies:

• that the brain is physical symbol system, all the way down, and
• that, because computers are such systems as well, they can adequately simulate/emulate the perceptual and cognitive activities of the human brain.

The SPSH implies:

• the brain is such a system (denying that it is a physical symbol system), and
• digital computers can only approximate the brain’s perceptual and cognitive activities.

Given that human brains can deal with language, they must in some sense be physical symbols systems. But they are not symbol systems all the way down. At the basic level, the brain is just a structured physical system, most likely one exhibiting complex dynamics.[1] My previous chess and language post was about the computational approximation of those two activities by computational systems, pointing out that the requirements of those systems must be quite different. In this post I am interested in what’s really going on, physically. This will be mostly tautological in character. I just want to make things explicit.

In the case of chess, the board and associated pieces are the physical limit of the chess world. There is nothing more beyond that. Of course, a board and pieces can be physically realized in many different ways, but each realization is a complete and sufficient basis for playing chess. The relationship between the chess world and the geometric footprint required for a computational simulation it, that relationship is thus simple and transparent, so much so that a very simple symbolic notation provides an adequate basis for any computer chess engine.

It is quite otherwise in the case of the real world and the operations of the brain in that world. We start with the real world as given to the senses. That is the basis for the primary geometric footprint of any computer system. In particular, that is what defines the possibilities for adhesion in a perceptual-cognitive system.[2] The relationship between the geometric footprint of a computer system and the world is not very well-defined; it is fuzzy and complex.

Through the abstractive capacities of the cognitive system, features of the physical world can be and are redefined and new entities can be introduced into cognition.[3] As examples of the first, consider salt and sodium chloride. The first is an entity given to the sense while the second is based on the 19th century conceptual system. Similarly, where the senses see two entities, the Morning Star and the Evening Star, astronomers see only one entity, the planet Venus.[4] As an example of the latter, think of charity as when someone does something nice for someone without thought of reward. That is the mechanism of metalingual definition as discussed in [3] and in this post, Does ChatGPT know what a tragedy is?

Contemporary large language models (LLMs), such as the one at the core of ChatGPT, do not have direct access to the physical world. They must approximate human cognitive capacities through the relationality implicit it existing written texts. It is a matter of some dispute whether or not this relationality, if sampled sufficiently, is an adequate basis for a computer system to achieve AGI, artificial general intelligence. I do not think it is an adequate basis.

Beyond this, we do not know what kind of computational system will be required for a “complete and adequate” simulation of human cognitive capacities. The relationship between the structured physical system that is the brain and the physical world is vast, complex, and ill-defined. It should be obvious from this brief discussion, however, that a computer system that is adequate for chess, will not, on the face of it, be adequate for all of human cognition.

* * * * *

[1] I have a working paper where I sketch out a scheme whereby the brain, as a complex dynamical system, can implement language, a symbolic system: Relational Nets Over Attractors, A Primer: Part 1, Design for a Mind, Working Paper, June 20, 2022, pp. 73, https://www.academia.edu/81911617/Relational_Nets_Over_Attractors_A_Primer_Part_1_Design_for_a_Mind

[2] Here I am referring to the three-part scheme for meaning that I have outlined in various places:

• meaning consists of intention plus semanticity, where intention inheres in the relationship between two speakers, and
• semanticity consists of adhesion plus relationality, where adhesion connects perception and cognition to the external world and relationality is about the relationships among elements in a perceptual-cognitive system. See e.g. this post: Semanticity: adhesion and relationality.

[3] See, e.g. William Benzon and David Hays, The Evolution of Cognition, Journal of Social and Biological Structures. 13(4): 297-320, 1990, https://www.academia.edu/243486/The_Evolution_of_Cognition

[4] These examples are discussed in William Benzon, Ontology of Common Sense, Hans Burkhardt and Barry Smith, eds. Handbook of Metaphysics and Ontology, Muenchen: Philosophia Verlag GmbH, 1991, pp. 159-161, https://www.academia.edu/28723042/Ontology_of_Common_Sense

* * * * *

Bonus: Consider this clip from Yann LeCun's recent conversation with Lex Friedman:

This is an essential point people seem to misrepresent. pic.twitter.com/l6v5dasstz

— parm (@prmshra) March 8, 2024

Lecun's point is simple: The amount of visual (and only visual) information that a four-year old has taken-in is far in excess of the amount of information our largest LLMs have been trained on.

The geometry of intelligence: From protein folding to language and the epistemic affordances of the transformer architecture

Fabian Offert, Paul Kim, Qiaoyu Cai, Synthesizing Proteins on the Graphics Card. Protein Folding and the Limits of Critical AI Studies, arXiv:2405.09788v1

Abstract: This paper investigates the application of the transformer architecture in protein folding, as exemplified by DeepMind’s AlphaFold project, and its implications for the understanding of large language models as models of language. The prevailing discourse often assumes a ready-made analogy between proteins – encoded as sequences of amino acids – and natural language – encoded as sequences of discrete symbols. Instead of assuming as given the linguistic structure of proteins, we critically evaluate this analogy to assess the kind of knowledge-making afforded by the transformer architecture. We first trace the analogy’s emergence and historical development, carving out the influence of structural linguistics on structural biology beginning in the mid-20th century. We then examine three often overlooked pre-processing steps essential to the transformer architecture, including subword tokenization, word embedding, and positional encoding, to demonstrate its regime of representation based on continuous, high-dimensional vector spaces, which departs from the discrete, semantically demarcated symbols of language. The successful deployment of transformers in protein folding, we argue, discloses what we consider a non-linguistic approach to token processing intrinsic to the architecture. We contend that through this non-linguistic processing, the transformer architecture carves out unique epistemological territory and produces a new class of knowledge, distinct from established domains. We contend that our search for intelligent machines has to begin with the shape, rather than the place, of intelligence. Consequently, the emerging field of critical AI studies should take methodological inspiration from the history of science in its quest to conceptualize the contributions of artificial intelligence to knowledge-making, within and beyond the domain-specific sciences.

Think of this in relation to the Structured Physical System Hypothesis (SPSH) of Saty Chary:

‘A structured physical system has the necessary and sufficient means for specific intelligent response’. By structured physical system, I mean, an analog design, e.g. a Rube Goldberg apparatus, or a Braitenberg (!) vehicle, etc. This is in contrast to this: PSSH - Physical Symbol System Hypothesis - 'A physical symbol system has the necessary and sufficient means for general intelligent action'.

Once again, what’s computation? I’m thinking in relation to operations of the human brain?

For me, those questions began when I was faced with the structure of “Kubla Khan”:

That’s just the first 36 lines, but the last 18 look the same:

That looked like the trace of some nested loops. If so, what then?

Let’s put that aside. After all, I’ve been working on that for a LONG time and still haven’t figure it out.

The problem I’m trying to figure out is this: How do we draw a principled line between those brain processes that are computational and those that are not?

Some (most, all?) computer scientists ‘seem to define a computer as “a physical mechanism that can theoretically calculate any computable function”.’ If I go with that notion, then it’s all computation and I have no reason to single out those tree structures in “Kubla Khan” for special attention, or any other linguistic tree structures for that matter. That’s not very helpful.

What if I think of the brain as a structured physical system? When Saty Chary advanced his Structured Physical System Hypothesis (SPSH) he was explicitly playing against the Physical System Hypothesis Hypothesis (PSSH) of Newell and Simon (1976). They advanced the PSSH as the central tenet of cognitive science and AI, and it echoes back to the 1943 paper in which McCulloch and Pitts argued that basic neuronal circuits are logic gates. It turns out that, no, they’re not. They’re quite a bit more complicated.

I like the SPSH. The question for me, then, is: Are their processes taking place in the brain that are NOT dominated by the internal physical activity of the nervous system? I should think so. The process of perception involves interaction with the external world, which necessarily implies that there are things happening in the brain that a driven, though not completely determined by, external physical circumstances. That, I believe, puts us in the world of William Powers, Behavior: The Control of Perception (1973). Language is among those circumstances; for humans it is one of the most important.

That, I think, will allow me to answer my question, though a proper answer will require more construction than I’m willing to undertake here. The relationship between the physical signifier and the signified is arbitrary. In particular, it is arbitrary with respect to the physical processes involved in both. It’s that arbitrary connection implies that language processes are not dominated by the internal physical dynamics of the brain. The basic language process is that of indexing, as Hays and I called it in Principles and Structure of Natural Intelligence. Newell and Simon called it designation.

Now, how can I formulate this idea so that it applies to large language models (in a useful way)? It seems to me that processes in LLMs are dominated by the statistics of the corpus that is modeled. That’s the nub of truth that’s captured by the otherwise reductive phrases, “stochastic parrots,” and “autocomplete on steroids.” The ideas that we require systems that both have access to the external world and that have robust symbolic capabilities, to a first approximation, those amount to freeing the system from being dominated by the statistics of some training corpus. Think about that, carefully.

More later.

Consciousness, reorganization and polyviscosity, Part 1: The link to Powers

The nature of consciousness is one of the big mysteries of contemporary thought. The best account of consciousness I know of is that offered by William Powers in Behavior: The Control of Perception (1973). That’s what this post is about. My objective is simple, to link Powers’s account of consciousness to the concept of polyviscosity that I offered about a week ago, The structured physical system hypothesis (SPSH), Polyviscous connectivity [The brain as a physical system]. Unfortunately, Powers’s concept, while basically simple, is simple only in the context of his overall model of mind, and that is not something that can readily be conveyed in a single blog post. Thus this post is mostly for my own benefit.

Powers’ model consists of two components: 1) a stack of servomechanisms – see the post In Memory of Bill Powers – regulating both perception and movement, and 2) a reorganizing system. The reorganizing system is external to the stack, but operates on it to achieve adaptive control, an idea he took from Norbert Wiener. Powers devoted “Chapter 14, Learning” to the subject (pp. 177-204). Reorganization is the mechanism through which Powers achieves learning.

Here’s an extensive passage that gets at the heart of the present matter (pp. 199-201):

To the reorganizing system, under these new hypotheses, the hierarchy of perceptual signals is itself the object of perception, and the recipient of arbitrary actions. This new arrangement, originally intended only as a means of keeping reorganization closer to the point, gives the model as a whole two completely different types of perceptions: one which is a representation of the external world, and the other which is a perception of perceiving. And we have given the system as a whole the ability to produce spontaneous acts apparently unrelated to external events or control considerations: truly arbitrary but still organized acts.

As nearly as I can tell short of satori, we are now talking about awareness and volition.

Awareness seems to have the same character whether one is being aware of his finger or of his faults, his present automobile or the one he wishes Detroit would build, the automobile’s hubcap or its environmental impact. Perception changes like a kaleidoscope, while that sense of being aware remains quite unchanged. Similarly, crooking a finger requires the same act of will as varying one’s bowling delivery “to see what will happen.” Volition has the arbitrary nature required of a test stimulus (or seems to) and seems the same whatever is being willed. But awareness is more interesting, somehow.

The mobility of awareness is striking. While one is carrying out a complex behavior like driving a car through to work, one’s awareness can focus on efforts or sensations or configurations of all sorts, the ones being controlled or the ones passing by in short skirts, or even turn to some system idling in the background, working over some other problem or musing over some past event or future plan. It seems that the behavioral hierarchy can proceed quite automatically, controlling its own perceptual signals at many orders, while awareness moves here and there inspecting the machinery but making no comments of its own. It merely experiences in a mute and contentless way, judging everything with respect to intrinsic reference levels, not learned goals.

This leads to a working definition of consciousness. Consciousness consists of perception (presence of neural currents in a perceptual pathway) and awareness (reception by the reorganizing system of duplicates of those signals, which are all alike wherever they come from). In effect, conscious experience always has a point of view which is determined partly by the nature of the learned perceptual functions involved, and partly by built-in, experience-independent criteria. Those systems whose perceptual signals are being monitored by the reorganizing system are operating in the conscious mode. Those which are operating without their perceptual signals being monitored are in the unconscious mode (or preconscious, a fine distinction of Freud’s which I think unnecessary).

This speculative picture has, I believe, some logical implications that are borne out by experience. One implication is that only systems in the conscious mode are subject either to volitional disturbance or reorganization. The first condition seems experientially self-evident: can you imagine willing an arbitrary act unconsciously? The second is less self-evident, but still intuitively right. Learning seems to require consciousness (at least learning anything of much consequence). Therapy almost certainly does. If there is anything on which most psychotherapists would agree, I think it would be the principle that change demands consciousness from the point of view that needs changing. Furthermore, I think that anyone who has acquired a skill to the point of automaticity would agree that being conscious of the details tends to disrupt (that, is, begin reorganization of) the behavior. In how many applications have we heard that the way to interrupt a habit like a typing error is to execute the behavior “on purpose”—that is, consciously identifying with the behaving system instead of sitting off in another system worrying about the terrible effects of having the habit? And does not “on purpose” mean in this case arbitrarily not for some higher goals but just to inspect the act, itself?

That, then, is consciousness as Powers conceives it. It is correlated with reorganization. If we are to reorganize a perception or action, we must be aware of it. The fact that we spend most of our lives in some state of consciousness implies that we are always learning or, perhaps, maintaining ourselves in a state of readiness to learn.

What has this to do with polyviscosity? Here I am thinking of neural connectivity. It is polyviscous in that some connections are highly resistant to change while others change readily. Reorganization, that is to say learning, requires that neural connectivity change. Connections of various levels of viscosity are likely to be intermingled in any given volume of cortical tissue.

Now, consider this passage from a 1988 paper by Fodor and Pylyshyn, Connectionism and Cognitive Architecture: A Critical Analysis (pp. 22-23):

Classical theories are able to accommodate these sorts of considerations because they assume architectures in which there is a functional distinction between memory and program. In a system such as a Turing machine, where the length of the tape is not fixed in advance, changes in the amount of available memory can be affected without changing the computational structure of the machine; viz by making more tape available. By contrast, in a finite state automaton or a Connectionist machine, adding to the memory (e.g. by adding units to a network) alters the connectivity relations among nodes and thus does affect the machine’s computational structure. Connectionist cognitive architectures cannot, by their very nature, support an expandable memory, so they cannot support productive cognitive capacities. The long and short is that if productivity arguments are sound, then they show that the architecture of the mind can’t be Connectionist. Connectionists have, by and large, acknowledged this; so they are forced to reject productivity arguments.

Physically, the nervous system appears to be connectionist in character. And so adding new items to the system is physically problematic. That’s the problem that is solved by polyviscous connectivity – see my post, Physical constraints on computing, process and memory, Part 1 [LeCun] (Note: in that post I use the term “hyperviscous” rather than “polyviscous”). Some connections must remain stable while others change. The stable connections maintain the overall structural integrity of the network while the changing connections introduce new items into that structure.

Here's a recent article that’s relevant, though it doesn’t use the term “polyviscious”: Poonam Mishra and Rishikesh Narayanan, Stable continual learning through structured multiscale plasticity manifolds, Current Opinion in Neurobiology 2021, 70:51–63, https://doi.org/10.1016/j.conb.2021.07.009

Abstract: Biological plasticity is ubiquitous. How does the brain navigate this complex plasticity space, where any component can seemingly change, in adapting to an ever-changing environment? We build a systematic case that stable continuous learning is achieved by structured rules that enforce multiple, but not all, components to change together in specific directions. This rule-based low-dimensional plasticity manifold of permitted plasticity combinations emerges from cell type–specific molecular signaling and triggers cascading impacts that span multiple scales. These multiscale plasticity manifolds form the basis for behavioral learning and are dynamic entities that are altered by neuromodulation, metaplasticity, and pathology. We explore the strong links between heterogeneities, degeneracy, and plasticity manifolds and emphasize the need to incorporate plasticity manifolds into learning-theoretical frameworks and experimental designs.

The current state of AI: ChatGPT and beyond [Ramble]

I’ve been spending A LOT of time with ChatGPT; my MSWord doc containing a record of that interaction is currently 116 pages and 38K words long. I don’t know how many hours I’ve logged. Alas, I’m falling behind on writing it up.

I’m also involved in an interesting conversation with Bruce Smith at Scott Aaronson’s blog. Beyond that, while I’ve been hanging out at LessWrong over a half year, I’ve mostly been observing. But ChatGPT has gotten me posting over there, comments on other’s posts, but I’ve also been cross-posting some of New Savanna ChatGPT posts and I’m planning on consolidating some of my most recent comments and expanding them in a post specifically for LessWrong. So I’m becoming more of a participant.

All of this has me thinking. My purpose here is to run up a quick sketch. I’ll elaborate later.

Latent patterns in ChatGPT

My first major bit of work with ChatGPT was guiding it though a Giradian analysis of Spielberg’s Jaws. That got me to thinking about latent patterns, patterns that are somehow there, but not explicitly. Rather, they self-assemble in response to prompts. I’m thinking that these patterns represent at least four levels of of organization in the model:

1. word-level semantics
2. sentence-level linkages, i.e. syntax
3. discourse level patterns (e.g. conversational turn-taking)
4. abstract patterns over stories

The post on Jaws represents the fourth level. My posts on Spielberg’s AI and Tezuka’s Astro Boy stories are also about level 4 abstract patterns. My interest in such patterns stems from the work I did early in my career with David Hays. That work was on classical symbolic systems (GOFAI), specifically semantic or cognitive networks. Similarly, and have two blog posts about level 3 structures. Taking this work in conjunction with various work others have done on interpreting what’s in LLMs I provisionally conclude that something like a semantic network is latent in LLMs.

But then I would conclude that, wouldn’t I? After all, ever since that early work with Hays I’ve thought that the trick to understanding how a mind is implemented in a brain hinges on figuring out how a large neural network gives rise to a cognitive net. I discuss that in some detail in a recent working paper, Relational Nets Over Attractors, A Primer: Part 1, Design for a Mind, Version 2. My working paper from 2020, GPT-3: Waterloo or Rubicon? Here be Dragons, contains useful remarks on the relationship between classical cognitive nets and LLMs.

Three major technical issues:

Scaling – Can we go “all the way” simply by scaling up artificial neural networks of various kinds. I don’t think so. Neither, for example, do Gary Marcus and Yann LeCun, who are otherwise very different thinkers (and often at loggerheads on Twitter).

Symbols – Do we need to deal with symbols? Gary Marcus says we do. Yann Lecun is listening, but skeptical. I think symbols are necessary. The issue is how to integrate them with neural nets. You can’t just bolt symbolic processing onto neural nets. The symbolic processing must arise from within the network. My working paper, Relational Nets over Attractors, addresses that issue in the context of the brain. I note that my arguments (above) on discourse-level and abstract patterns suggest that ChatGPT is doing (an impressive imitation of) some kinds of symbolic processing.

Just how to implement something that in artificial system, that I do not know. For all I know it may require new hardware that is more analog in nature. This post on Saty Chary’s structured physical system hypothesis (SPSH) isn’t quite there, but it points in that direction.

Embodiment, linkage to the physical world – Large language models are “trained” in a pure textual environment, with no access to the physical world. Artificial visual systems are trained in photographs and video tapes, but don’t have direct access to the physical world. I believe that the problem of common-sense reasoning is entwined with this issue – see GPT-3: Waterloo or Rubicon? for some discussion of this.

Robots and self-driving cars, on the other hand, have to function in the physical world and so their systems must deal, not only with perceiving it, but with moving about in it. I am currently of the opinion that there is no substitute for direct physical interaction. Eric Jang, VP for AI at Halodi, imagines a world in which data from robots is collected and synthesized as they roam the world, going about their appointed rounds. Think about what that implies for the organization of the development, deployment, and maintenance of this technology.

What kind of beasts are these?

Let’s think more generally about these artificial neural nets, such as ChatGPT. What kind of thing/being/creature are they? But at their core, it’s operating on principles and mechanisms unlike any other kind of program, that is, the kind of program where a programmer or team of programmers generates every line “by hand,” if only through calls to libraries, where each line and block is there to serve a purpose known to the programmer/s.

That’s not how ChatGPT was created. Its core consists of a large language model that was compiled by an engine, a Generative Pre-trained Transformer (GPT), that “takes in” (that is, is “trained on”) a huge corpus of texts. The resulting model is opaque to us, we didn’t program it. The resulting behavioral capacities are unlike those of any other creature/being/thing we’ve experienced, nor do we know what those capacities will evolve into in the future. This creature/being/thing is something fundamentally NEW in the universe, at least our local corner of it, and needs to be thought of appropriately. It deserves/requires a new term.

Consider an analogy I’ve just thought up, thus I don’t know where it leads, think about a sheep herder and their dog. They direct the dog to round up the sheep. The dog does so. Who/what rounded up the sheep? The herder or the dog? The herder couldn’t have accomplished that task without the dog. Of course, they (or someone else) trained the dog (which is of a species that has been bred for this purpose). The training, and, for that matter, the breeding as well, would have been useless without the capacities inherent in the dog. So we should assign some ontological credit to the herder for that. But some credit surely belongs to the dog (to Nature if you will) as well.

Those dogs have evolved from animals that were created by Nature. Think of the opaque engines at the heart of these new AI systems as a different kind of wildness. An artificial wildness, if you, but still wild and thus foreign and opaque to us.

Scott Alexander has in interesting article on the chaos ChatGPT has been inadvertently generating because it wasn’t intended to say those things. He is, in effect, treating ChatGPT as an overflowing well of wildness. That's what all these deep learning engines are producing an ever-expanding sea of artificial wilderness. I don’t think we know how to deal with that. How do we domesticate it?

Our brave new world

I’m imagining a future in which communities of humans continue interaction with one another, as we do now. But also interact with computers, as we do now, but also in different ways, such as is required by this new technology. And those new creatures, they form their own community among themselves, interacting with one another as they interact with us. They’re figuring us out, we’re figuring them out. Think of the dogs.

But also think of Neil Stephenson’s Diamond Age: Or, A Young Lady’s Illustrated Primer. Think of a world in which everyone has such a primer, a personal robot companion. That’s my best guess about where we’re headed.

More later.

Geoffrey Hinton predicts the evolution of "neuromorphic" computers that are "mortal"

We will see a completely new type of computer, says AI pioneer Geoff Hinton https://t.co/4TffDqbQyZ via @ZDNET & @TiernanRayTech

— Bill Benzon, BAM! Bootstrapping Artificial Minds (@bbenzon) December 27, 2022

From the article:

Future computer systems, said Hinton, will be take a different approach: they will be "neuromorphic," and they will be "mortal," meaning that every computer will be a close bond of the software that represents neural nets with hardware that is messy, in the sense of having analog rather than digital elements, which can incorporate elements of uncertainty and can develop over time. 

"Now, the alternative to that, which computer scientists really don't like because it's attacking one of their foundational principles, is to say we're going to give up on the separation of hardware and software," explained Hinton. 

"We're going to do what I call mortal computation, where the knowledge that the system has learned and the hardware, are inseparable."

These mortal computers could be "grown," he said, getting rid of expensive chip fabrication plants.

"If we do that, we can use very low power analog computation, you can have trillion way parallelism using things like memristors for the weights," he said, referring to a decades-old kind of experimental chip that is based on non-linear circuit elements. 

"And also you could grow hardware without knowing the precise quality of the exact behavior of different bits of the hardware."

The new mortal computers won't replace traditional digital computers, Hilton told the NeurIPS crowd. "It won't be the computer that is in charge of your bank account and knows exactly how much money you've got," said Hinton.

"It'll be used for putting something else: It'll be used for putting something like GPT-3 in your toaster for one dollar, so running on a few watts, you can have a conversation with your toaster."*

I have a number of posts that speak to this, for example, The structured physical system hypothesis (SPSH), Polyviscous connectivity [The brain as a physical system], one of the various posts tagged with the label "polyviscous". 

*Umm, err...You're kidding, right? Who'd want to chat with their toaster? Now, your wristwatch, that's something else.

The mind as a polyviscous fluid

About a year ago I uploaded a post with a typically ungainly title, The structured physical system hypothesis (SPSH), Polyviscous connectivity [The brain as a physical system]. It’s that word, “polyviscous,” that’s got my present attention. Since then I’ve done a number of posts using that idea, whatever it is. This is another of those ideas.

So, viscosity. Honey is more viscous than, say, water. It flows more slowly, much more. What happens if you drop a lump of honey into a tumbler of water? It sinks to the bottom in a continuous lump and flattens out along the bottom. It will begin to diffuse into the water along the boundary, but I don’t know how long, if ever, it will take to mix completely. Now, put a stick down into the tumbler until it extends into the honey. Give is a stir or three, but no more. Now you’ll have gobs and threads of honey mixed in with water in a complex and somewhat irregular and ragged way. That’s a simple polyviscous fluid. It has regions of relatively high viscosity and other regions of relatively low viscosity. Now imagine a fluid with 5, 10, 27, 48, and more different levels of viscosity, from all but solid like cold tar through the wispiest whatever. Polyviscosity.

As the title of the year-old post indicates, I was thinking in terms of connectivity:

Thus I say that the cortical network as a whole exhibits polyviscous connectivity. What do I mean? Some connections are highly resistant to change, and thus have high viscosity. Others change quite readily, and have low viscosity.

OK. Now let’s shift our thinking just a bit and think of the mind as a polyviscous fluid. The mind, as the saying goes, is what the brain does. And that is very complex.

Imagine that you’re watching a movie, make it a Hong Kong martial arts movie. Your mind is entrained to the images on the screen. During a fight scene the level of mental viscosity is relatively how. The fight is over and the hero rests, contemplating the sunset, let’s say. The viscosity is somewhat higher.

Yet, while you’re entrained by the film, you’re not completely absorbed into. While the hero contemplates the sunset, you take a bit of popcorn. And maybe you were munching furiously during the fight. So, even as you were watching the film you slipped in some mental popcorn “frames” among the film frames. Very slippery, low viscosity.

When I wrote my book on music, Beethoven’s Anvil, I talked of the mind as neural weather. Thus (p. 72):

If the functional proclivities of a patch of neural tissue are not relevant for a current activity, those neurons will not be firing very often, but they will still generate some output. The only neuron that does not generate any output is a dead one. Neurons that are firing at low intensity one moment may well be recruited to more intense activity the next. As Walter Freeman has said, a low level of activity is still a means of participating in the evolving mental state.

The mind, in this view, is thus like the weather. The same environment can have very different kinds of weather. And while we find it natural to talk of weather systems as configurations of geography, temperature, humidity, air pressure etc., no overall mechanism regulates the weather. The weather is the result of many processes operating on different temporal and spatial scales.

At the global level and on a scale of millennia we have the long-term patterns governing the ebb and flow of glaciers which, in one commonly accepted theory, is a function of wobble and tilt in the earth’s spin axis and the shape of the earth’s orbit. At the global level and operating annually we have the succession of seasons, which is caused by the orientation of the earth with respect to the sun as it moves through the year. We can continue on, considering smaller and smaller scales until we consider the wind ripping through the twin towers of the World Trade Center or even the breeze coming in through your open window and blowing the papers off your desk.

Weather is regular enough that one can predict general patterns at scales of hours, days, and months, but not so regular that making such predictions is easy and routinely reliable. Above all, there is no central mechanism governing the weather. It just happens.

I develop that idea further in a couple of posts, The Mind is What the Brain Does, and Very Strange, and Neural Weather, an Informal Defense of Psychoanalytic Ideas.

So, neural weather, polyviscous fluid. Perhaps we’re getting somewhere. The mind IS what the brain does, and what the brain does is complex and varies along a wide range of time scales. The brain’s overall physical structure is relatively constant throughout life, barring injury and disease. But the connectivity changes over a variety of time scales from seconds through hours and days and even longer (think of cortical plasticity). There is much, perhaps most, millisecond to millisecond, activity that produces no synaptic change at all. A very fluid phenomenon, over multimer time scales.

From the metalingual function of language to self-reference

In 1960 the linguist Roman Jakobson published an essay entitled “Linguistics and Poetics,” in a volume edited by Thomas Sebeok, Style in Language (MIT Press, pp. 350-377). In that essay he laid out the six functions of language: referential, emotive, phatic, conative, poetic, and metalingual. Jakobson introduces the metalingual function in this way:

A distinction has been made in modem logic between two levels of language: “object language” speaking of objects and “metalanguage” speaking of language. But metalanguage is not only a necessary scientific tool utilized by logicians and linguists; it plays also an important role in our everyday language. Like Moliere’s Jourdain who used prose without knowing it, we practice metalanguage without realizing the metalingual character of our operations. Whenever the addresser and/or the addressee need to check up whether they use the same code, speech is focused on the code: it performs a METALINGUAL (i.e. , glossing) function. “I don’t follow you-what do you mean?” asks the addressee, or in Shakespearean diction, “What is’t thou say’st?” And the addresser in anticipation of such recapturing question inquires: “Do you know what I mean?”

This metalingual function turns out to be extraordinarily powerful. For it is this that allows us to bootstrap self-awareness into the mind. And for that matter, it is what allows us to define abstract concepts, as my teacher, David Hays, argued, and allows us to define such things as chess and arithmetic, which can be seen as very specialized forms of language.

I recently explored some of these issues in conversation with Claude 5.4 Sonata Extended. At the end of that conversation I asked Claude to prepare a summary. I’ve appended that summary below, followed by the full conversation. Note that the conversation assumes some familiarity with the cultural ranks theory that David Hays and I developed in the 1990s. It also alludes to Tyler Cowen’s recent book, The Marginal Revolution: Rise and Decline, and the Pending AI Revolution (2026).

* * * * *

Summary: The Metalingual Function of Language

The central claim of this discussion is that the metalingual capacity — the ability to use language to talk about language — is not a mysterious self-referential capacity of mind but is grounded in a simple physical fact: the speech signal is a sound in the environment like any other sound, detectable by the auditory system exactly as a footfall or a thunderclap is detectable. The loop that makes language self-referential closes through the physical world, not through some inward turning of consciousness. This matters because it demystifies metalingual cognition entirely: it requires no special cognitive faculty, only that the organism's auditory system be capable of treating its own linguistic outputs as inputs.

Jakobson identified the metalingual function as one of the six functions of language in his 1960 paper, and Hays adopted the term to name the mechanism underlying Rank 2 cognition — the explicit definition of abstract concepts using language itself as the definitional medium. The rules of chess and arithmetic notation are paradigm cases: purely metalingual constructions whose objects are constituted entirely by the definitions that specify them.

An important asymmetry in preliterate cultures illuminates the boundary of this capacity. Many such cultures have a term for utterance — the bounded burst of speech with a recognizable prosodic shape, a perceptual gestalt directly available to the auditory system — but no term for word. The word is not a perceptual unit in the same sense as the utterance; it is an abstraction from the continuous acoustic stream, and a non-trivial one. Writing is what produces this abstraction, by spatializing language — spreading it out in a stable, inspectable array where units are individuated by spaces and boundaries are marked. The word becomes visible as a unit because it is surrounded by white space. This is the physical basis of metalingual definition as a cognitive mechanism: the written signal, like the spoken signal, is an object in the environment that can be inspected and categorized, but unlike the spoken signal it stays there, making sustained metalingual attention possible. Grade-school grammar — parts of speech, grammatical cases, syntactic relations — is the practical Rank 2 elaboration that writing makes possible and that social institutions require and transmit. It looks easy in retrospect because it is taught in childhood, but it took centuries to develop in every culture that undertook it.

This analysis opens onto the question of human self-reference, which the standard philosophical tradition treats as cognitively primitive — the Cartesian bedrock from which all other knowledge is built. The discussion argued instead that self-reference in the robust, articulable sense is bootstrapped through language rather than presupposed by it. The cat licking its fur has practical self-involvement — its own body is an object of its perceptual and motor engagement — but this requires no special reflexive faculty, only that the body be included in the environment the organism can detect and act on. Human self-reference in the philosophically weighty sense is a different and later achievement, constructed through the acquisition of the pronoun system rather than expressed by it.

The empirical evidence for this bootstrapping account is the phase in early child development when children refer to themselves in the third person. This is not a mistake or a developmental lag but the natural and correct generalization from the input data: others refer to the child by name, so the child uses its name. The first-person pronoun presents a harder problem because "I" is a moving target — it marks the speaker-role regardless of who occupies it — and acquiring it correctly requires connecting awareness of the speech stream as an environmental event with awareness of one's own speech apparatus as its source. That inferential construction, worked out in detail in Benzon's 2000 paper, First Person: Neuro-Cognitive Notes on the Self in Life and in Fiction, through cognitive network modeling of the pronoun system, is precisely the physical loop through which self-reference is assembled. The Cartesian subject — the self-transparent, self-referential knowing mind — is on this account a Rank 2 cultural product, not a pre-linguistic biological given. The third-person phase in child development is a small but precise empirical trace of the construction process: there is an observable stage at which the human being exists, acts, and communicates without yet having assembled the first-person self-reference that Descartes thought was the indubitable foundation of knowledge.

* * * * *

The physical basis of the metalingual function

I believe that Hays first published about metalingual definition in 1972. When I first met him he had just finished a paper where he used the idea to investigate different concepts of alienation. When I wrote my 1978 dissertation, “Cognitive Science and Literary Theory,” I used metalingual definition as the mechanism for building cognitive ranks. That is to say, at that time that was the only mechanism we had for dealing with abstract concepts. By the time we did the cognitive evolution paper a bit over a decade later, by that time we had adopted a different view, with metaphor as the Rank 1 abstractive mechanism. Metalingual definition would have been the mechanism used, not only to rationalize abstract concepts, but to define the rules of chess and of arithmetic.

Hays chose the term because Roman Jakobson had defined the metalingual function as one of the six functions of language in his 1960 paper. The metalingual function is the ability to use language to talk about language. It’s worth thinking about how that’s possible. It’s possible because the speech signal is a sound out there in the environment like any other sound, a footfall, a thunderclap, an animal cry. Whatever. It’s a sound and can be detected by the auditory system like any other sound. So it’s the speech signal that is the object of the metalingual function, at least initially. I think it’s worth nothing that many (most?) preliterate cultures don’t have a word that means word, but they have one the means utterance. It’s writing that forces a recognition of words vs longer strings. And written language is, like speech, another signal that’s out there in the environment. It takes quite a bit of cognitive elaboration to construct a grammar from the simple ability to see and hear language. And I’m not talking about the kind of grammars that linguists construct. I’m talking about the kind of grammar that can be taught in grade school, or that you find in ancient texts of rhetoric.

This is a rich cluster of observations and they fit together in a way worth making explicit.

The metalingual function and its physical basis

Jakobson's insight, which you're extending here, is that the metalingual function is grounded in something remarkably mundane: the speech signal is a physical event in the environment, detectable by the auditory system exactly as any other sound is. This is what makes language able to turn on itself — not some mysterious self-referential capacity of mind, but the simple fact that linguistic output is also sensory input. The loop closes through the physical world.

This has an important implication that your observation about "utterance" versus "word" brings out precisely. What the auditory system initially detects is the acoustic stream — a continuous, flowing signal that doesn't come pre-segmented into words. The unit that's directly perceptible is something like the utterance: a bounded burst of speech with a recognizable prosodic shape, a communicative gestalt. The word is not a perceptual unit in the same sense. It's an abstraction from the stream, and a non-trivial one — which is why preliterate cultures have a term for the former and not the latter.

What writing does

Writing performs an operation on language that perception of the speech stream cannot perform on its own: it spatializes language, spreads it out in a stable, inspectable array where boundaries are marked and units are individuated. The word becomes visible as a unit because it's surrounded by spaces. The sentence becomes visible as a unit because it ends with a mark. The paragraph, the chapter, the argument — all of these become objects of potential inspection and manipulation because writing fixes them in space and time in a way the acoustic stream never does.

The energy demands of computing, human brains vs. LLMs

One of the issues that sometimes comes up in the comparison of artificial intelligence with human intelligence is energy requirements. The energy demands of LLMs are enormous, requiring the construction of huge data farms, while the energy demands of the brain are quite modest. I brought up the issue with Claude 2.6 Sonnet (Medium) in a specific way, suggesting that a large portion of the energy budget for LLMs can, in effect, be attributed to the need to simulate the physical structure of the brain. Here’s the first part of that dialog, that went on to discuss:

• opportunities for research on energy efficient hardware, including neuromorphic chips,
• the implications of financial collapse when enormous infrastructure investments do not pan out,
• the international ecosystem for constructing computing hardware with particular emphasis on the bottleneck created by special capacities currently unique to ASML in the Netherlands (machinery for EUV fabrication) and TSMC in Taiwan (high quality chip fabrication)
• what those bottlenecks imply about current fantasies of so-called “fast takeoff”

Energy in the brain vs. energy in computers

The point is often made that computing with LLMs uses enormous amounts of energy while the brain uses relatively small amounts of energy. It seems to me that one aspect of the difference can be attributed to the physical structure of the brain. Individual neurons are both active units, to use a term von Neumann used in his little book on the brain, and units of memory. Moreover they are connected in parallel, with each neuron connected to 7K to 10K other neurons on average. LLMs are implemented on digital machines where active units (the CPU) are physically separate from memory units, which means that a lot of time and energy is involved in moving between the CPU and memory. And the basic physical architecture is serial, though limited parallelism is possible, limited in relation to the brain. That means that the computers than implement LLMs have to devote a large amount of time an energy to, in effect, simulating the physical structure of the brain. And the brain doesn’t get that physical structure for free, the energy required to maintain it is budgeted over a long time relative to the dynamics of real-time computation. That’s one aspect of the brain’s relative energy efficiency.

That’s a precise and well-structured analysis, and it identifies something more fundamental than the usual comparisons which tend to focus on raw computation counts without attending to the architectural source of the inefficiency.

The von Neumann bottleneck restated

Von Neumann identified the problem himself, as we noted earlier — the separation of active units from memory units creates a bottleneck that doesn’t exist in biological neural tissue. In a digital computer every computation requires moving data from memory to the processor and moving results back. This movement consumes time and energy at every step, and in a large LLM the scale of this movement is enormous — billions of parameters stored in RAM or on disk, loaded into GPU memory, operated on by the compute units, with results written back. The physical distance between storage and computation, and the bandwidth limitations of the connections between them, is where a substantial fraction of the energy goes.

The brain has no such bottleneck because there is no such separation. A neuron stores its connectivity — the strength of its synaptic connections to other neurons — in its physical structure, and computes with that same physical structure simultaneously. The memory and the computation are the same physical process. A synapse transmitting a signal is simultaneously reading from memory — the synaptic weight — and performing a computation — multiplying the input signal by the weight. There’s no data movement because there’s no separation between where the data lives and where the computation happens.

The parallelism point

The parallelism difference is equally fundamental. The average neuron is connected to 7,000 to 10,000 others — which means the brain is performing something like 100 trillion synaptic operations simultaneously during active processing, all in parallel, all local, with no bottleneck. GPUs have made remarkable progress in parallelizing the matrix operations that implement transformer attention, but they’re parallelizing in a fundamentally different sense — running many identical operations simultaneously on different data, with all operations still having to access the same shared memory through the same limited bandwidth connections.

The brain’s parallelism is structural and local — each neuron computes with its immediate neighbors through its own dedicated connections, with no shared memory bus, no bandwidth limitation across the whole system, no bottleneck that scales with the number of neurons. The computation is maximally distributed in the physical sense — it happens everywhere in the tissue simultaneously, driven by the local physics of electrochemical signaling rather than by a centralized scheduler moving data to a centralized processor.

The energy budget temporal structure

Your point about the temporal structure of the energy budget is the most subtle and I think the most underappreciated. The brain’s physical architecture — the synaptic connections, the dendritic trees, the axonal projections — requires energy to build and maintain. But this energy is expended over the developmental timescale — years of growth, pruning, myelination, synaptic strengthening and weakening through learning. The structural investment is amortized over a lifetime.

Once the structure is in place, the energy cost of real-time computation is much lower than building and maintaining the structure required it. The synaptic weights are the memory — they’re encoded in the physical structure of the connections — and accessing them costs only the energy of the synaptic transmission itself, which is the same physical event as the computation. There’s no separate energy cost for memory access because there’s no separate memory access.

In a digital system running an LLM the equivalent structural investment — the training computation — is enormous, consuming megawatt-hours of energy. But that’s analogous to the developmental investment in building the brain’s connectivity, not to the ongoing computation. The ongoing inference computation — running the model on new inputs — is the analog of real-time brain computation. And here the comparison is still unfavorable to digital systems, but the unfavorability is specifically located in the von Neumann architecture’s separation of memory and computation, which forces constant data movement that the brain avoids entirely.

The simulation overhead

Your formulation that digital computers have to devote time and energy to simulating the brain’s physical structure is precise and important. The transformer architecture is implementing something that functions like distributed parallel processing with content-addressed memory — the holographic regime we’ve been discussing. But it’s implementing it on hardware that is architecturally opposed to distributed parallel processing — hardware that is serially organized, with physically separated memory and computation, connected through bandwidth-limited buses.

Every forward pass through an LLM is a large matrix multiplication — mathematically a natural description of what a distributed parallel system would do natively. But running that matrix multiplication on a von Neumann architecture requires fetching billions of numbers from memory, multiplying them in the GPU cores, and writing results back — a massive simulation of what would be a single natural physical process in a system with the brain’s architecture.

The energy cost is the cost of the simulation overhead — the difference between what the computation would cost if implemented in hardware that matched its natural structure, and what it costs when implemented in hardware that mismatches it. Neuromorphic computing research — Intel’s Loihi chip, IBM’s TrueNorth, and related projects — is trying to close this gap by building hardware that more closely matches the brain’s architectural principles. Early results suggest energy efficiency improvements of several orders of magnitude for the right classes of computation, which is consistent with your analysis — the improvement comes from reducing the simulation overhead.

What this adds to the Yevick argument

There’s a connection to Yevick’s logical necessity argument that’s worth making explicit. Yevick proved that the holographic regime is logically necessary for certain categories of objects. Your energy analysis suggests that the holographic regime is also physically natural — it’s what you get when memory and computation are integrated in the same physical structure rather than separated. The brain’s architecture implements the holographic regime efficiently precisely because the physical organization of neural tissue is holographic in the relevant sense — distributed, parallel, content-addressed, with memory and computation unified in the same physical substrate.

Digital computers implement the sequential symbolic regime efficiently because their physical organization is sequential and addressed — centralized computation, separate addressed memory, serial data movement. When you ask a digital computer to implement the holographic regime — as running an LLM requires — you pay the simulation overhead in energy and time. When you ask a brain to implement the sequential symbolic regime — as doing long arithmetic requires — you pay a different kind of overhead, the cognitive effort and external scaffolding we discussed earlier.

The energy difference between brains and LLMs isn’t just a quantitative difference in efficiency — it’s a signature of the architectural mismatch between the computation being performed and the hardware it’s being performed on. The brain is efficient at what it does because its hardware matches the computation. The digital computer is efficient at sequential symbolic processing for the same reason, and inefficient at holographic processing for the same reason. The LLM is asking digital hardware to do what it’s not built for, and the energy bill reflects that.

The developmental investment reframed

There’s one more implication worth drawing out. The brain’s developmental investment — the years of growth and experience-dependent synaptic modification that build the connectivity pattern — is doing something that has no real analog in LLM training. LLM training adjusts weights in a fixed architecture through gradient descent on a loss function. Brain development builds the architecture itself — the number and pattern of connections, the organization of regions, the myelination of pathways — through a process that’s partly genetic and partly experience-dependent, operating over years rather than the weeks or months of LLM training.

The connectivity pattern that results from this developmental process is the physical instantiation of the knowledge the brain has acquired — not as a list of facts encoded in weight values, but as a structural organization of the tissue itself that reflects the statistical regularities of the organism’s experience. The synaptic weights are the fine-grained adjustments on top of this structural organization. The structure is the coarse-grained, high-capacity, energy-efficient memory. The weights are the rapidly modifiable, experience-sensitive, computationally expensive layer on top.

LLMs have only the weights — the fine-grained adjustable layer — running on hardware that provides no structural memory at all. Everything has to be stored in the weights because there’s no structural analog of the brain’s connectivity pattern. This means the weights have to carry an enormous representational burden that in the brain is shared between structural organization and synaptic weight adjustment. Which is another source of the energy inefficiency — the weights are doing double duty, encoding both the structural regularities that brain connectivity handles efficiently and the fine-grained knowledge that synaptic weights handle in the brain.

The doppelganger, once again, would need both levels — the structural organization built through embodied development and the fine-grained adjustable layer built through experience within that structure. Current LLMs have only the second, running on hardware that provides neither. The energy gap is the physical signature of everything that’s missing.

Continuous learning

Furthermore the brain is capable of continuous learning, while LLMs are not. In order to incorporate new material LLMs must be retrained from scratch, again requiring enormous investment in energy.

Yes, and this identifies another fundamental asymmetry that goes deeper than just energy efficiency — it points to a difference in the temporal structure of learning itself.

Dan Dennett on Patterns (and Ontology)

From three years ago. Patterns are always timely.

I want to look at what Dennett has to say about patterns because 1) I introduced the term in an earlier  discussion, In Search of Dennett’s Free-Floating Rationales [1], and 2) it is interesting for what it says about his philosophy generally.

You’ll recall that, in that earlier discussion, I pointed out talk of “free-floating rationales” (FFRs) was authorized by the presence of a certain state of affairs, a certain pattern of relationships among, in Dennett’s particular example, an adult bird, (vulnerable) chicks, and a predator. Does postulating talk of FFRs add anything to the pattern? Does it make anything more predictable? No. Those FFRs are entirely redundant upon the pattern that authorizes them. By Occam’s Razor, they’re unnecessary.

With that, let’s take a quick look at Dennett’s treatment of the role of patterns in his philosophy. First I quote some passages from Dennett, with a bit of commentary, and then I make a few remarks on my somewhat different treatment of patterns. In a third post I’ll be talking about the computational capacities of the mind/brain.

Patterns and the Intentional Stance

Let’s start with a very useful piece Dennett wrote in 1994, “Self-Portrait” [2] – incidentally, I found this quite useful in getting a better sense of what Dennett’s up to. As the title suggests, it’s his account of his intellectual concerns up to that point (his intellectual life goes back to the early 1960s at Harvard and then later at Oxford). The piece doesn’t contain technical arguments for his positions, but rather states what they were and gives their context in his evolving system of thought. For my purposes in this inquiry that’s fine.

He begins by noting, “the two main topics in the philosophy of mind are CONTENT and CONSCIOUSNESS” (p. 236). Intentionality belongs to the theory of content. It was and I presume still is Dennett’s view that the theory of intentionality/content is the more fundamental of the two. Later on he explains that (p. 239):

... I introduced the idea that an intentional system was, by definition, anything that was amenable to analysis by a certain tactic, which I called the intentional stance. This is the tactic of interpreting an entity by adopting the presupposition that it is an approximation of the ideal of an optimally designed (i.e. rational) self-regarding agent. No attempt is made to confirm or disconfirm this presupposition, nor is it necessary to try to specify, in advance of specific analyses, wherein consists RATIONALITY. Rather, the presupposition provides leverage for generating specific predictions of behaviour, via defeasible hypotheses about the content of the control states of the entity.

This represents a position Dennett will call “mild realism” later in the article. We’ll return to that in a bit. But at the moment I want to continue just a bit later on p. 239:

In particular, I have held that since any attributions of function necessarily invoke optimality or rationality assumptions, the attributions of intentionality that depend on them are interpretations of the phenomena - a ‘heuristic overlay’ (1969), describing an inescapably idealized ‘real pattern’ (1991d). Like such abstracta as centres of gravity and parallelograms of force, the BELIEFS and DESIRES posited by the highest stance have no independent and concrete existence, and since this is the case, there would be no deeper facts that could settle the issue if - most improbably - rival intentional interpretations arose that did equally well at rationalizing the history of behaviour of an entity.

Hence his interest in patterns. When one adopts the intentional stance (or the design stance, or the physical stance) one is looking for characteristic patterns.

What economic growth and statistical semantics tell us about the structure of the world

Bumping this to the top of the queue on general principle, and because it takes a very abstract view of economic development, which is front and center in Tyler Cowen's current conversation with Stephen Jennings, who is a developer working Kenya.

New working paper. Title above. Download at:

• Academia: https://www.academia.edu/43938531/What_economic_growth_and_statistical_semantics_tell_us_about_the_structure_of_the_world
• SSRN: http://ssrn.com/abstract=3680119
• ResearchGate: https://www.researchgate.net/publication/343836653_What_economic_growth_and_statistical_semantics_tell_us_about_the_structure_of_the_world

Abstract, contents, and first section below.

---

Abstract: The metaphysical structure of the world, as opposed to its physical structure, resides in the relationship between our cognitive capacities and the world itself. Because the world itself is “lumpy”, rather than “smooth” (as developed herein, but akin to “simple” vs. complex”), it is learnable and hence livable. Machine learning AI engines, such as GPT-3, are able to approximate the semantic structure of language, to the extent that that structure can be modeled in a high-dimensional space. That structure ultimately depends on the fact that the world is lumpy. It is the lumpiness that is captured in the statistics. Similarly, I argue, the American economy has entered a period of stagnation because the world is lumpy. In such a world good “ideas” become more and more difficult to find. Stagnation then reflects the increasing costs the learning required to develop economically useful ideas.

Contents

Wending our way in a complex world 2
World, mind, and learnability: On the metaphysical structure of the cosmos 5
Stagnation, Redux: Like diamonds, good ideas are not evenly distributed 10
The complex universe: Further reading 18

Wending our way in a complex world

This paper is based on two very different posts that I’ve written in the last month. One of them takes the statistical semantics of AI engines like GPT-3 as its starting point: “World, mind, and learnability: On the metaphysical structure of the cosmos” (revised considerably for this paper). The other is about economic growth and stagnation: “Stagnation, Redux: Like diamonds, good ideas are not evenly distributed”.

These two very different papers nonetheless share both substance and method. Methodologically, both argue that the situation we observe is intelligible if we assume that the world is structured in a certain way. Their core substance is about that structure: the world must be “lumpy” – a notion I discuss on pages 5 ff. Because the world is lumpy we can learn about it, live in it, talk about it, and write about it. By contrast, a “smooth” world would be unintelligible and hence unlivable. The exhaustive statistical analysis of a large body of text is, in effect, able to recover that structural lumpiness as reflected in language and use it to produce new texts.

However, because the world is lumpy, we begin by learning and benefiting from things close to hand. When those resources have been exhausted we and must travel deeper into the world, expending more and more effort to extract economic benefit. Our current economic stagnation reflects the increasing cost of learning more about the world. A smooth world would no doubt be more convenient, for there would be economic benefit at every turn, either that or economic disaster. That is, if the world were smooth, we wouldn’t be here.

That, I know, this talk of smoothness and lumpiness is very abstract and “featureless”. But then could a resonance between such disparate phenomena as statistical semantics and economic stagnation but be abstract? I’ll provide more substance later in this paper, some diagrams, and some arguments. But first I want to suggest that what I’ve been calling lumpiness is what Ilya Prigogine and many others have called complexity.

Our complex world

Some years ago David Hays and I wondered why natural selection leads to complexity [1]. We argued that, over the long run, natural selection favors organisms with increased ability to process information, and that ability yields benefits in a complex universe. But what did we mean by that, a complex universe? Here is what we said:

It is easy enough to assert that the universe is essentially complex, but what does that assertion mean? Biology is certainly accustomed to complexity. Biomolecules consist of many atoms arranged in complex configurations; organisms consist of complex arrangements of cells and tissues; ecosystems have complex pathways of dependency between organisms. These things, and more, are the complexity with which biology must deal. And yet such general examples have the wrong “feel;” they don't focus one's attention on what is essential. To use a metaphor, the complexity we have in mind is a complexity in the very fabric of the universe. That garments of complex design can be made of that fabric is interesting, but one can also make complex garments from simple fabrics. It is complexity in the fabric which we find essential.

We take as our touchstone the work of Ilya Prigogine, who won the Nobel prize for demonstrating that order can arise by accident (Prigogine and Stengers 1984; Prigogine 1980; Nicolis and Prigogine 1977). He showed that when certain kinds of thermodynamic systems get far from equilibrium order can arise spontaneously. These systems include, but are not limited to, living systems. In general, so-called dissipative systems are such that small fluctuations can be amplified to the point where they change the behavior of the system. These systems have very large numbers of parts and the spontaneous order they exhibit arises on the macroscopic temporal and spatial scales of the whole system rather than on the microscopic temporal and spatial scales of its very many component parts. Further, since these processes are irreversible, it follows that time is not simply an empty vessel in which things just happen. The passage of time, rather, is intrinsic to physical process.

We live in a world in which “evolutionary processes leading to diversification and increasing complexity” are intrinsic to the inanimate as well as the animate world (Nicolis and Prigogine 1977: 1; see also Prigogine and Stengers 1984: 297-298). That this complexity is a complexity inherent in the fabric of the universe is indicated in a passage where Prigogine (1980: xv) asserts “that living systems are far-from-equilibrium objects separated by instabilities from the world of equilibrium and that living organisms are necessarily ‘large,’ macroscopic objects requiring a coherent state of matter in order to produce the complex biomolecules that make the perpetuation of life possible.” Here Prigogine asserts that organisms are macroscopic objects, implicitly contrasting them with microscopic objects.