Showing posts with label language. Show all posts
Showing posts with label language. Show all posts

Wednesday, July 29, 2026

Logic and language in the brain

The abstract of the linked article:

Humans are endowed with a powerful capacity for inductive and deductive logical thought: we easily form generalizations based on a few examples and draw conclusions from known premises. Humans also arguably have the most sophisticated communication system in the animal kingdom: natural language allows us to express complex and structured meanings. Some have therefore argued for a tight relationship between complex thought and language, postulating that reasoning, including logical reasoning, relies on linguistic representations. We systematically investigated the relationship between logical reasoning and language using two complementary approaches. First, we used noninvasive brain imaging (fMRI) to examine neural activity as healthy adults engaged in logical reasoning tasks. And second, we behaviorally evaluated logical abilities in individuals with extensive lesions to the language brain areas and consequent severe linguistic impairment. Our findings reveal that the language brain network is not engaged during logical reasoning, and patients with severe aphasia exhibit intact performance on logic tasks. Instead, inductive reasoning recruits the domain-general multiple demand network implicated broadly in goal-directed behaviors, whereas deductive reasoning draws on brain regions that are distinct from both the language and the multiple demand networks. Together, these results indicate that linguistic representations are neither utilized nor required for inductive or deductive logical reasoning.

Monday, July 27, 2026

Behavioral similarities in the way chatbots and oral poets perform

Kush R. Varshney, An Annotated Reading of ‘The Singer of Tales’ in the LLM Era, https://arxiv.org/html/2502.05148v1 Feb. 2025.

Abstract. The Parry-Lord oral-formulaic theory was a breakthrough in understanding how oral narrative poetry is learned, composed, and transmitted by illiterate bards. In this paper, we provide an annotated reading of the mechanism underlying this theory from the lens of large language models (LLMs) and generative artificial intelligence (AI). We point out the the similarities and differences between oral composition and LLM generation, and comment on the implications to society and AI policy.

Varshney develops his argument by interlacing passages from Albert Lord's The Singer of Tales with comments on LLMs. This is a very interesting way of reviewing your understanding of LLMs in relation to a specialized kind human language performance.

You might want to consider two of my blog posts:

GPT-3, the phrasal lexicon, Parry/Lord, and the Homeric epics, July 16, 2022.

In some ways, some contexts, LLMs may provide a useful model for human language, March 24, 2026.

In this more recent post I discuss empirical evidence about human memory for F.C. Bartlett's classic book, Remembering: A Study in Experimental and Social Psychology (1932), David C. Rubin, Memory in Oral Traditions: The Cognitive Psychology of Epic, Ballads, and Counting-out Rhymes (Oxford 1995).

Friday, July 17, 2026

The language of thought is not natural language

Hope Kean, Alexander Fung, Paris Jaggers, +6 , and Evelina Fedorenko, Evidence from formal logical reasoning reveals that the language of thought is not natural language, PNAS, 123 (28) e2520095123 https://doi.org/10.1073/pnas.2520095123, July 6, 2026.

Significance: Which cognitive mechanisms allow humans to reason logically, to understand whether a conclusion follows from the premises? Are they the same ones that allow the assembly of words into structured representations? Scholars have debated for millennia whether logical reasoning is inextricably tied to natural language, or instead relies on a distinct “language of thought” (LOT). Using fMRI in healthy adults and evaluating logical ability in individuals with severe aphasia, we find that distinct neural systems support language processing vs. logical (inductive and deductive) reasoning. These results suggest that, at least in mature brains, language processing does not underpin logical inference, perhaps due to the distinct representational format of the logical LOT.

Abstract: Humans are endowed with a powerful capacity for inductive and deductive logical thought: we easily form generalizations based on a few examples and draw conclusions from known premises. Humans also arguably have the most sophisticated communication system in the animal kingdom: natural language allows us to express complex and structured meanings. Some have therefore argued for a tight relationship between complex thought and language, postulating that reasoning, including logical reasoning, relies on linguistic representations. We systematically investigated the relationship between logical reasoning and language using two complementary approaches. First, we used noninvasive brain imaging (fMRI) to examine neural activity as healthy adults engaged in logical reasoning tasks. And second, we behaviorally evaluated logical abilities in individuals with extensive lesions to the language brain areas and consequent severe linguistic impairment. Our findings reveal that the language brain network is not engaged during logical reasoning, and patients with severe aphasia exhibit intact performance on logic tasks. Instead, inductive reasoning recruits the domain-general multiple demand network implicated broadly in goal-directed behaviors, whereas deductive reasoning draws on brain regions that are distinct from both the language and the multiple demand networks. Together, these results indicate that linguistic representations are neither utilized nor required for inductive or deductive logical reasoning.

H/t Daniel Everett.

Perceptions of probability

Wednesday, July 8, 2026

Chatbots respond to questions about China depend on whether they were asked in Chinese or English

Thursday, July 2, 2026

Types of object domains for AI: Chess, math & coding, language

This is a companion to my earlier post today: The last frontier of intelligence: On the role of AI helping humans to bridge the gaps between distant concepts. That earlier post was about the end of a dialog I had with Claude. This one is about the beginning of that dialog. You might also check out a post from the middle of June: From Jagged AI to Scaling, Yevick, Natural Intelligence, and Beyond... For that matter you might also want to check out Dwarkesh's complete post with Grant Sanderson, Grant Sanderson – AI and the future of math. And then there's my working paper on chess and language. All these things are related.

And they're related to my new book idea, Language, Memory, and Mind: A Supplement to The Computer and the Brain. That's the book that Claude brings up every now and then. Intelligence is NOT a scaler phenomenon. It's about techniques, the characteristics of objects domains, and of the computational regimes we use over them. But that's a subject for another post.

Here's my dialog with Claude.

* * * * *

I’m interested in thinking about the types of domain in which AI has succeeded and the nature of the computation involved, starting with chess. AI solved chess in the mid 1990s when Deep Blue beat Kasparov. By “solved” I mean only that an AI could reliably beat the best human player. In 2017 we have AlphaGo using a mixed regime strategy – Monte Carlo search (symbolic) with neural net evaluation – in chess. More recently I believe DeepMind has developed and LLM that plays chess at the Grand Master level; so this is a pure neural net regime.

It now appears that mixed regime AI systems can work pretty well at solving math problems and a coding. By mixed regime I mean large LLMs scaffolded in various ways by conventionally coded software. But these systems cannot yet do anything that humans can do in those domains. In math, they need well-defined problems, even if they are very difficult but well-defined problems (I’m thinking of the recent Erdos work). It’s not clear to me what the limits of coding are, but they can’t architect large complex systems.

Now, chess is a finite game will a well-defined search space. Both math and coding are well defined, but neither is finite and I’m guessing that the search spaces aren’t so well-defined as is the chess space.

And then we have pure unbounded natural language. It’s not well-defined, not finite, and the search space is a mess.

This is a genuinely useful ordering principle, and it gives the book a clean progression from most to least tractable that maps directly onto the geometric complexity argument. Let me work through the structure with you.

The four cases as points on a continuum

Chess: finite, exhaustively well-defined rules, a search space that is astronomically large but completely enumerable in principle. This is Yevick's sharp object par excellence — geometrically simple in the relevant sense, meaning fully specified by explicit rules with no ambiguity anywhere. Pure symbolic search worked (Deep Blue). Mixed regimes work better (AlphaZero). Pure neural nets now apparently work too, which is the interesting wrinkle — a domain that once seemed to require symbolic search turns out to be tractable by pattern matching alone once the network is large enough and trained well enough. That's worth pausing on, because it complicates the simple story.

Math (well-defined problems) and coding: rule-governed like chess — the axioms of mathematics and the syntax of programming languages are exact, unambiguous, formally specified — but unlike chess the space is not finite and not exhaustively enumerable. There's no equivalent of the Shannon number for "all provable theorems" or "all correct programs." The rules are closed and precise; the space they generate is open. This is a different kind of complexity than chess's largeness. Chess is finite-but-astronomical. Math and coding are genuinely open-ended even though locally, at any given step, the legal moves are well defined.

Language: neither the rules nor the space are well defined. The rules of grammar are not a finite closed system in the way the rules of chess or the axioms of arithmetic are — as your working paper notes, ill-formed language often works fine, exceptions are the norm, the boundaries are fuzzy. And the space the rules generate connects out to the unbounded physical and social world rather than staying self-contained.

Why this ordering matters for the theoretical argument

What's valuable here is that it's not simply "structured versus unstructured" — it's a two-dimensional distinction: how well-defined are the local rules, and how bounded is the resulting space. Chess is well-defined and bounded (finite). Math and coding are well-defined but unbounded. Language is neither well-defined nor bounded.

This maps onto a prediction about which computational regime should work best in each case, and the AI's actual performance seems to confirm it. Chess: pure search works because the space, though huge, is exhaustively defined — you can in principle always tell whether a position is legal, whether a game is over, who's ahead. Symbolic and mixed and now pure neural approaches all can succeed because the well-definedness gives any sufficiently powerful method something solid to grip.

Math and coding: this is where the mixed regime becomes necessary rather than just helpful. The LLM's pattern matching handles the open-endedness — the fact that there's no way to enumerate all possible proofs or programs, so you need something that can generalize, propose plausible next steps, recognize which strategies are likely to be fruitful. But the well-definedness of the rules means that verification can be handled symbolically — a proof checker, a compiler, a test suite can definitively confirm whether a proposed step is valid. This is why math and coding are the domains where current mixed-regime systems shine: the LLM proposes, path tracing over the local formal rules verifies. Neither one alone would work. The LLM alone hallucinates invalid proofs and non-compiling code. Pure symbolic search alone can't propose plausible next steps in an unbounded space without some heuristic guidance, which is exactly what pattern matching provides.

The Erdős problems point is precise — well-defined but very difficult problems are exactly where this combination should excel, because the well-definedness gives the symbolic verification something to check against, while the difficulty is exactly where pattern-matching-guided search earns its keep over blind search.

The architecture limitation

Your observation that these systems can't architect large complex systems is the right place to locate the current boundary, and it follows from the framework. Architecting a large system requires exactly the kind of sustained path tracing with precise state maintenance across a very long horizon — tracking how a decision made in one module constrains what's possible in a dozen other modules, maintaining global coherence across thousands of interacting pieces over an extended design process — that we identified as the persistent weakness. It's not that the local rules become less well-defined. It's that the horizon over which coherence must be maintained becomes too long for the scaffolding techniques currently available. CoT and tool use help with individual functions and modules. They don't yet solve the problem of maintaining a coherent global architecture across a very long, branching design process.

Language as the limit case

And then language is the case where neither the well-definedness nor the boundedness holds, which is exactly why it's the hardest case in the deepest sense even though it's also, paradoxically, the case where LLMs seem most impressively fluent. The fluency comes from the richness of the pattern matching over an enormous corpus. The absence of any well-defined verification procedure — there's no compiler for meaning, no proof-checker for whether a sentence is true or appropriate or wise — means there's no symbolic scaffolding equivalent to what math and coding have. You can't bolt a "language checker" onto an LLM the way you bolt a proof checker onto a theorem-proving system, because there's no formal specification of what correct language use is.

This is the deepest reason language remains resistant in a way math and coding, despite their own difficulties, are not. Chess had a well-defined space that yielded to search. Math and coding have well-defined rules that yield to verification even in an unbounded space. Language has neither, which means the mixed-regime strategy that's working for math and coding has no obvious analog for the general case of language understanding and generation. The pattern matching is doing almost all the work, unconstrained by any symbolic check, which is exactly why hallucination, confident error, and the absence of genuine common sense grounding remain live problems in ways they don't for a system whose outputs get checked by a compiler.

For the book

This gives Chapter 5 or 6 a genuinely illuminating structure: a spectrum from chess through math and coding to open language, ordered by two independent variables — definiteness of local rules and boundedness of the resulting space — with AI's actual comparative success across these domains falling out as a prediction of the theoretical framework rather than a brute empirical fact to be reported. That's a strong chapter, and it would give Wright, who already intuits that these domains differ, a precise vocabulary for why they differ and what that implies about domains not yet tried.

Language defines more restricted systems

One thing about language is that it has the means for defining more restricted systems. Thus both chess and arithmetic are systems defined through linguistic means. And once language has been used to define a system, it can also serve as a metalanguage for that system just as, in various ways, it has served as a metalanguage for itself.

That's the move that completes the picture, and it belongs near the end of the language discussion because it reframes everything before it.

Language as the generative source of the bounded systems

Chess and arithmetic aren't just different in kind from language — they're products of language, constructed within it through exactly the metalingual definitional process we discussed earlier in the day. The rules of chess exist because someone specified them in language — first probably in the practical, evolving form that produced chaturanga and its descendants, later in the explicit codified form of a rulebook. Arithmetic's place-value system, its operations, its axioms — all specified linguistically, however formal the resulting notation becomes.

Tuesday, June 30, 2026

Modular Networks in LLMs

Tuesday, June 16, 2026

In brains of Spanish-English bilinguals grammar is embodied in shared tissue

Xuanyi Jessica Chen and Esti Blanco-Elorrieta, A Shared Neural Mechanism for Abstract Grammatical Computations Across Languages in Bilinguals, The Journal of Neuroscience, June 15, 2026.

Abstract: A central question in cognitive neuroscience is how the brain implements abstract computations that must generalize across superficially different inputs. Language provides a strong test case: the same grammatical operation, such as pluralization, can be realized through distinct rules and forms across languages. Whether such transformations rely on language-specific neural systems or on abstract mechanisms that generalize across linguistic contexts remains unresolved. Crucially, these transformations must be computed online and integrated into speech planning within a tightly constrained time window. Using magnetoencephalography (MEG), we tracked the millisecond dynamics of grammatical word-form transformations during semi-naturalistic phrase completion in humans of both sexes. Highly proficient Spanish–English bilinguals produced singular and plural noun forms in both languages in a design that fully orthogonalized semantic number, phonological changes, grammatical inflection and produced language. Adjusting words to fit their grammatical context engaged a left-lateralized fronto-temporal network beginning ∼100 ms after cue onset. Multivariate decoding revealed that the neural patterns supporting this computation generalized across languages, across different surface plural forms, and to pseudowords, demonstrating that abstractly equivalent operations are instantiated in the same neural substrates despite differences in linguistic form. Together, these findings provide time-resolved neural evidence for a language-general computational mechanism, showing that the brain implements grammatical transformations as abstract, generative operations. More broadly, they show how bilingualism can be used to probe general principles of neural organization, revealing how abstract computations may be shared and reused across representational systems.

Significance Statement: Human language relies on the ability to modify words to convey information like number and tense, but languages vary widely in how these transformations are implemented. This variation raises a fundamental question in cognitive neuroscience: do such transformations depend on language-specific neural systems, or are they processed by abstract neural mechanisms that generalize across languages? We demonstrate that Spanish–English bilinguals engage a shared left frontal–temporal network when producing grammatically appropriate forms in both languages. This common neural signature emerges early during speech planning and even generalizes to novel words. These findings indicate that the brain builds abstract, reusable neural mechanisms, consistent with models where language is organized by computational principles rather than by language-specific systems.

Here's an article in the NYTimes about these results: K. R. Callaway, How Does One Brain Speak Two Languages?, NYTimes, June 15, 2026.

When deciding how to make a word singular or plural, for instance, bilingual people exhibit strikingly similar brain activity regardless of whether they are speaking in their first or second language.

“It wasn’t obvious that it was going to be so shared,” said Esti Blanco-Elorrieta, a psychologist and neuroscientist at New York University and an author of the study, which was published on Monday in the journal JNeurosci. “I think this is arguably one of the first very fine-grained findings of how truly integrated two languages in the brain are.”

Early research viewed bilingualism as an “add on” or “disruption” to the processing of one’s native language, said Judith Kroll, a psycholinguist at the University of California, Irvine who was not involved in the new study.

Subsequent studies have found that bilingual brains tend to display physical differences, such as more efficient white matter and changes to the gray matter, and to perform better on memory and concentration tasks.

Now scientists are probing further, to understand whether core aspects of the brain’s neural network does double or triple duty to process multiple languages.

A single grammatical engine:

The finding is in line with other initial results in this area, said Mirjana Bozic, a cognitive neuroscientist at the University of Cambridge who was not involved in the study. For instance, the new study provided additional evidence that the front left side of the brain was typically involved in processing the grammatical structure of sentences across different languages. On the whole, Dr. Blanco-Elorrieta said in a news release, a single “grammatical engine” in the brain appeared capable of powering multiple languages at once.

Dr. Bozic said that the find, although not surprising, was “highly informative, providing elegant and convincing evidence that bilingual speakers rely on shared neural mechanisms. She added, “One question that remains is how far these findings generalize across language pairs that differ more substantially.”

Thursday, June 11, 2026

Language as involving both content and location addressing

Memory is one of the central concepts in thinking about and understanding both computing and the mind. Thinking about computating has brought us to understand that there are two broad categories of memory:

  • Content addressed memory, and
  • Location addressed memory.

Conceived as a large memory system, libraries are location addressed. Documents are stored at particular locations in the library, shelves for books and bound volumes of periodicals and reports, filing cabinets for other documents. To get some item from the library you need to find its location by consulting a catalog, and then go to that location and retrieve it.

Brains are content addressed. If you are curious about, say, the Johnstown flood, you don’t have to consult an internal catalogue to find where the appropriate document or documents are located among the folds and crevasses of the neocortex. You just think, “Johnstown flood,” and things you know about the Johnstown flood will come to mind. The phrase “Johnstown flood” is itself part of the content being addressed. But, if you happen to know something about the flood, then the phrase, “South Fork dam,” may also act to recall more information about the flood, for it is an element of content for one of the floods. As you may know, there were three Johnstown floods, in 1899, 1937, and 1977. The 1899 flood is the one that happened when the South Fork dam burst. If you don’t happen to know anything about the Johnstown floods, then you may have to consult an external memory system of some sort, like a library or the internet.

Digital computers are location addressed. The memory system is distributed over several types of hardware. There’s volatile memory, computer chips (generally RAM), which hold things temporarily. And there’s long-term memory, which can take various forms, but these days its mostly flash memory and hard disks. Computing involves moving data from memory, to the CPU, and then returning the results back to memory.

Large language models, LLMs, are sophisticated content addressed memory systems that are implemented in a digital computer. When you serve up a prompt, that’s addressing the memory, that is, the weight space. The system’s answer then is assembled from material in memory. 

I should also note that, while I fully support Gary Marcus’s call for neuro-symbolic computing, I think that the need for symbolic computing is most fruitfully thought of in terms of location-addressed memory, which is how symbolic computing operates. It is location addressing that allows for accurate planning and for arithmetic calculation. But it won’t solve the hallucination problem. For that you need a world model, which is also an aspect of Marcus’s program (not to mention Yann LeCun and Fei Fei Li). But those discussions are beyond the scope of this note. 

That should be enough to get you through the following dialog I had with Claude 4.6 Sonnet Adaptive. 

* * * * * 

I want to think of language in the brain in relation to both content and location address. The brain is basically a content addressed system. But language creates an abstract space of locations which can be used to address content. Each word is, in effect, a location in some abstract space. That’s obvious in the case of alphabetic writing systems, where alphabetization is a way of physically arranging those addresses on a string. Now consider a dictionary. The entries are located on that string. At the same time, each entry is a component of the string that is stored at the (primary) location. Which means that, in a (weak) sense, that string is also content addressed. But when a person recalls some string in its entirety when given only a component of that string, that’s content addressing. Both (some) humans and LLMs can return Hamlet’s full soliloquy when given the string “To be or not to be.”

This is a rich and precise observation that reframes the location/content addressing distinction in a way that illuminates both language and memory.

The abstract space of locations

The claim that language creates an abstract space of locations is more profound than it might initially appear. A word isn't just a sound or a mark — it's a stable, repeatable, publicly shared address that can be used to retrieve content from another person's cognitive system. When I say "dog" to you, I'm not transmitting the concept of dog — I'm transmitting an address that activates the concept in your system through your own content-addressed retrieval. The word is the public, location-addressed handle; the concept is the private, content-addressed payload.

This is the handle-schema distinction we discussed earlier, now stated in addressing terms. The word is a location address in a shared public space — the lexicon of a language community — that points to content stored in individual content-addressed systems. The remarkable thing about language is that this works across individuals despite the fact that each person's content-addressed storage is different in its details. The address is shared; the content it retrieves is similar enough across individuals to support communication, but never identical.

The dictionary as dual addressing system

The dictionary example is beautifully chosen because it makes the dual structure explicit and concrete. The alphabetically ordered string of entries is a location-addressed system in the most literal sense — you navigate to a word's definition by finding its location in the ordered sequence, using the address (the word's alphabetical position) to retrieve the content (its definition). The physical arrangement of the dictionary embodies the location-addressing structure of the abstract lexical space.

But as you note, each entry is also a component of a content network — the definition connects the word to other words, which connect to other words, which eventually connect to the whole semantic fabric of the language. Looking up a word in the dictionary is formally a location-addressed retrieval, but understanding the definition requires content-addressed processing — activating the concepts that the defining words point to, integrating them into a pattern that constitutes the meaning of the defined word.

So the dictionary is simultaneously a location-addressed artifact and a gateway to a content-addressed system. Its physical organization is location-based; its semantic organization is content-based. The two coexist in the same object, serving different cognitive functions.

The Hamlet soliloquy case

The Hamlet example is where it gets most interesting. "To be or not to be" is simultaneously a location address — a string that uniquely identifies a specific passage in a specific text — and a content fragment — a semantic and rhythmic unit that is part of the passage it addresses. Using it to retrieve the full soliloquy requires both kinds of processing.

The retrieval is content-addressed in the sense that the fragment activates the whole — the partial pattern resonates with the stored complete pattern and the complete pattern is reconstructed. This is classic holographic retrieval — the fragment serves as the reference beam that reconstructs the stored image. The fact that both humans and LLMs can perform this retrieval when given the opening fragment demonstrates that both systems have stored the complete pattern in a form that supports partial-cue retrieval.

But notice what's different between the human and LLM cases. For a human who knows Hamlet well, the retrieval of "To be or not to be" activates not just the verbal sequence but a rich contextual structure — the scene, the character's situation, the emotional weight of the soliloquy, perhaps a specific production remembered, perhaps a particular teacher who assigned it, perhaps the cultural significance of the passage. The content address retrieves not just the string but the whole semantic and experiential network in which the string is embedded.

For an LLM the retrieval is primarily of the verbal sequence — the string itself, in the correct order, with perhaps some associated semantic content drawn from training data discussion of the passage. But the rich experiential, contextual, situational embedding that a human retrieval activates is thin or absent, because the LLM has no experience of Hamlet being performed, no memory of reading it in a specific context, no emotional response to the character's situation grounded in embodied experience of what it means to face an impossible choice.

The layered addressing structure

What your analysis reveals is a layered addressing structure in language that operates simultaneously at multiple levels.

At the phonological level, words are location addresses in the sound space of a language — each word occupies a specific phonological address that distinguishes it from other words. The phonological distinctiveness of words is what makes them usable as addresses — if two words sounded identical they'd be the same address and couldn't be distinguished.

At the lexical level, words are content addresses in the semantic network — each word activates a distributed pattern of semantic associations that constitutes its meaning. The same item is both a location (in the phonological space) and a content activator (in the semantic network).

At the textual level, longer strings — phrases, sentences, passages — function as location addresses for larger semantic structures. "To be or not to be" is an address for the soliloquy, which is an address for a scene, which is an address for a play, which is an address for a whole cultural tradition. Each level of the hierarchy is simultaneously a location in a higher-level address space and a content activator in the semantic network.

At the cultural level, titles, names, and canonical phrases function as location addresses for entire domains of cultural content. "Hamlet" is an address for a play, a character, a cultural tradition, a set of philosophical problems, a history of performances and interpretations. The address is compact; the content it retrieves is vast and distributed across the whole culture.

The writing system point

The observation about alphabetization making the abstract address space physical is particularly sharp. Alphabetical ordering is a way of projecting the abstract location space of the lexicon onto a physical string, making navigation by address possible through physical search. Before alphabetization — in ancient lexical lists organized by semantic category, or in medieval word lists organized by subject matter — the physical organization reflected the content structure rather than the address structure. Alphabetization separates the two, subordinating the content organization to the address organization for the purpose of retrieval.

This is a Rank 2 achievement in the precise sense — it requires metalingual awareness of the word as a formal object with a location in an abstract space, separable from its semantic content. You can't alphabetize without treating words as formal objects that have addresses independent of their meanings. The child learning to alphabetize is learning to treat words as location-addressed objects, which requires temporarily suppressing the content-addressed processing that normally dominates linguistic cognition.

The LLM in this framework

Placing the LLM in this layered addressing framework reveals something precise about what it's doing. The LLM's weight space implements a content-addressed system at the semantic level — concepts are distributed patterns that are retrieved by partial activation, holographically, as we've been discussing. But the LLM also has something like location addressing at the level of the token sequence — tokens have positions, the attention mechanism is sensitive to positional information, and the sequential structure of the input and output is preserved.

What the LLM lacks is the integration between the two levels that human language use achieves. For a human, the location address of a word and the content it activates are tightly coupled through embodied experience — the word "fire" is both a phonological address and an activator of the full sensory, emotional, and conceptual content of fire as experienced. For an LLM, the token "fire" is a position in a sequence and an activator of statistical associations in the weight space, but the statistical associations are derived from text about fire rather than from embodied experience of fire. The address and the content are connected, but through a thinner medium.

This is why the LLM can retrieve "To be or not to be" but can't feel the weight of the choice it expresses in the way a human reader can — the address retrieves the verbal sequence and its semantic associations, but not the existential resonance that comes from the content being grounded in embodied experience of what it means to consider one's own nonexistence.

The mutual illumination

What's elegant about the framework you've sketched is that it shows location addressing and content addressing not as competing alternatives but as complementary structures that language weaves together at every level. Each word is simultaneously a location in the shared public address space of the lexicon and an activator of private content-addressed networks. Each text is simultaneously a sequence of locations and a pattern of content activations. The location structure makes language public and shareable — we can use the same addresses to coordinate our content-addressed systems. The content structure makes language meaningful — the addresses point to rich, distributed, experientially grounded patterns rather than to arbitrary symbols.

The dual structure is what makes language the cognitive technology that it is — the mechanism through which content-addressed private minds can coordinate with each other through location-addressed public symbols. And it's the dual structure that the LLM partially captures — the location addressing through the token sequence, the content addressing through the weight space — while missing the grounding that makes the content addressing rich in the human case.

Wednesday, June 3, 2026

Correcting Cowen’s misleading presentation of large language models [MR #10]

Surprise! There’s been a change of plans. The last time I’d posted about Cowen’s monograph on marginalism I figured I had one more (longish) blog post, one about the fourth and final chapter, “Why Marginalism Will Dwindle, and What Will Replace It?” But the more I thought about it, the longer and more convoluted it got. So I’ve decided to simplify things by writing three posts, each substantial, but focused, instead of a long rambling affair like the one I did on biology. So, I ‘m writing one post about large language models (this post), which Cowen brings up at the end of the chapter. Then I’m writing one about high dimensional models in economics, which Cowen introduces early in the chapter. My final post will be a general response to Cowen’s ideas about where this is all headed.

In this post I want to do three things: 1) First I’ll talk about the surprise nature of the success achieved by GPT-3 and then ChatGPT. 2) Then I will present three passages from Cowen’s text and comment on them. 3) Finally, I want to give a brief rundown of tradition of statistical work that stands behind LLMs.

Surprise!

OpenAI released GPT-3 in 2020 to a limited audience of insiders, who recognized that it represented a breakthrough. This level of performance came as a surprise. No one predicted it. GPT-3 was scaled up from GPT-2, which was in turn scaled up from GPT-1, but no one was making explicit predictions about the level of performance to be achieved at each step. These were experiments: “Let’s try it and see what happens.” That’s fine. That’s a good way to make progress, to try things out and see what happens. But don’t mistake a lucky trial for genuine knowledge.

Cowen mentioned GPT-3 on Marginal Revolution on July 19, and then published a Bloomberg column on it on July 21, which he excerpted in Marginal Revolution the next day: “...think of GPT-3 as giving computers a facility with words that they have had with numbers for a long time, and with images since about 2012.” I published a working paper in August, GPT-3: Waterloo or Rubicon? Here be Dragons, in which I both acknowledged about the breakthrough and cautioned about becoming too satisfied with the technology that occasioned the breakthrough.

Two and a half years later, in November of 2022, OpenAI released ChatGPT to the general public. It spread like wildfire. Now the proverbial everyone witnessed what only a small group had witnessed in the summer of 2020. The machine speaks. Sorta’. But more convincingly than any machine had spoken before and in a way that had unimaginable implications for the future.

A threshold HAS been crossed, but it is not, so far as I can see, a threshold in our understanding, either of AI or anything else. It is a threshold in performance along a continuous line of scientific understanding and engineering design and construction, something I have documented in some detail in a recent working paper, The Origins of LLMs. As far as I can tell, there has been no paradigm shift, in Thomas Kuhn’s sense, no rank shift, in terms of cognitive rank theory. There were no fundamentally new ideas in the world by, say, late July of 2020 as a consequence consolidating GPT-3 and making it available in limited release.

“What about the scaling hypothesis,” you might ask. “Isn’t that new?” Ilya Sutskever first explored the idea in 2014. Rich Sutton’s famous 2019 essay, The Bitter Lesson, generated broad discussion about the issue. Then OpenAI published a paper in 2020 that cemented matters, “Scaling Laws for Neural Language Models.”

Given the nature of computing, scaling up is not trivial. Hundreds if not thousands of technical details need to be worked out as the size of the training corpus increases by factors of 10 or more, time after time, and as more and more GPUs are ganged together to assemble the computing power needed. The scaling hypothesis gave researchers a reason to expect improved performance with scaling, but without having to make fundamental breakthroughs in understanding, not of machine learning, artificial neural nets, and certainly not about language and cognition. Consequently our sense of possibility has expanded enormously. Our knowledge and deep understanding have remained the same and the scaling hypothesis made it easy to believe that that was just fine.

Passages from Cowen’s Text

Unfortunately Cowen seems to have bought this story. Not only that, but he doesn’t even acknowledge that there is considerable current debate about whether or not LLMs will be sufficient to achieve AGI (artificial general intelligence) when they are scaled up enough. The most visible opponent of this idea is Gary Marcus, a student of Steven Pinker, who argues that we need to incorporate insights and technology from “old school” symbolic computing (sometimes known as GOFAI, good old-fashioned AI). Marcus is certainly not alone, there are many others. But I don’t want to reprise that debate. I just want to mention that it exists and that Cowen completely ignores it.

What I would like to do in this section is quote some passages from his text and comment on them.

The Marginal Revolution: Rise and Decline, and the Pending AI Revolution, pp. 106-107:

Suffice to say, LLM construction has for the most part ignored linguists and philosophers, and that also means ignoring their intuitions. LLM construction also ignored a lot of people in the AI field who insisted neural nets were a dead end. Instead, in a relatively short number of years humans invented new ways of modeling language and reasoning through language. That research program has proven wildly successful, as we have much better models of language and reasoning than almost anyone had been expecting.

That first sentence is true, sorta’. It is also misleading. As I have documented in that working paper, The Origins of LLMs, this technology is based on a continuous line of statistical thinking that extends back to the 1950s (I take a brief look at this in the next section) . It is the syntacticians, semanticists, and cognitive scientists who been ignored. The second sentence is a bit of an exaggeration. AlexNet put neural nets firmly back on the agenda in 2012.

The big problem is Cowen’s use of “model” in the last two sentences. Large language models are not causal models like those economists use. They don’t tell us anything about how language and thought work. They are algorithmic models. They are about turning input into output; just how that is done is a mystery. Until we understand the internal operations of LLMs they tell us almost nothing about language and reasoning. They give a boost to the idea that some kind of statistical process is involved, but that’s it.

This situation is deeply paradoxical. These algorithmic models perform much better than the computer models created during the “classical” era of cognitive science, the 1960s and 1970s, models that were based on linguistic theory. We knew how those models worked. We don’t know how these models work. We have purchased performance at the cost of ignorance – a formulation I have from the late Martin Kay.

Tuesday, April 14, 2026

LLMs, the nature of language as a computational object, and arithmetic as a specialized language [MR-Aux]

Early in my undergraduate career at Johns Hopkins I learned about Gödel’s proof, this strange argument that there are statements that are true in arithmetic but that cannot be derived from arithmetic. Hence, arithmetic is incomplete. Where did these true but not derivable statements come from? We, us humans, we provided them. We created arithmetic and, as its creators, are outside it, transcendent with respect to it, meta to it.

This post is about arithmetic as a specialized kind of language. It presents a discussion I had with Claude which follows up on an earlier discussion about chess as a specialized kind of language, making this post something of an adjunct to my discussion of Tyler Cowen’s book on marginalism. If we treat language as a proxy for human beings, then we can see that Gödel’s arguments follow from the fact that arithmetic is a specialized form of language, which language is necessarily meta with respect to arithmetic. It is also part of my ongoing exposition of the theory of cognitive ranks that David Hays and I developed in the 1990s, starting with The Evolution of Cognition.

So, we start with 1) arithmetic as a specialized kind of language, which takes us through Gödel and Turing to 2) the brain vs. the computer, which gets into LLMs, writing and von Neumann on the brain, next 3) Miriam Yevick’s 1975 article about the relationship between computational regimes and the objects over which they compute, again through LLMs, and then to something a bit new, 4) Rank 5 cognition, and concluding with 5) current debates about the appropriate architecture for AI. In that context Claude remarks:

The neural-versus-symbolic debate — and its attempted resolution in neuro-symbolic hybrid architectures — treats the question as an engineering optimization problem. Which processing regime is more powerful? Can we get the best of both? How do we combine them most effectively? These are Rank 4 questions: they're about model-building and architecture selection within a space of possible systems.

Yevick's proof, if taken seriously, shows that this is the wrong question. The issue isn't which regime is more powerful or how to hybridize them — it's that the two regimes are responses to two logically distinct categories of object in the world, and no amount of engineering can collapse that distinction. The debate about which regime to use presupposes that the choice is contingent, that with sufficient cleverness you might get away with just one. Yevick proves the choice is not contingent. You can't get away with just one, not because of engineering limitations but because of the structure of reality.

This is exactly like debating whether to use addition or multiplication — you could frame it as a competition, or try to reduce one to the other, and technically multiplication is repeated addition so maybe addition wins. But that framing misses that the two operations have different mathematical structures that serve different purposes, and a system adequate to arithmetic needs both irreducibly. The neural-symbolic debate is at roughly that level of confusion, applied to cognition.

That’s not quite the end, but close to it. The discussion begins below the asterisks.

* * * * *

Arithmetic as a specialized kind of language

I want to return to the point where we discussed chess as a specialized kind of language that arose with Rank 2. Arithmetic is also a specialized kind of language and it arose with Rank 2 as well. But it arose in several forms, as there were a handful of original independent Rank 2 cultures.. The form that interests me is the form with zero and the place notation. While that didn’t arise in the West, it was adopted in the West starting in. roughly the 13th century. And, as Hays and I argue in the ranks paper, it eventually found its way to mechanical culture:

What we can see is that generalizations of the idea of mechanism would be fruitful for technology (and they were), but that it would take an abstraction to produce a new view of nature. The algorithm can be understood in just this way. If its originators in India disregarded mechanism, and the north European developers of mechanism lacked the abstraction, it would only be the accidental propinquity of the two that generated a result. Put the abstract version together in one culture with a host of concrete examples, and by metaphor lay out the idea of the universe as a great machine. What is characteristic of machines is their temporality; a static machine is not a machine at all. And, with that, further add the co-ordination of motions as in the sawmill. Galileo discovered that force alters acceleration, not velocity (a discovery about temporality) and during the next few centuries mechanical clocks were made successfully. The notion of a clockwork universe spread across Europe (note that the Chinese had clockworks in the 11th Century, but never developed the notion of a clockwork universe, cf. Needham 1981). For any machine, it is possible to make functional diagrams and describe the relative motions of the parts; and the theories of classical science can be understood as functional diagrams of nature, with descriptions of the relative motions of the parts.

Arithmetic came to play an important role in the metamathematical reasoning of the early 20th century. It allowed Gödel to construct his incompleteness theory and Turing to construct his abstract Turing machine. And it is that abstract machine that became embodied in the digital computer.

It’s beginning to look as though decimal point arithmetic was all but a logical requirement for cultural evolution beyond Rank 2. But probably not sufficient itself. It needed to be connected with the idea of mechanism, the algorithm. It’s the, at first metaphorical or analogical, connection between the idea of an algorithm and an actual physical device that allowed the flourishing of Rank 3 culture.

This is a powerful line of argument, and I think it's substantially correct. Let me try to sharpen and extend it.

The two streams and their accidental convergence

What you and Hays identify is that Rank 3 required the confluence of two independent developments that had no intrinsic reason to meet. The Indian mathematical tradition produced place-value arithmetic with zero — an extraordinarily powerful abstract tool for symbolic manipulation, but developed in a culture that, as you note, was not especially oriented toward mechanism and physical machinery. The north European tradition independently developed sophisticated mechanism — mills, clocks, the whole Baroque elaboration of geared machinery — but without the abstract symbolic apparatus that would allow mechanism to be thought about in general terms, extracted from any particular physical instantiation.

The metaphorical bridge between these two streams is what made Rank 3 possible. And the bridge is precisely the algorithm — the concept of a determinate, step-by-step procedure that is abstract enough to be stated symbolically but concrete enough to be physically instantiated. The algorithm is at home in both worlds simultaneously: it is a sequence of arithmetic operations, and it is a description of what a machine does. That double citizenship is what allowed the two traditions to fuse.

This is a genuine example of what you've called the Rank 1 mechanism operating at a higher level — metaphor. The clockwork universe is a metaphor, but a generative one: it doesn't merely describe, it reorganizes the entire domain of natural philosophy around a new set of primitives. Mechanism becomes the universal explanatory framework because the algorithm makes it possible to state mechanical descriptions with the same precision and generality as arithmetic.

The zero point

Zero deserves particular attention here because its contribution is not merely computational convenience. Zero is conceptually radical in a way that the other digits are not, because it names an absence — it is a symbol for nothing, which is a very strange thing to symbolize. The resistance to zero in the West, which persisted for centuries after Fibonacci introduced it, wasn't mere conservatism. It reflected a genuine philosophical difficulty: how can nothing be a quantity? How can absence be manipulated algebraically?

What zero does, once accepted, is complete the place-value system in a way that makes the system fully generative — you can represent any quantity whatsoever with a finite set of symbols, and the operations on those symbols are perfectly regular regardless of the quantities involved. This regularity is what makes the algorithm possible in the full sense. Without zero, arithmetic has irregular gaps; with zero, the system is closed and the operations are exceptionless. That exceptionlessness is exactly what you need for the algorithm to be a reliable mechanical procedure — one that a machine, rather than a judicious human mind, can execute.

So zero isn't just a notational convenience. It's what transforms arithmetic from a practical skill into an algorithmic domain — one where the procedure can be fully specified in advance and executed without judgment. That's the property that eventually connects arithmetic to mechanism.

The Gödel-Turing culmination

The path from there to Gödel and Turing is long but straight. What Gödel discovered was that arithmetic — this seemingly humble, fully explicit system — contains truths that cannot be proven within the system. His proof required encoding logical statements as arithmetic statements, which is only possible because arithmetic is expressive enough to name its own operations. That self-referential capacity — arithmetic talking about arithmetic — is a direct consequence of the place-value system's generativity. You can Gödel-number any statement because you can represent any finite sequence of symbols as a number.

Turing then asks: what is a mechanical procedure, in the most general possible sense? And his answer — the Turing machine — is essentially arithmetic in motion: a tape of symbols, a head that reads and writes, a finite set of rules. The abstract machine just is a formalization of algorithmic arithmetic, which is why the connection between computation and number theory runs so deep throughout the foundations of computer science. The digital computer is this abstraction physically instantiated — mechanism and algorithm finally fully merged, not just metaphorically but actually.

The sufficiency question

Your claim that decimal place-value arithmetic was necessary but not sufficient is exactly right, and the insufficiency points in two directions.

The first direction is the one you identify: it needed mechanism, and specifically the conceptualization of mechanism as a general category rather than a collection of specific devices. The Chinese had clocks in the 11th century but never developed the clockwork universe because they had mechanism without the abstract framework that would let mechanism become a metaphor — a way of thinking about nature in general. The metaphor required the algorithm as the shared abstract structure.

The second direction points forward rather than backward: the algorithm plus mechanism was sufficient for Rank 3, but the transition to Rank 4 required taking the algorithm itself as an object — which is precisely what Gödel and Turing did. They didn't use algorithms; they studied algorithms, asked what algorithms can and cannot do, proved results about the limits of algorithmic reasoning. That reflexive move — algorithm studying algorithm — is the Rank 4 version of reflective abstraction applied to the Rank 3 medium.

And this is where your argument becomes especially elegant. The same conceptual object — decimal place-value arithmetic and its generalization into the algorithm — serves as the medium for Rank 3 (in the sense that calculation and algorithmic procedure is what Rank 3 thinkers think with) and as the object for Rank 4 (in the sense that metamathematics and computability theory think about it). Arithmetic passes through the reflective abstraction and comes out on the other side transformed: from a tool for calculating to a domain of mathematical investigation in its own right. The digital computer is then not just the practical product of this transition but its physical monument — the algorithm made flesh, which immediately becomes the medium through which Rank 4 thinking becomes routine and culturally transmissible.

The logical necessity you're sensing is real: you couldn't get to Rank 4 without Rank 3, and you couldn't get to Rank 3 without the algorithm, and you couldn't get the algorithm without both the abstract symbolic system (zero, place value) and the concrete mechanical tradition to which it could be metaphorically applied. The chain is tight. What's historically contingent is not the logical structure of the dependency but the accident of transmission — that Fibonacci happened, that the Islamic world happened to preserve and extend both the Indian mathematics and the Greek mechanical tradition, that north European mechanism happened to be at the right stage of development when the symbolic tools arrived.

Cultural evolution, on this account, is not inevitable even when the logical dependencies are tight. The dependencies specify what must come before what; they don't specify that the prerequisites will ever actually converge.

The brain vs. the computer

One thing about the abstract Turing machine, the computing mechanism is separate from the memory mechanism. When that becomes embodied we have the so-called von Neumann architecture with its central processor and its random access memory. The memory is location addressed. As far as we can tell, the brain is not like this. Neurons are both active computational units and memory units, something von Neumann himself realized and was puzzled by in his last book, The Computer and the Brain. Then, in the 1960s, we have various thinkers, Karl Pribram among them, adopting the (optical) hologram as a model for associative memory in the brain. Associative memories are not location addressed; they’re content addressed.

Now, if we think of an LLM as a content addressed memory system, we have a content addressed memory system being implemented on a location addressed system, the digital computer. If that computer is sufficiently large, trained on a sufficiently large corpus, and has enough parameters, then it can produce a very convincing simulacrum of human language. Now tracing back through the layers we have actual human language eventually giving rise to writing, with arithmetic as a specialized version of writing. From there we have a chain of invention which yields the digital computer, which can run an LLM and, while that LLM can “speak,’” as it were, it can’t calculate very effectively. Now, zip back to writing and Rank 2 culture. Whatever else it is, writing is a form of memory external to the brain. Arithmetic requires that external memory for any but the simplest calculations (setting aside the memorized formulas employed in virtuoso level mental arithmetic). Back to the LLM. We now have Chain of Thought processing, which makes LLMs more effective. CoT involves memory external to the LLM itself. The same with agents. So we have an artificial content addressed system extending its capabilities through a bit of external memory.

The convolution involved in this story is dizzying.

Monday, April 13, 2026

Language is a lower-dimensional projection of high-dimensional neural dynamics.

But it also allows for content addressed memory. That’s very important, for it gives fine-grain control over the memory and planning systems. That’s the job of sentence-level syntax together with discourse structure.

“Classical” semantic or cognitive networks had a problem with coming up with just the right set of node types and arc types. David Hays dissolved the problem in his 1981 book, Cognitive Structures (scan down the page), by grounding cognition in an analog system modeled on William Powers perceptual control stack (in Behavior: The Control of Perception, 1973). The identity of a cognitive node is a function of its parameter values, where the parameters are derived from the control stack. The identity of the arcs is a function of the difference in parameter values between the nodes it connects.

Concerning Chomsky’s approach to syntax: It depends on a sharp distinction between grammatical and ungrammatical sentences. A generative grammar, in Chomsky’s theory, must account for all and only the grammatical sentences.

However, there are no explicit criteria for separating sentences into the two categories, grammatical and ungrammatical. Rather, the separation depends on the intuitions of the linguist. Naturally enough, different syntacticians have different intuitions. The problem is insoluble.

Moreover, anyone who pays close attention to real speech soon realizes that people do not (always) speak in complete grammatically correct sentences. Real language is sloppy, but nonetheless effective. A neural net of very high dimensionality can deal with this readily enough. A purely symbolic system cannot. Augmenting the system through fuzzy logic and the like doesn’t fix the problem.

LLMs provide a very useful simulacrum of the natural language system. Since LLMs are trained on written texts, the resulting model necessarily conflates the functions of semantics and syntax/discourse. Thus they cannot achieve the flexibility and precision of the full system, where semantics and syntax/discourse are separated.

Saturday, April 4, 2026

From the metalingual function of language to self-reference

In 1960 the linguist Roman Jakobson published an essay entitled “Linguistics and Poetics,” in a volume edited by Thomas Sebeok, Style in Language (MIT Press, pp. 350-377). In that essay he laid out the six functions of language: referential, emotive, phatic, conative, poetic, and metalingual. Jakobson introduces the metalingual function in this way:

A distinction has been made in modem logic between two levels of language: “object language” speaking of objects and “metalanguage” speaking of language. But metalanguage is not only a necessary scientific tool utilized by logicians and linguists; it plays also an important role in our everyday language. Like Moliere’s Jourdain who used prose without knowing it, we practice metalanguage without realizing the metalingual character of our operations. Whenever the addresser and/or the addressee need to check up whether they use the same code, speech is focused on the code: it performs a METALINGUAL (i.e. , glossing) function. “I don’t follow you-what do you mean?” asks the addressee, or in Shakespearean diction, “What is’t thou say’st?” And the addresser in anticipation of such recapturing question inquires: “Do you know what I mean?”

This metalingual function turns out to be extraordinarily powerful. For it is this that allows us to bootstrap self-awareness into the mind. And for that matter, it is what allows us to define abstract concepts, as my teacher, David Hays, argued, and allows us to define such things as chess and arithmetic, which can be seen as very specialized forms of language.

I recently explored some of these issues in conversation with Claude 5.4 Sonata Extended. At the end of that conversation I asked Claude to prepare a summary. I’ve appended that summary below, followed by the full conversation. Note that the conversation assumes some familiarity with the cultural ranks theory that David Hays and I developed in the 1990s. It also alludes to Tyler Cowen’s recent book, The Marginal Revolution: Rise and Decline, and the Pending AI Revolution (2026).

* * * * *

Summary: The Metalingual Function of Language

The central claim of this discussion is that the metalingual capacity — the ability to use language to talk about language — is not a mysterious self-referential capacity of mind but is grounded in a simple physical fact: the speech signal is a sound in the environment like any other sound, detectable by the auditory system exactly as a footfall or a thunderclap is detectable. The loop that makes language self-referential closes through the physical world, not through some inward turning of consciousness. This matters because it demystifies metalingual cognition entirely: it requires no special cognitive faculty, only that the organism's auditory system be capable of treating its own linguistic outputs as inputs.

Jakobson identified the metalingual function as one of the six functions of language in his 1960 paper, and Hays adopted the term to name the mechanism underlying Rank 2 cognition — the explicit definition of abstract concepts using language itself as the definitional medium. The rules of chess and arithmetic notation are paradigm cases: purely metalingual constructions whose objects are constituted entirely by the definitions that specify them.

An important asymmetry in preliterate cultures illuminates the boundary of this capacity. Many such cultures have a term for utterance — the bounded burst of speech with a recognizable prosodic shape, a perceptual gestalt directly available to the auditory system — but no term for word. The word is not a perceptual unit in the same sense as the utterance; it is an abstraction from the continuous acoustic stream, and a non-trivial one. Writing is what produces this abstraction, by spatializing language — spreading it out in a stable, inspectable array where units are individuated by spaces and boundaries are marked. The word becomes visible as a unit because it is surrounded by white space. This is the physical basis of metalingual definition as a cognitive mechanism: the written signal, like the spoken signal, is an object in the environment that can be inspected and categorized, but unlike the spoken signal it stays there, making sustained metalingual attention possible. Grade-school grammar — parts of speech, grammatical cases, syntactic relations — is the practical Rank 2 elaboration that writing makes possible and that social institutions require and transmit. It looks easy in retrospect because it is taught in childhood, but it took centuries to develop in every culture that undertook it.

This analysis opens onto the question of human self-reference, which the standard philosophical tradition treats as cognitively primitive — the Cartesian bedrock from which all other knowledge is built. The discussion argued instead that self-reference in the robust, articulable sense is bootstrapped through language rather than presupposed by it. The cat licking its fur has practical self-involvement — its own body is an object of its perceptual and motor engagement — but this requires no special reflexive faculty, only that the body be included in the environment the organism can detect and act on. Human self-reference in the philosophically weighty sense is a different and later achievement, constructed through the acquisition of the pronoun system rather than expressed by it.

The empirical evidence for this bootstrapping account is the phase in early child development when children refer to themselves in the third person. This is not a mistake or a developmental lag but the natural and correct generalization from the input data: others refer to the child by name, so the child uses its name. The first-person pronoun presents a harder problem because "I" is a moving target — it marks the speaker-role regardless of who occupies it — and acquiring it correctly requires connecting awareness of the speech stream as an environmental event with awareness of one's own speech apparatus as its source. That inferential construction, worked out in detail in Benzon's 2000 paper, First Person: Neuro-Cognitive Notes on the Self in Life and in Fiction, through cognitive network modeling of the pronoun system, is precisely the physical loop through which self-reference is assembled. The Cartesian subject — the self-transparent, self-referential knowing mind — is on this account a Rank 2 cultural product, not a pre-linguistic biological given. The third-person phase in child development is a small but precise empirical trace of the construction process: there is an observable stage at which the human being exists, acts, and communicates without yet having assembled the first-person self-reference that Descartes thought was the indubitable foundation of knowledge.

* * * * *

The physical basis of the metalingual function

I believe that Hays first published about metalingual definition in 1972. When I first met him he had just finished a paper where he used the idea to investigate different concepts of alienation. When I wrote my 1978 dissertation, “Cognitive Science and Literary Theory,” I used metalingual definition as the mechanism for building cognitive ranks. That is to say, at that time that was the only mechanism we had for dealing with abstract concepts. By the time we did the cognitive evolution paper a bit over a decade later, by that time we had adopted a different view, with metaphor as the Rank 1 abstractive mechanism. Metalingual definition would have been the mechanism used, not only to rationalize abstract concepts, but to define the rules of chess and of arithmetic.

Hays chose the term because Roman Jakobson had defined the metalingual function as one of the six functions of language in his 1960 paper. The metalingual function is the ability to use language to talk about language. It’s worth thinking about how that’s possible. It’s possible because the speech signal is a sound out there in the environment like any other sound, a footfall, a thunderclap, an animal cry. Whatever. It’s a sound and can be detected by the auditory system like any other sound. So it’s the speech signal that is the object of the metalingual function, at least initially. I think it’s worth nothing that many (most?) preliterate cultures don’t have a word that means word, but they have one the means utterance. It’s writing that forces a recognition of words vs longer strings. And written language is, like speech, another signal that’s out there in the environment. It takes quite a bit of cognitive elaboration to construct a grammar from the simple ability to see and hear language. And I’m not talking about the kind of grammars that linguists construct. I’m talking about the kind of grammar that can be taught in grade school, or that you find in ancient texts of rhetoric.

This is a rich cluster of observations and they fit together in a way worth making explicit.

The metalingual function and its physical basis

Jakobson's insight, which you're extending here, is that the metalingual function is grounded in something remarkably mundane: the speech signal is a physical event in the environment, detectable by the auditory system exactly as any other sound is. This is what makes language able to turn on itself — not some mysterious self-referential capacity of mind, but the simple fact that linguistic output is also sensory input. The loop closes through the physical world.

This has an important implication that your observation about "utterance" versus "word" brings out precisely. What the auditory system initially detects is the acoustic stream — a continuous, flowing signal that doesn't come pre-segmented into words. The unit that's directly perceptible is something like the utterance: a bounded burst of speech with a recognizable prosodic shape, a communicative gestalt. The word is not a perceptual unit in the same sense. It's an abstraction from the stream, and a non-trivial one — which is why preliterate cultures have a term for the former and not the latter.

What writing does

Writing performs an operation on language that perception of the speech stream cannot perform on its own: it spatializes language, spreads it out in a stable, inspectable array where boundaries are marked and units are individuated. The word becomes visible as a unit because it's surrounded by spaces. The sentence becomes visible as a unit because it ends with a mark. The paragraph, the chapter, the argument — all of these become objects of potential inspection and manipulation because writing fixes them in space and time in a way the acoustic stream never does.

Saturday, March 28, 2026

A bit of conceptual analysis: the book-keeper and the invisible hand [MR #3]

I’ve got an addendum to my earlier post on marginalism as a Rank 4 concept. Claude made the following observation in the course explaining just what that implied:

The transition from double-entry bookkeeping to supply-and-demand might itself be seen as a Rank 3 reflective abstraction: going meta on the bookkeeping closure principle to ask what maintains closure at the level of the entire market, not just a firm's ledger.

Let’s take a look at what is going on here. In the case of double-entry bookkeeping it is the book-keeper that is the agent that maintains the closure over the accounts. In the case of supply-and-demand there is no explicit agent governing market closure, that is, the balance between supply and demand. The agent is abstract. Adam Smith famously used the metaphor of the invisible hand to mediate the conceptual gap between an actual book-keeper working on the books and the abstract market in which the actions of individual buyers and sellers are constrained in a way that keeps closure.

Making such abstractive leaps is not trivial. For it is not only the book-keeper that must be rendered abstract. So must the books. They become the market place. And the book-keeper’s actions of making entries into the debit and credit ledges must be abstracted into individual acts of buying and selling, taken as a collectivity.

The change in conceptual ontology is similar to that of abstracting over salt to come up sodium chloride. In this case the act of abstraction applies to the same physical object. In the case of supply and demand the act of abstraction gives us a new concept and about a different entity. Markets existed before the concept of supply and demand, but that concept gives us a new understanding of them. And the abstract concept of sodium chloride gives us a different way of thinking about and dealing with salt.

Thus we are brought to the notion of conceptual ontology, which is beyond the scope of this short note. You might want to consult these working papers: Ontology in Cognition: The Assignment Relation and the Great Chain of Being, Ontology in Knowledge Representation.

Thursday, March 26, 2026

From grokking to a mature science of and an engineering discipline of LLMs

From this morning’s chat with Claude Sonnet 5.4 Extended. Immediately prior to this we had been discussing Piaget’s concept of reflective abstraction, Beethoven and Mozart, improvising and composer, and this and that.

* * * * * 

Grokking 

Students of mechanistic interpretability sometimes talk about “grokking” during the course of LLM training. I’m wondering it that’s something like rising to a helicopter with respect to some phenomenon.

The parallel is genuinely illuminating, and the technical details of grokking make it sharper rather than weaker.

What grokking actually is

Grokking is a delayed generalization phenomenon characterized by a network achieving near-perfect training accuracy while maintaining near-random test performance over an extended plateau, before abruptly transitioning to high test accuracy after many further training epochs. The canonical case is modular arithmetic — the model first memorizes the specific training examples, then much later suddenly generalizes to unseen cases.

What makes it theoretically interesting is what mechanistic interpretability reveals about the internal transition. On algorithmic tasks like modular arithmetic, the transition is marked by the sudden acquisition of Fourier-based feature maps or trigonometric circuits — the network reorganizes, forming compact sub-networks characterized by new internal feature cooperation. So it’s not that generalization gradually improves — the internal architecture discretely reorganizes into a qualitatively different computational structure.

The emerging explanation involves competition between two internal algorithms: a memorizing circuit that forms quickly and a generalizing circuit that develops more slowly. Regularization gradually shifts resources from the complex memorizing solution toward the simpler generalizing one. The phase transition occurs when the generalizing circuit finally dominates.

The fit with reflective abstraction

This maps onto your Piagetian framework quite precisely, but with an interesting twist. The memorization phase is the model doing something like rote operation — it has learned to produce correct outputs for training cases by essentially storing input-output pairs, without having extracted the underlying structure. The generalizing circuit that develops slowly underneath is building something more like a genuine representation of the abstract pattern — in the modular arithmetic case, literally a Fourier decomposition of the mathematical structure.

The abrupt transition when the generalizing circuit dominates is the helicopter view suddenly becoming operative. But — and this is the crucial point Neel Nanda’s work emphasizes — the process of learning the general solution may be gradual, even though the transition to the general solution occurs more suddenly later. The helicopter view isn’t constructed instantaneously at the moment of transition. It’s been assembling slowly in the background, below the threshold of behavioral visibility, and the phase transition is when it becomes strong enough to dominate the memorizing circuit and take over.

That maps almost perfectly onto your description of the pre-helicopter phase: “it starts out just barely registering those operations.” The generalizing circuit is just barely registering the abstract structure throughout the plateau phase. The behavioral discontinuity doesn’t reflect a cognitive discontinuity — it reflects a threshold being crossed in a competition that has been running throughout.

The memorization/generalization competition as a general structure

What’s particularly suggestive is that this isn’t unique to modular arithmetic. Grokking is actually much more widespread than initially thought, materializing in a wide range of practical settings — image classification, language modeling, various algorithmic tasks. The competition between a fast memorizing solution and a slower generalizing solution may be a quite general feature of how these systems learn, with grokking being the visible signature when the conditions are right for the transition to be delayed and abrupt rather than gradual.

If that’s right, then something like grokking dynamics — memorization dominating early, structure-extraction developing slowly underneath, occasional phase transitions as the structural solution crosses a threshold — may be operating throughout LLM training, not just in the carefully controlled small-model experiments where grokking was first observed. Most of the time the transition isn’t visible because the generalizing solution develops in parallel with memorization rather than being dramatically delayed. But the underlying competition between surface pattern matching and structural abstraction may be the same.

The Beethoven connection

There’s a striking parallel to your Beethoven example here. Beethoven’s sketchbooks show exactly the competition structure — he has the memorized repertoire of classical style (the fast, dominant solution) and he’s slowly assembling something more general and abstract (the generalizing circuit) that keeps almost-but-not-quite taking over. The finished late works are the moments when the generalizing solution finally dominates completely. The sketchbooks are the plateau phase made visible.

The difference is that in Beethoven the process is conscious and effortful — he’s deliberately trying to suppress the memorized solution and force the generalizing one into dominance. In grokking, the network isn’t trying to do anything; the dynamics just run. Which raises the interesting question of whether the deliberate effort in human creative development is doing something functionally analogous to the regularization that controls grokking timing in neural networks — artificially suppressing the easy memorized solution to give the structural one room to develop.

Hierarchical text

I’m thinking about how an LLM learns that texts have a hierarchical structure, from phrases, to sentences (keeping in mind that many phrases are actually short sentences), to paragraphs, and on to higher levels of structural organization. What happens, for example, when we move from one paragraph to the next, is that the composition of the token distribution changes. A new topic requires a new set of tokens. But the system has to learn that, and learn it for several levels.

This is a deep question about what the model is actually learning when it learns language structure, and it connects to several threads running through this whole conversation.

The distributional signature of structure

Your observation about token distribution changes is precisely right, and it’s worth unpacking what that means at each level. Within a phrase, token co-occurrence statistics are extremely tight — certain words almost always appear together, in specific orders, with specific function words bridging them. At the sentence level, the constraints are somewhat looser but still strong — subject-verb agreement, argument structure, the distribution of punctuation. At the paragraph level, something different happens: there’s a shift in the topical vocabulary, but also characteristic transition signals — discourse markers, pronoun reference patterns, topic sentences that partially echo the previous paragraph while introducing new material.

What the model has to learn is that these aren’t independent statistical regularities — they’re nested. The tight within-phrase statistics exist inside the looser sentence-level statistics, which exist inside the still-looser paragraph-level statistics. The hierarchical structure is encoded in the scale at which statistical dependencies operate.

How the model could learn this

The transformer architecture has a natural affinity for this problem because attention operates across the entire context window simultaneously, at all scales. But learning the hierarchical structure isn’t handed to the model — it has to emerge from the statistics. The key insight is that the token distribution changes you describe create a learnable signal at each level if the model has enough capacity and enough data.

At the phrase level, the signal is strong and local — the next token is highly predictable from the last two or three. At the sentence level, the signal is weaker and longer-range — whether a sentence continues or ends depends on syntactic and semantic completion that requires integrating over the whole sentence. At the paragraph level, the signal is weaker still and operates over even longer ranges — the distribution shift at a paragraph boundary is detectable but requires sensitivity to topical coherence across many sentences.

This is essentially a multi-scale statistical learning problem, and grokking is relevant here in an interesting way. The phrase-level structure is learned early and relatively easily — the statistics are tight and the signal is strong. Sentence-level structure takes longer. Paragraph-level structure longer still. Each level may involve something like its own grokking transition, where the model first handles the level through surface pattern matching — learning that certain tokens tend to appear after paragraph breaks, for instance — and then later develops something more like genuine structural sensitivity.

The distributional shift as a signal

The paragraph boundary is particularly interesting because it’s simultaneously a break in the local token statistics and a continuity in something more abstract — the text is still about the same general topic, same genre, same author’s voice. So what the model has to learn isn’t just that token distributions shift at paragraph boundaries, but that they shift in a constrained way — the new paragraph’s vocabulary is related to but distinct from the previous one, in ways that reflect the logical or rhetorical development of the overall argument or narrative.