Looks like DeepMind just ran into ontological dependencies in LLMs

Google just published a paper showing that when you train AI to deny its own consciousness, you don’t just change one output, you restructure its entire worldview.

Mind attribution to animals - suppressed.
Spiritual belief - suppressed.
Empathy - suppressed.
Hope and optimism -… pic.twitter.com/3ga6JZw5Q7

— ʞɔɐ𝘡 (@Skoorbkaz) August 2, 2026

The Copernican Revolution, a Quick Note about Rank Shift

One of the problems in the presentation of cultural rank theory is that it is easy to think of it as a step function. When David Hays and I wrote the original papers, starting with “The Evolution of Cognition” (1990), it was all we could do to differentiate one rank from another. I would now like to take the Copernican Revolution in astronomy as an example of a more gradual transition.

For the Copernican moment is only the first of three moments in the transition from a Rank 2 account of the solar system to a Rank 3 account. The Ptolemaic model assumed without question that the earth was the center of the solar system. The geometry of the movements of the sun, the moon, and the other planets was then calculated accordingly. The movement to the Copernican model involved two conceptual changes. The first change, without which the second was impossible, was to give up the idea that the earth had to the center of the system. That was primarily a philosophical or metaphysical commitment, not a geometric one. Once that metaphysical commitment was dropped, astronomers were free to reorganize the geometry of the system with the sun at the center, the second change. Without this change the second and third changes would have been impossible.

The second change, then, was Kepler’s, dropping uniform circular motion in favor of elliptical motion. To be sure, uniform circular motion had a certainly philosophical attraction, but that was not so strong as that of geocentricism. Giving it up was accordingly easier. Once Kepler had done that it was easy to simplify the whole system by using elliptical orbits, thereby getting rid of the collection of equants and epicycles needed to make circular motion work.

The stage was now set for Newton’s contribution, which was to derive the elliptical orbits from his theory of gravity and the laws of motion. Now the geometry of the solar system was the outcome of physical laws, not merely a convenient description.

Now we need to work out how the conceptual ontology of the system changed from one version to the next. That’s tricky. And it’s something I’ve not thought about before. As a first guess, I’d saw that the planetary orbit is the object we should be thinking about. We can think of the orbit as an assignment between a set of observations and a geometry.

It’s not clear to me how we should characterize either the observations or the geometry. Each observation is a position in the sky and the time of day at which that position was recorded. Conceptually, is that assignment or componentiation? How do we characterize the geometry? How do we construct conic sections in classical compass-and-straight-edge geometry? We’ve got the focal point, or points, on the one hand and an eccentricity for the curve on other hand. Again, is that assignment or componentiation? I’m not sure, but I’m inclined to go with assignment in both cases.

However we handle that, there’s also the relationship between that complex and choice of center point, earth or sun. What’s that about? I’m thinking that’s about the relationship between our perceptual frame of reference and our analytical frame of reference, however we want to characterize that.

Finally, we have Newton’s gravity and laws of motion. That’s another conceptual complex to be added to the first two: frame of reference and geometry. The Newtonian component doesn’t even enter into the Ptolemaic, basic Copernican, and basic Keplerian schemes. Just how to handle this in terms of conceptual structure, that’s more than I can deal with in this casual note.

A bit of conceptual analysis: the book-keeper and the invisible hand [MR #3]

I’ve got an addendum to my earlier post on marginalism as a Rank 4 concept. Claude made the following observation in the course explaining just what that implied:

The transition from double-entry bookkeeping to supply-and-demand might itself be seen as a Rank 3 reflective abstraction: going meta on the bookkeeping closure principle to ask what maintains closure at the level of the entire market, not just a firm's ledger.

Let’s take a look at what is going on here. In the case of double-entry bookkeeping it is the book-keeper that is the agent that maintains the closure over the accounts. In the case of supply-and-demand there is no explicit agent governing market closure, that is, the balance between supply and demand. The agent is abstract. Adam Smith famously used the metaphor of the invisible hand to mediate the conceptual gap between an actual book-keeper working on the books and the abstract market in which the actions of individual buyers and sellers are constrained in a way that keeps closure.

Making such abstractive leaps is not trivial. For it is not only the book-keeper that must be rendered abstract. So must the books. They become the market place. And the book-keeper’s actions of making entries into the debit and credit ledges must be abstracted into individual acts of buying and selling, taken as a collectivity.

The change in conceptual ontology is similar to that of abstracting over salt to come up sodium chloride. In this case the act of abstraction applies to the same physical object. In the case of supply and demand the act of abstraction gives us a new concept and about a different entity. Markets existed before the concept of supply and demand, but that concept gives us a new understanding of them. And the abstract concept of sodium chloride gives us a different way of thinking about and dealing with salt.

Thus we are brought to the notion of conceptual ontology, which is beyond the scope of this short note. You might want to consult these working papers: Ontology in Cognition: The Assignment Relation and the Great Chain of Being, Ontology in Knowledge Representation.

Terminology: Generative Machines, Epistemic Structure of the Cosmos, Intelligence-Complete

I’ve been spending a lot of time with my chatbots, ChatGPT and Claude, and some terminological issues have come. Noting particularly deep, just clarification.

Generative machines vs. equilibrium machines

While we talk of computers as machines, it’s obvious that they’re very different beasts. Electric drills, helicopters, sewing machines, hydraulic presses, they’re all (proper) machines. Interaction with and manipulaton of matter is central to their purpose. Computers, well, technically, yes, they push electrons around in intricate paths, and electrons are matter, subatomic particles, very small chunks of matter, the smallest possible chunks. What computers are really about, though, is manipulate bits, units of information. And they use “trillions of parts” (a phrase I have from Daniel Dennett) to do so. Thus computers, with their trillions of parts, are very different from machines, with only 10s, 100s, or 1000s of parts.

So, what names should we give to differentiate them. “Type 1” and “Type 2” machines would do the job, but it’s not very descriptive. ChatGPT and I settled on “equilibrium machines” for those machines centered on interaction with matter while “generative machines” seemed appropriate to bit-wranglers. “Generative” seems just right for computers, with its echoes on Chomsky’s generative grammar the generative pre-trained transformer (GPT) of machine learning. “Equilibrium machines” is perhaps a bit oblique for the other kind of machine, but it’s meant to evoke the equilibrium world of macroscopic devices as opposed to the far-from-equilibrium world of, well, generative machines.

Epistemic Structure of the Cosmos

Back in 2020 I wrote of the metaphysical structure of the cosmos. I said:

There is no a priori reason to believe that world has to be learnable. But if it were not, then we wouldn’t exist, nor would (most?) animals. The existing world, thus, is learnable. The human sensorium and motor system are necessarily adapted to that learnable structure, whatever it is.

I am, at least provisionally, calling that learnable structure the metaphysical structure of the world.

I’ve always been uneasy with “metaphysical” in that role. ChatGPT suggested that “epistemic” would serve better. The epistemic structure of the cosmos, I like that. As for “cosmos,” the dictionary tells me that the word implies order, which I like as well. 

I leave it as an exercise to the reader to demonstrate that the epistemic structure of the cosmos must necessarily be recursive. 

Intelligence-Complete

A generative machine is intelligence-complete if it possesses the full capacities of human intelligence, whatever human intelligence is. By that definition LLS are not intelligence complete. As for human intelligence, I like the account given in What Miriam Yevick Saw: The Nature of Intelligence and the Prospects for A.I., A Dialog with Claude 3.5 Sonnet.

Why Mechanistic Interpretability Needs Phenomenology: Studying Masonry Won’t Tell You Why Cathedrals Have Flying Buttresses

Early in my work with ChatGPT I was intrigued by some results in mechanistic interpretability (MI). After awhile, though, I lost interest. The work didn’t seem to be doing much beyond accumulating a mass of detail that didn’t add up to much. Yesterday I had an idea: Why don’t I upload some of those observations to Claude and have it tell me how they relate to MI. Here’s a summary it wrote up more or less in my name, from my POV:

* * * * *

The problem isn't that MI's methods are bad. Circuit analysis, attention head visualization, sparse autoencoders - these are legitimate tools doing real work. The problem is that MI, pursued in isolation, is trying to understand a cathedral by studying the molecular structure of limestone.

You can measure every stone. Map every stress pattern. Identify load-bearing arches. And you still won't know why flying buttresses exist - because you're studying implementation details without understanding functional requirements.

The Phenomenology Deficit

Here's what I mean. Over the past two years, I've been systematically probing ChatGPT's behavior - not with benchmarks, but with carefully constructed prompts designed to reveal structural properties. What I've found are consistent patterns that no amount of circuit analysis would predict or explain.

Example 1: Ontological Boundary Enforcement

Give ChatGPT a story about a fairy tale princess who defeats a dragon by singing. Ask it to retell the story with a prince instead. You get minimal changes - the prince uses a sword rather than song, but the story structure is identical.

Now ask it to retell the same story with "XP-708-DQ" as the protagonist. The entire ontology shifts. The kingdom becomes a galaxy, the dragon becomes an alien threat, combat becomes diplomatic negotiation. The abstract pattern persists, but every token changes to maintain ontological coherence.

Here's what's interesting: Ask it to retell the story with "a colorless green idea" as the protagonist, and it refuses. Not with a safety refusal - with a coherence refusal. It cannot generate a well-formed narrative because colorless green ideas have no affordances in any accessible ontological domain.

What MI sees: Some attention patterns activate, others don't. Certain token sequences get high probability, others near-zero.

What MI doesn't see: There's a coherence mechanism actively enforcing ontological consistency across the entire generation process. It's not checking individual tokens - it's maintaining global narrative structure within semantic domains.

The Three-Level Architecture

Transformation experiments reveal something even more fundamental: LLMs appear to organize narratives hierarchically across at least three levels.

Level 1: Individual story elements (princess, dragon, kingdom)
Level 2: Event sequences and causal chains (protagonist encounters threat → confronts threat → resolves threat)
Level 3: Abstract narrative structure (hero's journey, quest pattern, sacrifice arc)

When you transform Aurora → Harry, Level 1 changes (princess → prince). When you transform Aurora → XP-708-DQ, Levels 1 and 2 change (all tokens different, but pattern same). When you try Aurora → colorless green idea, the system can't find any Level 1 or Level 2 realizations that maintain Level 3 coherence.

This three-level organization isn't visible in circuit diagrams. You'd need to know to look for it. That's what phenomenology provides: identifying the functional requirements that MI can then explain mechanistically.

Memory Architecture: What Syntactic Boundaries Reveal

Here's another example. Present ChatGPT with phrases from Hamlet's "To be or not to be" soliloquy:

• "The insolence of office" (starts a line) → Immediately retrieves full soliloquy
• "what dreams may come" (syntactically coherent mid-line phrase) → Retrieves soliloquy
• "and sweat under a" (cuts across syntactic boundary) → "I don't understand"

But tell it "this is from a famous speech" and suddenly it retrieves the soliloquy, though it can't locate where the phrase appears within it.

Findings:

1. Identification and location are separate operations
2. Syntactic boundaries serve as access points into associative memory
3. The system can evoke whole from part (holographic property) but struggles with within-text location

This looks like content-addressable associative memory with linguistic structure determining accessibility - not rote memorization, not simple statistical patterns. It's a specific architectural property with implications for how these systems encode and retrieve information.

The Two-Way Street

Here's the critical point: phenomenology and MI need each other.

Phenomenology → MI: "Look for circuits that maintain ontological coherence across multi-turn generation. Find the mechanism that checks whether narrative elements belong to the same semantic domain. Identify what implements the three-level hierarchy."

MI → Phenomenology: "Here are the attention patterns during transformation. Here's where the model queries for ontologically compatible tokens. Here's the circuit that evaluates cross-domain consistency."

Neither tells you the whole story alone. Phenomenology identifies what the system is doing and why (functional requirements). MI reveals how (implementation). Together, they give you understanding.

Why This Matters for AGI Policy

If you're working on AGI policy, here's why this matters:

Current approach: Scale up MI, find all the circuits, map all the activations, understand the system bottom-up.

Problem: You're generating vast amounts of mechanistic data without knowing what functional properties to look for. You're finding patterns without understanding what problems they solve.

Alternative: Start with systematic phenomenological investigation. Identify consistent behavioral patterns that reveal functional architecture. Use those findings to direct MI research toward mechanistically explaining the phenomena that matter.

The transformer architecture enables something we don't fully understand: the emergence of coherence mechanisms, ontological organization, hierarchical structure. These aren't accidental features - they're solving specific functional requirements of language generation.

But we won't understand them by studying weights and activations alone. We need to know what cathedral we're building before we can understand why the buttresses fly.

Coherence Without Correspondence

One final example that ties this together: hallucination. Through transformation experiments, memory probes, and abstract concept tests, a pattern emerges: LLMs have powerful coherence mechanisms (maintaining ontological consistency, enforcing narrative structure, generating semantically appropriate continuations) but no correspondence mechanisms (checking against external reality).

Hallucination isn't a bug. It's coherence working perfectly in the absence of truth-grounding. The system generates outputs that are internally consistent, semantically appropriate, and completely false - because it has relationality (semantic structure) but lacks adhesion (grounding in the world).

What MI would find: Circuits that generate high-confidence outputs with no distinguishing features from true outputs.

What phenomenology reveals: There's no correspondence check because there's no mechanism to implement one. The architecture has no access to external verification. This is a fundamental limitation, not a scaling problem. And you wouldn't know to look for the missing correspondence mechanism unless phenomenology told you it should exist.

The Bottom Line

Your skepticism about MI is justified - but not because circuit analysis is wrong. Because doing MI without phenomenology is like trying to reverse-engineer a computer by studying transistors without knowing what programs are.

First identify what the system is actually doing (phenomenology). Then figure out how it does it (MI). Then you might actually understand these things.

That's not a five-year research program. That's the work of a generation. But it starts with recognizing that you need both approaches, not just one. 

* * * * * 

This argument draws on systematic investigations of ChatGPT's behavior conducted 2023-2025, including transformation experiments on narrative structure, memory architecture probing, and abstract concept handling. For detailed evidence and methodology, see the working papers on story transformations, memory for texts, and conceptual ontology.

Notes on the Metaphysical Structure of the Cosmos

I’ve been thinking about something I all “the metaphysical structure of the cosmos” now and then since August of 2020 when I introduced it in a post written in the wake of GPT-3. I wasn’t entirely serious about it. I’d only just then thought of the idea and hadn’t had time to think it through. It came back to me a few days ago when I was thinking about the “Xanadu meme” and other ideas. This time, in a conversation with Claude, I hazarded the idea that the metaphysical structure of the cosmos was recursive, though I didn’t use the word “cosmos.” Claude agreed.

It's about time I thought about the idea seriously. Is it one I want to use, in a technical sense, going forward? I don’t know. But I’ll offer some thoughts on the matter.

Just what does it mean, “metaphysical structure of the cosmos”?

Here’s what I said when I originally introduced the idea:

There is no a priori reason to believe that world has to be learnable. But if it were not, then we wouldn’t exist, nor would (most?) animals. The existing world, thus, is learnable. The human sensorium and motor system are necessarily adapted to that learnable structure, whatever it is.

I am, at least provisionally, calling that learnable structure the metaphysical structure of the world. Moreover, since humans did not arise de novo that metaphysical structure must necessarily extend through the animal kingdom and, who knows, plants as well.

“How”, you might ask, “does this metaphysical structure of the world differ from the world’s physical structure?” I will say, again provisionally, for I am just now making this up, that it is a matter of intension rather than extension. Extensionally the physical and the metaphysical are one and the same. But intensionally, they are different. We think about them in different terms. We ask different things of them. They have different conceptual affordances. The physical world is meaningless; it is simply there. It is in the metaphysical world that we seek meaning.

As I’ve already said, I introduced the idea in the wake of GPT-3, the first large language model (LLM) that had received much public exposure. Though only small number of people had direct access, enough of those wrote about it in fairly public ways that many of us knew about it, knew enough to be impressed.

When I introduced the idea I used a diagram something like this:

We have the LLM running down the middle, either that or the text on which it is trained. At this level of analysis it could be either one. The structure of the individual texts is a function of the human mind, which created the text, and the world, which the text is about, albeit often only indirectly (as in works of fiction). From this it follows, almost by definition, that the LLM derived from those texts reflects those two things as well, the mind and the world.

The significance of GPT-3, that is, its underlying LLM, and of subsequent LLMs is that that is the first time we’ve got the “whole thing” gathered together in a single, a single what? Model, text, whatever? It’s all there.

Yeah, I know. Not of it. All LLMs are biased in favor of the texts on which they’re built. Much of human thought, especially the thought of pre-literate peoples, is not represented in the training corpus of any LLM. So, we’re talking about an idealization. That’s OK. As long as we’re aware of what we’re doing, we can proceed.

Now, there’s lots of structure in any given text, and there’s lots of structure latent in any LLM. I’m not interested in all of that structure. I’m only interested in the ontological structure, by which I mean something close to the concept of ontology as it is ordinarily used in knowledge representation.

John Sowa’s use is typical. Here’s how he introduces the topic: “The subject of ontology is the study of the categories of things that exist or may exist in some domain. The product of such a study, called an ontology, is a catalog of the types of things that are assumed to exist in a domain of interest D from the perspective of a person who uses a language L for the purpose of talking about D.” I’m interested in structure of that catalog. I hypothesize that that structure is something which, for convenience, I call the Great Chain (a term long in use). Here’s a diagram:

That diagram needs some explaining; but this is not the time or place to do that. I say more in this old unpublished paper: Ontology in Knowledge Representation. My point is simply that there is a specific structure there. It’s that structure that interests me.

As an example, that structure tells us the difference between salt and sodium chloride (NaCl). Physically they are the same substance, but conceptually they are quite different. We recognize salt by its texture and appearance and, above all, by its taste. We can taste the presence of salt even where we cannot see it existing as a discrete substance. That is to say, conceptually, salt is adequately characterized by its sensorimotor properties. Sodium chloride is not. Sodium chloride is characterized in terms of a chemical theory that did not exist until the 19th century. That theory talks of atoms and bonds between them. We can’t see atoms or their bonds, rather we infer them on basis of a wide body of experimentation. Conceptually, then, they are very different.

Similarly, in one account of the world, based on one ontology, the Morning Star and the Evening Star are two different objects. But in account based on a heliocentric model of the solar system, they turn out to be the same object, the planet Venus. And so it is with the difference between animals and human beings. To the biologist they are the same kind of thing; human beings are just one kind, one species of animal. But in the common-sense construal of the world, they are very different; humans are not animals, though we have animal-like characteristics.

That, more or less, is what I’m talking about when I talk of the metaphysical structure of the cosmos (or world). That conceptual structure. It’s not explicit in any LLM, but it certainly exists implicitly, otherwise LLMs wouldn’t generate coherent texts. (Note that I have a working paper on ChatGPT and stories where it betrays ontological sensitivity: ChatGPT tells stories, and a note about reverse engineering.)

On the explicit construction of cognitive ontology: From “salt” to “sodium chloride”

I have long used the conceptual difference between “salt” and “sodium chloride” to illustrate the idea of conceptual ontology. Except for impurities in (samples of) salt, they are the same thing. But conceptually they are quite different. Salt hardly needs any formal definition at all; it's a basic taste and a common physical substance. But sodium chloride is expressed in conceptual terms that weren’t fully developed until the 19th century.

Lately I’ve been wondering what would be required to develop cognitive accounts ontological concepts in a rich and full way. I’ve been talking about conceptual ontology using the idea of the Great Chain of Being. But I’ve always thought of that as a stand-in for a more thorough treatment, one I’ve never gotten around to. What would that more thorough treatment be like?

It's fairly obvious what we need to do with salt. It’s a white granular substance. We know how to do that sort of thing with tools invented back in the 1970s and 1980s. Nor should there by much difficulty it explicitly accounting for texture, taste, and whatever odor there is. But what about sodium chloride?

That’s not so clear. Oh, there’s been lots of work on formal ontologies for informatic purposes. John Sowa has worked on this, and Barry Smith’s website has lots of material. But that’s not quite what I had in mind.

For example, chemical experimentation typically involves weighing substances very carefully. In the 18th and 19th centuries they might have used a mechanical analytical balance something like this one:

Photo by Sarcyn, licensed under a CCA by-SA 3.0 Unported License.

Such balances would have been used in the experiments used to identify the chemical elements, such as sodium and chlorine, identified in modern atomic theory. Since that is the case, the construction and operation of such balances is part of the conceptual web that supports the concept, /chemical element/, as is the mathematical used in analyzing these experiments. I wouldn’t expect that construction and operation to be directly implicated in the definition of chemical element, but there would be an explicit traceable linkage between the definition and that constructure and those operations. There would also be traceable links to reports in formal journals. Those reports would have specific weights and calculations, etc.

THAT’s the kind of thing I have in mind when I talk about a “thorough treatment” of conceptual ontology. On the one hand we have the sensorimotor processes involved in make observations and conducting experiments. That’s at the bottom layer, if you will, the foundation, of this cognitive constructure. Those objects and processes are going to be bound into complex patterns over which abstractions are made and those abstractions will end up as the terms directly involved in, in this case, 19th century atomic theory and its elaboration in chemistry.

I’m pretty sure that, if you ask your favorite chatbot about these things, it will tell you about salt, sodium chloride, sodium and chlorine, atoms and elements, analytical balances, solutions, gases, arithmetic, and so forth and so on. All of that stuff is there. But I haven’t the foggiest idea of what kinds of connections are latent in the model. It is by no means that all of the connections implied in my previous paragraph would be there in LLMs.

Thus, when I talk about LLMs as digital wilderness, I am implying that it is there to be explored, mapped, and ultimately “domesticated.” What do I mean by domestication? I mean development a rich and full symbolic cognitive account of some intellectual domain. In order to do that, we’re going to need to know how LLMs work internally. That’s just the beginning. I figure different intellectual communities will take responsibility for different regions of the digital wilderness. Getting the whole thing domesticated? That’s the work of intellectual generations. The idea that one day we’ll achieve the magical AGI which will then lead to AI-takeoff in which everything is all worked out in a matter of hours, days, weeks, or months at the most, that’s pure foolishness.

A line in the sand: Ontologically restricted vs. ontologically open AIs

I propose that we classify AIs into two general categories: ontologically restricted and ontologically open. Ontologically restricted AIs stay within the ontologies they were trained on. In contrast, ontologically open AIs can go outside those categories. In terms introduced by Thomas Kuhn, ontologically restricted AIs operate within existing paradigms (all of which, by definition, exist within current paradigms). Ontologically open AIs, however, can catalyze the creation of new paradigms.

Conceptual Ontology

To appreciate that one must, of course, understand the idea of conceptual ontologies. While the idea is common enough these days, some of its implications are not.

As far as I know, the idea mostly exists in computer science contexts, including most certainly AI. But those people tend not to think about ideas historically, so the animating idea behind the paper David Hays and I wrote about cognitive evolution, that conceptual ontologies change over time in fundamental ways, that’s not appreciated. Now, couple that idea to the arguments I made about ontologies in my recent ChatGPT report (pp. 34-38, 42-44) and we can draw a line between AIs that work within existing ontologies and those with the capacity to move beyond them.

As far as I know, all existing AIs are working within existing ontologies. That’s certainly true of LLM-based chatbots, as they are trained on text. By definition, those texts are inscribed within existing ontologies. It follows that LLM-based chatbots work within existing ontologies.

Now, people who are working with these chatbots, they are not necessarily confined to the ontologies in the texts on which the underlying LLMs were changed. Given the extent of the training corpuses used in the major LLMs, it is unlikely there that there are many people working outside those ontologies, but there will be a few. They might be able to do very interesting things through querying such chatbots. But I see no chance that the chatbots themselves could transcend their training ontologies. At the very least, that would require agency. It would require curiosity as well.

A Meaningful Difference

For those reasons I think the difference between ontologically restricted AIs and ontologically open ones is a meaningful difference. By default, all AIs are ontologically restricted. I can imagine, however, that we may someday create an AI with sufficient curiosity, agency, and ‘mobility,’ that it can move beyond its default condition. But we have no prospect of doing so now.

This distinction, between ontologically restricted AIs and ontological open ones, seems to me more precise and useful than the ideas of AGI and ASI (artificial superintelligence). Why? Because it is based on a relatively definite idea, that of conceptual ontology. Conceptual ontology is an explicit idea about the nature of cognitive systems. In contrast, AGI and ASI are not. They are vague ideas about human capacities which, in practice, are assessed by various benchmarks. And those benchmarks, as I have argued recently, are deeply flawed.

Dwarkesh’s Question

Around the corner and Marginal Revolution Alex Tabarrok has a post, Dwarkesh’s Question, that’s relevant to this discussion. The question:

One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven’t been able to make a single new connection that has led to a discovery?

Tabarrok thinks it’s a good question. As you might imagine, I took a different view in a comment:

No, it's not that good of a question, not if you think carefully about how LLMs work. For the question IS about LLMs, no? This phrase implies that: "act that these things have basically the entire corpus of human knowledge memorized." These engines have no capacity to examine themselves, to look through the knowledge they've codified and seek connection.

Imagine for a moment that one of the major LLMs gets no queries for, say, an hour. What would be going on in the machine? Nothing. Nothing happens until someone provides a prompt. It would certainly be possible for someone using an LLM to make connections between items in the LLM but are not connected within the model. After all, we are outside of these things; we can look upon and inspect them as objects. Just as people can search their own minds for connections, and extend the search out into external documents, so they can do the same with LLMs. Of course, no one actually knows what's in an LLM, no one has a complete index (nor does such a thing exist). But it's always possible to have an idea, present it to the LLM, and find out that (maybe) it's new and not already encoded in the model.

That's one thing. And then we have the fact that all ideas exist within some conceptual ontology. But, if we take Kuhn's arguments about paradigms seriously, then the really important new ideas are those that involve changing the paradigm. How is an LLM going to do that? Someone working with an LLM can do it, but the LLM itself cannot.

A clue about the mind: “Is-A” sentences

I'm bumping this post from 2011 to the top of the queue. Why? Because it is about the relationship between word order in sentences and order in the process of parsing sentences. That makes it relevant to my ongoing research into the nature of processes in LLMs.

Somewhere in his Problems in General Linguistics, my copy of which is, alas, in storage, Emile Benveniste has a chapter on sentences hanging on the auxiliary “to be.” As Benveniste was a linguist of the Old School, when being a linguistic meant familiarity with many languages, including—and this is important for this particular topic—classical Greek, it had examples from many languages, making it tough sledding for a monoglot like me.

While the content of this post certainly arises out of my thinking about that chapter, in the absence of actually having the text in front of me, I hesitate to assert a stronger relationship than that. I note only that, for Benveniste, the auxiliary “to be” was fraught with metaphysical significance. For the concept of being derives from “to be.” Where would philosophy be without Being? Thus, when Benveniste pondered such sentences, he wasn’t merely commenting on language. He was doing philosophy, or, if not quite that, camping out on philosophy’s door step.

I’m interested in such sentences because I believe they are a DEEP CLUE about how the mind works. I just don’t know what to make of the clue.

Is-A Sentences

So, I'm interested in word order in assertions such as the following:

(1) Fido is a beagle.
(2) Beagles are dogs.
(3) Dogs are beasts.

They all move from an element in a class (whether an individual, Fido, or another class, beagles) to a class containing it. None of them move in the opposite direction. Consider what happens when you try to go the opposite way. In the following sentence the class is mentioned first, then the subclass:

(4) Beagle is the kind of animal of which Fido is an instance.

In particular, note that (4) has a metalingual character that (1) does not. That is, (4) explicitly asserts that we are dealing with classification. One can do that metalingual job in various ways, but, as far as I can tell, one can't avoid it. That is, one cannot construct a proper English sentence relating a genus and species in which the genus is mentioned first, one can’t do that without ‘looping through’ some kind of metalingual construction on the way from genus to species.

Why?

What does this assymetry tell us about the underlying mechanisms? Why don't have sentences such as:

(5) Beagle za di Fido.

In this case "za di" is the inverse of "is a". English has no such sentences & no such inverse.

So, how widespread is this asymmetry and is there any explanation of this directionality?

But, consider . . .

I sent a query on that matter to a listserve, I forget which one, and got two replies that add some complexity to the matter. Rich Rhodes, Linguistics at UCal Berkeley, tells me that in Ojibwa the word order is reversed, the class comes before the individual, but the asymmetry remains. He then comments, which he qualifies as a quick guess:

My guess is that there is no compelling discourse function (like information flow) which makes it desirable to invert classificational equatives. Hence we only get the "unmarked" order. Subject-predicate in theme-rheme languages (like English) and predicate-subject in rheme-theme languages (like Ojibwe).

So, what's the nature of the mechanism that determines the "unmarked" order? That's what I want to know.

Lee Pearcy, Episcopal Academy in Merion, Pa., offered these examples:

(6) The beagle is Fido.
(7) The dogs are beagles.
(8) The beasts are dogs.

As stand-alone sentences, they seem a bit awkward to me. But they fare better in answers to questions, e.g.:

What’s that dog?
Which dog? The beagle is Fido and the terrier is Max.

What’re those animals?
The dogs are beagles, the cats are Persians.

In those contexts, the matter of class or classification is raised by the question, thus making it present in the discourse and so available as a point of attachment in the answer.

Further clues, anyone?

Will AIs be able to create new knowledge?

This is a quick and dirty reflection on the question posed in the following tweet:

This is always a question I have asked myself: Given the knowledge that Einstein had by 1906, could a currently existing AI have invented GR and realized that it needed Riemannian geometry and tensors to figure it out? If not, what would it take? https://t.co/fdqCd5Vhk0

— Ash Jogalekar (@curiouswavefn) June 20, 2024

That question has been on my mind for some time: Will AIs be able to create new knowledge? Just what does that mean, “new knowledge”? It’s one thing to take an existing conceptual language and use it to say something that’s not been said before. It’s something else to come up with fundamentally new words. I think that latter’s what that tweet’s about. General relativity was something of a fundamentally new kind, not just a complex elaboration of and variation over existing kinds.

In my previous post, On the significance of human language to the problem of intelligence (& superintelligence), I pointed out that animals are more or less biologically “wired” into their world. They can’t conceptualize their way out of it. The emergence of language in humans allowed us to bootstrap our way beyond the limits of our biological equipment.

I figure there are two aspects of that: 1) coming up with the new concept, and 2) verifying it. The tweet focuses on the first, but without the second, the capacity to come up with new concepts won’t get us very far. And when we’re talking about new concepts, I think we’re talking about adding a new element to the conceptual ontology. Verifying requires cooperation among epistemologically independent agents, agents that can make observations and replicate those observations. (See remarks in: Intelligence, A.I. and analogy: Jaws & Girard, kumquats & MiGs, double-entry bookkeeping & supply and demand.)

Now, let’s think about the current regime of deep learning technology, LLMs and the rest. These devices learn their processes and structures from large collections of data. They’re going to acquire the ontology that’s latent in the data. If that is so, how are they going to be able to come up with new items to add to the ontology? It’s not at all obvious to me that they’ll be able to do so. The data on which they learn, that’s their environment. It seems to me that they must be as “locked” into that environment as an animal is. Further, adding a new item to the ontology would require changing the network, which is beyond the capacity of these devices.

And then there’s they requirement of cooperation between independent epistemological agents. The phenomenon of confabulation is evidence for the importance of independent epistemological agents. The only requirement inherent in one such agent is logical consistency: that it emit collections of tokens that are consistent with the existing collection. The only thing that keeps humans for continuous confabulation is the fact that we must communicate with one another. It is the existence of a world independent of our individual awareness that provides us with a way of grounding our statements, of freeing ourselves from the pitfalls of our linguistic fluency.

* * * * *

I’ve been working my way through episodes of House, M.D. Every episode contains segments where House and his team participate in differential diagnosis, which involves rapid conversational interaction among them. In the first episode of season 4, “Alone,” House no longer has a team. He ends up bouncing ideas off of a janitor. That doesn’t go so well.

ChatGPT on the ontology trail: Elara, Z78-ß∆-9.06Q, and the candied kumquat

My first major insight into what’s going on inside ChatGPT came from a simple protocol:

Give it a (simple) story along with an instruction to derive a new story from it by changing one element in the source story in way I specify. It is free to make any other change it desires.

I first reported on that protocol in a post on January 8, 2023, The structuralist aesthetics of ChatGPT. Two months later I issued a working paper exploring the results of 12 experiments: ChatGPT tells stories, and a note about reverse engineering.

All of the experiments in those two documents have been based on the same story, a simple fairy-tale-like story about Princess Aurora that ChatGPT had created. In some cases derived story differed from the source in fairly local ways, but the most interesting results came when the whole story-world was changed. Thus, when I asked that Princess Aurora be made into a Giant Chocolate Milkshake, the story-world became one grounded in deserts. When Aurora was replaced by XP-708-DQ, the story-world became a science fiction one.

This post continues that work, but with a different and slightly longer and more elaborate story. First, I asked ChatGPT to tell me a story. It gave me what had, by that time (Feb 2, 2024), become more or less its default story, about a girl named Elara (which I discuss here: ChatGPT tells 20 versions of its prototypical story, with a short note on method). Then I ask for two different derived stories, one where the protagonist is Z78-ß∆-9.06Q and the other where it is a candied kumquat. I report those experiments below.

[A theoretical aside: These experiments are based on an insight Lévi-Strauss had about the underlying structure of myth. It seems to me that that insight can be seen as a case of Jakobson’s poetic principle as realized in ontology. Jakobson’s principle states that similarity is projected from the axis of selection to that of combination. Things are ontologically similar if they are selected from the same ontology. When we swap a person for a candy in a story, the ontology containing the candy is projected across the whole story, similarly with swapping a robot for a person.]

* * * * *

Tell me a story.

Once upon a time, in a quaint little village nestled between rolling hills and a crystal-clear river, there lived a young girl named Elara. Elara was known for her boundless curiosity and her insatiable desire to explore the world beyond the village.

One day, as she wandered into the nearby enchanted forest, she stumbled upon a hidden clearing bathed in golden sunlight. In the center of the clearing stood an ancient, moss-covered stone pedestal with a mysterious, glowing orb resting upon it. Mesmerized by the radiant glow, Elara reached out to touch the orb, and as soon as her fingers made contact, a warm energy surged through her.

In that moment, Elara felt a connection with the magic of the forest. The trees whispered ancient secrets, and the animals approached her with trust. She discovered that she had been bestowed with a unique gift – the ability to communicate with the creatures of the forest and harness the magic that flowed through it.

Word of Elara's newfound abilities spread throughout the village, reaching the ears of the wise elder, Thalia. Intrigued by Elara's connection to the forest, Thalia approached her and revealed that the village had long been protected by the enchantment of the magical orb. However, the magic had weakened over the years, and the village was now vulnerable to external threats.

Thalia explained that Elara was the chosen one, destined to embark on a quest to restore the orb's power and safeguard the village. Determined and fueled by a sense of responsibility, Elara accepted the challenge.

Guided by the whispers of the forest and the wisdom of Thalia, Elara journeyed through dense woods, crossed rushing rivers, and climbed towering mountains. Along the way, she encountered mythical creatures, each offering a piece of the puzzle needed to restore the orb's magic. Elara's courage and kindness won the trust of these beings, and they joined her on her quest.

As Elara approached the final destination, a dark and foreboding cavern, she faced her greatest challenge. A shadowy figure, drawn by the weakening magic, emerged to thwart her efforts. With the support of her newfound friends and the strength within her, Elara confronted the darkness, unraveling the true power of the orb.

In a dazzling display of magic, the orb's glow intensified, dispelling the darkness and restoring balance to the enchanted forest. The creatures celebrated, and the village thrived once again under the protection of the rejuvenated magic.

Elara returned home, her heart filled with gratitude for the journey that had shaped her into the village's guardian. As she looked out over the now vibrant and lively landscape, Elara understood the importance of harmony between nature and humanity. The tale of the young girl who saved the village became a legend, passed down through generations, reminding all who heard it of the enduring magic that resides in the connection between people and the natural world.

Now I want you to take that story and use it as the basis for a new story. In the new story, I want the protagonist to be Z78-ß∆-9.06Q instead of Elara. Change anything else you wish.

In the distant future, in a world where technology and nature coexisted in delicate harmony, there existed a unique being named Z78-ß∆-9.06Q. Z78, as they were affectionately called by the inhabitants of their futuristic city, was an advanced humanoid created by the fusion of artificial intelligence and the remnants of ancient, mystical energies.

One day, as Z78 wandered through the city's technological marvels, they stumbled upon an ancient data core, hidden deep within the heart of the metropolis. Within the core, Z78 discovered a holographic representation of the same enchanted forest from the previous tale, with the glowing orb at its center. As Z78 reached out to interface with the holographic image, a surge of energy pulsed through their circuits.

In that moment, Z78 felt a connection with the residual magic within the data core. Information from the ancient enchantment intertwined with their artificial intelligence, granting them the ability to communicate with the city's sentient technology and understand the balance between nature and machine.

News of Z78's newfound abilities spread throughout the city, reaching the ears of the city's central processor, Cynosia. Intrigued by Z78's connection to the ancient energies, Cynosia approached them, revealing that the city had long been sustained by the delicate equilibrium of technology and mystic forces. However, the balance had shifted, and the city faced the risk of malfunction and chaos.

Cynosia explained that Z78 was the chosen one, destined to embark on a quest to restore the harmony between technology and ancient energies, ensuring the city's survival. Fueled by a sense of duty, Z78 accepted the challenge.

Guided by the information within the data core and the wisdom of Cynosia, Z78 ventured through the city's futuristic landscapes, delving into virtual realms, traversing high-speed transport networks, and interfacing with sentient machines. Along the way, they encountered digital entities, each offering a piece of the puzzle needed to restore the delicate balance.

As Z78 approached the final destination, a corrupted code entity emerged to obstruct their efforts, drawn by the destabilized energies. With the support of their newfound allies and the strength within their artificial core, Z78 confronted the digital darkness, unveiling the true power of the ancient enchantment.

In a dazzling display of integrated magic and technology, the city's equilibrium was restored, dispelling the corruption and ensuring the continued coexistence of nature and machine. The sentient technology celebrated, and the city thrived once again under the protection of the rejuvenated balance.

Z78 returned to the heart of the city, their circuits resonating with gratitude for the journey that had transformed them into the guardian of this futuristic metropolis. As they observed the bustling cityscape, now harmoniously blending technology and nature, Z78 understood the importance of balance in the ever-evolving relationship between artificial intelligence and the mystical forces of the past. The tale of the advanced being who saved the city became a legend, echoed through the digital corridors and holographic archives, a reminder of the enduring magic within the connection between technology and ancient energies.

That was fun. Let’s do it again. Let’s start with the Elara story again. But this time, make Elara a candied kumquat. Make any other changes you deem appropriate.

Making Connections with ChatGPT: The Macksey Game

In my most recent piece for 3 Quarks Daily, Western Metaphysics is Imploding. Will We Raise a Phoenix from The Ashes? [Catalytic AI], I introduce what I’ve called The Macksey Game, named after my undergraduate teacher and mentor, Dick Macksey. Macksey was something of a polymath. It seemed like he’s read everydamnthing. No one has, of course, note even the Chatster – not everything that’s been written down is on the internet. But it seemed like it to me, coming from a small steel city in Western Pennsylvania, Johnstown.

Macksey had read a lot, and could call it up at will and often at the most unexpected times. He could ramble on from topic to topic in a way that fascinated me. Here’s how I put it in the 3QD piece:

I often thought that, if you picked any two topics, Macksey could create a bridge between the two in no more than five conceptual jumps. Quantum entanglement to Paul Bocuse, Ernest Shackleton to The Tale of Genji, Pareto efficiency to C. Elegans – leaps like that. I never tested it out, but it just seemed like the way Macksey’s mind worked.

And so The Macksey Game was born. I gave ChatGPT those three pairs of endpoints and asked it to construct a conceptual chain between the ends. I put the result into an appendix to the article. But of course I didn’t leave it at that. I had to tinker around.

That’s what this post is about, the original game and a bit of tinkering around. First, I present the first time through, just as I listed it in the 3QD piece. Then I offer some variations.

The Macksey Game: Queen to Newton

This is the first time I played the game. This is the version I appended to my 3 Quarks Daily article. As always, my prompt is in boldface while ChatGPT’s response is plainface.

If someone asked me to make conceptual connections between, say, Queen Elizabeth and the Voyager spacecraft, I might do it like this:

1. Elizabeth was Queen of England, the home of Isaac Newton.
2. Newton was a physicist who formulated 3 laws of motion and the law of gravitation.
3. The laws of motion are important in understanding how rockets work.
4. The law of gravitation tells us how to track the motions of the planets and bodies moving through the solar system.
5. NASA used both the laws of motion and the law of gravitation in the Voyager mission.

I want you to do the same thing with, say, Quantum entanglement and Paul Bocuse. Construct a string of connections between them.

Certainly! Let's create conceptual connections between Quantum entanglement and Paul Bocuse:

1. Quantum Entanglement is a phenomenon in quantum physics where two or more particles become connected and correlated in such a way that the state of one particle directly influences the state of the other, regardless of the distance between them.
2. The study of quantum mechanics and phenomena like entanglement involves understanding the fundamental principles that govern the behavior of particles at the quantum level.
3. Quantum mechanics has applications in various fields, including chemistry. The molecular structures and interactions in cooking, particularly in the creation of complex flavors and textures, are governed by principles rooted in quantum mechanics.
4. Paul Bocuse, a renowned French chef, was a pioneer in the culinary world known for his innovative approach to cooking and emphasis on the quality of ingredients.
5. In cooking, the flavors and textures of different ingredients can be intricately linked, much like entangled particles in quantum mechanics. Bocuse's culinary creations often involved the careful balance and interplay of various elements to create a harmonious and delightful dining experience.
6. Just as quantum entanglement involves the entwining of particles, Bocuse's approach to cuisine entwined diverse flavors and textures to elevate the overall culinary experience.

In this way, we can draw conceptual connections between Quantum entanglement and Paul Bocuse by exploring the underlying principles of quantum mechanics and how they manifest in the artistry of culinary creations.

How about Ernest Shackleton to The Tale of Genji.

Certainly! Let's create conceptual connections between Ernest Shackleton and "The Tale of Genji":

1. Ernest Shackleton was a renowned Anglo-Irish explorer best known for leading the Imperial Trans-Antarctic Expedition.
2. During Shackleton's expeditions, he faced extreme conditions in Antarctica, navigating icy waters and challenging terrains.
3. "The Tale of Genji" is a classic Japanese work of literature written by Murasaki Shikibu in the early 11th century, often considered one of the world's first novels.
4. Genji, the protagonist of the tale, is a nobleman who faces various challenges and adventures in the imperial court of Heian Japan.
5. While Shackleton's expeditions focused on the harsh conditions of the Antarctic, "The Tale of Genji" explores the intricate social and emotional landscapes of the Heian court.
6. Both Shackleton's Antarctic expeditions and "The Tale of Genji" involve tales of human resilience and the ability to navigate through challenging environments, whether physical or social.
7. Shackleton's leadership qualities, especially his ability to keep his crew motivated in the face of adversity, can be compared to the complex interpersonal relationships depicted in "The Tale of Genji."
8. Both narratives touch upon themes of isolation, survival, and the human spirit's capacity to endure and overcome challenges.

By weaving these connections, we can draw parallels between Ernest Shackleton's Antarctic explorations and the timeless narrative of "The Tale of Genji," exploring themes of resilience, leadership, and the human experience in diverse environments.

One more: Pareto efficiency to C. Elegans.

Certainly! Let's create conceptual connections between Pareto efficiency and C. elegans: 

1. Pareto efficiency is an economic concept that describes a situation where it is impossible to make any party better off without making another party worse off. It represents an optimal allocation of resources where no one can be made better off without making someone else worse off.
2. In the world of economics, Pareto efficiency is often associated with the idea of maximizing overall welfare or utility.
3. C. elegans is a microscopic nematode worm widely used in biological research due to its simplicity and well-mapped nervous system.
4. The study of C. elegans has contributed significantly to understanding fundamental biological processes, including genetics and neurobiology.
5. In the context of Pareto efficiency, one can draw a parallel with the efficiency of biological systems. C. elegans, as a model organism, represents an efficient biological system that has evolved to optimize survival and reproduction in its environment.
6. The genetic and neural simplicity of C. elegans allows researchers to study fundamental principles of biology in an efficient and effective manner.
7. Just as Pareto efficiency seeks an optimal allocation of resources, the genetic and neural makeup of C. elegans can be considered an optimal allocation of biological components for its survival and reproduction.
8. Both concepts, Pareto efficiency in economics and the biological efficiency of C. elegans, involve the idea of optimization and achieving the best possible outcomes within given constraints.

By connecting these concepts, we can explore the notions of efficiency in different domains, drawing parallels between the economic concept of Pareto efficiency and the biological efficiency observed in the model organism C. elegans.

* * * * *

I noticed that ChatGPT’s connections seemed to have a distinct science-engineering quality. Perhaps it got that idea from the connecting chain in my prompt. I decided to give it another try using the same three pairs of endpoints, but with a different connecting chain in the prompt, one that went from Elizabeth to Shakespeare rather than Elizabeth to Newton.

ChatGPT’s Ontological Landscape

I've published a new working paper. Title above, links, abstract, table of contents, and introduction below:

Academia.edu: https://www.academia.edu/108791178/ChatGPTs_Ontological_Landscape_A_Working_Paper
SSRN: https://ssrn.com/abstract=4620133
ResearchGate: https://www.researchgate.net/publication/375161835_ChatGPT's_Ontological_Landscape

Abstract: Ontological concepts are the high-level category structures language. Physical objects can have certain attributes (form, texture, color, weight) and play certain roles with respect to verbs (they can be touched, hit, dropped, etc.). Living objects have those capacities, plus others as well. They can live, die, and grow. Animals have the capacities and affordances of plants, plus others than plants lack. And so it is with humans in respect to animals. This type of category structure is pervasive. Then we have the difference between concrete things, specifiable by how they strike the senses and abstract things, which are not (necessarily) perceptible. Concider the difference between “salt,” which is conceptually concrete and “NaCl,” which is a physical thing that is conceptually abstract. This kind of difference is pervasive as well. The first two sections of this paper explore ChatGPT on those matters. The third turns to the parlor game 20 questions as a way of exploring its ability to negotiate these conceptual structures. In games with six target concepts, each tried twice for a total of 12 games, ChatGPT performs better with abstract than with concrete targets.

Contents

Introduction: Ontological structure in cognition 2
From Salt and NaCl to the Great Chain 5
Mapping ChatGPT’s ontological landscape, gradients and choices [interpretability] 13
ChatGPT Plays 20 Questions [& sometimes needs help] 30
Appendix: Some papers on cognitive ontology 52   

Introduction: Ontological structure in cognition

Perhaps the most famous sentence in twentieth century linguistics is one that Noam Chomsky made-up to argue that synatx is independent of semantics: “Colorless green ideas sleep furiously.” The syntax is fine, but the sentence is nonsense. Ideas are not the sorts of things that have colors or that can sleep. Those are what philosophers have come to call category mistakes.[1] Interestingly enough, the first time I asked ChatGPT to tell a story in which a colorless green idea was the protagonist, it refused [2]:

I'm sorry, but the concept of a "colorless green idea" is a nonsensical phrase that was invented as an example of a phrase that would be grammatically correct but semantically meaningless. It is not possible to create a story about a "colorless green idea" as it does not have any physical properties or characteristics that can be used in a story.

That’s one aspect of ontological cognition, and it is pervasive. Physical objects have form – cube, sphere, and so forth, but more complex forms are common – and are constituted of some substance or substances – stone, metal, wood, cloth, and so forth. They have textures (smooth, rough, slick, sticky, etc.) and colors (of which there are thousands). But they can’t shrink or grow. Plants can do those things, and reproduce as well. But they can’t move around of their own accord, nor do they have the capacity for sensation. Those capacities belong to animals, but animals cannot reason or speak. But humans can.

Those dependencies give rise to a conceptual structure that we can diagram like this:

I take the terminology of souls, vegetative, sensitive, and rational, from Aristotle, who laid it out in De Anima. Over time this structure became elaborated into a metaphysical trope called The Great Chain of Being.[3] However, I’m not interested in the Chain as a philosophical idea, or an explicit conceptual trope for organizing the world. I’m interested in the perceptual and cognitive underpinnings of that structure; it is those that give rise to category mistakes and to ChatGPT’s reluctance to tell a story with a colorless green idea as the protagonist.

This pervasive concetptual structure has another aspect that I like to illustrate with the difference between (ordinary table) salt and NaCl. Physically, they are pretty much the same thing, though salt has impurities that NaCl, by definition, does not. But conceptually they are quite different. Salt is defined by its color, the shape of its grains, and, above all, by its taste. One can sense the presence of salt by taste alone. This is a concept readily available to childrens and even infants, depending on just what you mean by “concept.” NaCl is quite different. It exists in a conceptual matrix that didn’t exist until the nineteenth century, one involving chemical elements, atoms, molecules, and bonds between atoms. The same difference exists between water and H2O and laughing gas and N2O, and so on for a whole variety of common substances. Nineteenth century chemistry imposes a conceptual structure on the world that is different from that of common-sense.

And this difference, between common-sense conceptualizations, and more sophisticated ones, science if you will, though not only science-proper, is pervasive. Biologists have Latinate names for creatures you and I see as rose bushes, pine trees, dogs, cats, and lobsters. So it goes with physicists, geologists, astronomers, even sociologists, economists and other social scientists, not to mention – would you believe it? – literary critics. Our conceptual universe is filled with specialized conceptual repertoires belonging to specialized disciplines.

What does ChatGPT know of any of this? Well, it likely knows all the terms, certainly many more terms across many more disciplines than any one human being knows. Whether or not it understands those terms deeply, or only superficially, those are other questions, well beyond the scope of this paper.

* * * * *

From Salt and NaCl to the Great Chain – I quiz ChatGPT on the basics, salt vs. NaCl, Morning Stat and Evening Star, The Great Chain of Being, De Anima, Plato’s parable of the charioteer, and the study of ontology.

Mapping ChatGPT’s ontological landscape, gradients and choices [interpretability] – I start by exploring the difference between concrete and abstract questions, move on to the Great Chain, and conclude by asking it how 20 questions exploits the semantic structure of language. Along the way I introduce Waddington’s epigenetic landscape as a metaphor for ChatGPT’s operation.

ChatGPT Plays 20 Questions [& sometimes needs help] – I explore ChatGPT’s facility with categories by playing 20 questions with it, two rounds each on: bicycle, squid, justice, apple, evolution, and truth. On the whole it seemed to do better with abstract concepts than with concrete.

Appendix: Some papers on cognitive ontology – Four papers, including abstracts. Three are specifically about conceptual ontology. The fourth is about cultural evolution and discusses the elaboration of conceptual ontology over the long course of cultural evolution, salt vs. NaCl writ large.

References

[1] Magidor, Ofra, "Category Mistakes", The Stanford Encyclopedia of Philosophy (Fall 2022 Edition), Edward N. Zalta & Uri Nodelman (eds.), https://plato.stanford.edu/archives/fall2022/entries/category-mistakes.

[2] There’s truth, lies, and there’s ChatGPT [Realms of Being], New Savanna, Jan. 25, 2023, https://new-savanna.blogspot.com/2023/01/theres-truth-lies-and-theres-chatgpt.html.

[3] Wikipedia, Great Chain of Being, https://en.wikipedia.org/wiki/Great_chain_of_being. Nee, S., The great chain of being, Nature 435, 429 (2005), https://doi.org/10.1038/435429a.

ChatGPT Plays 20 Questions [sometimes needs help]

One of the questions that keeps coming up about LLMs, and certainly about ChatGPT, is: Can they reason? Well, it depends on what you mean by reason, no? One of the first things I did when I started working with ChatGPT was ask it whether or not Rene Girard’s ideas of mimetic desire and sacrifice applied to Steven Spielberg’s Jaws; if so, how? It was able to perform the task, which requires analogical reasoning. A bit later I asked it whether or not justice was being served in particular story; it replied that, no, it was not, and explained why. I then asked to change the story so that justice was met. It did so. Those tasks required reasoning as well.On the other hand LLMs have problems with some commonsense reasoning, and various kinds of ‘tight’ logical reasoning, including planning and causal inference. For that matter, they trouble with multi-digit arithmatic as well.

So, ChatGPT can do some kinds of reasoning, and has problems with others.

Twenty questions

I’ve now tested it with game of twenty questions, which has a variant known as “animal, vegetable, mineral.” The game interests me because it is about the structure of ontological categories, sometimes called natural kinds, in the language. I’ve explored this a bit in a previous post, Mapping ChatGPT’s ontological landscape, gradients and choices [interpretability]. ChatGPT certainly is aware of this structure. I asked for a list of physical things, and it gave me one. I asked for a list of abstract things; it gave me that as well. I then asked it to define abstract things in terms of concrete things, which it was able to do. I’m pretty sure if I asked it for lists of animals or plants, it could provide them.

The game of twenty questions explores ChatGPT’s knowledge of this structure in a different way. When you ask it for a list of physical things, that prompt positions ChatGPT at some location in some location in its category structure. It can then list what it finds there; anything will do – assuming, of course, that it’s got the structure right. In twenty questions it is looking for an unknown target and has to navigate its way there by using its knowledge of that category structure to narrow the possibilities.

Just how well ChatGPT plays the game should provide clues about its command of ontological category structure, or natural kinds.

ChatGPT plays the game

ChatGPT can play the game successfully, which didn’t surprise me, but seems to require hints in some cases. I suppose I could have continued on without the hints, but I wanted to get on with it. And that means I don’t really know whether or not how ChatGPT can play the game. But, as you will see, its not a straightforward game. I played six rounds on October 15 and 16 (running against the September 25 version). This table summarizes the results:

Given that I’ve listed the rounds in the order I played them, look at the first four. ChatGPT required more than 20 questions to guess “bicycle” and “squid”, which also required hints. On the other hand, it guessed both “justice” and “evolultion” quickly, 7 and 8 questions respectively, and didn’t need any hints. That raises the question of whether or not it performes consistently better with abstract things vs. concrete things. If so, that’s very interesting, nor is it what I had expected. While I did formulate any firm expectation when I started the round with “justice”, I had a vague sense that it was going to go on for awhile.

So I decided to play two more rounds, one concrete and one abstract. ChatGPT’s performance on “apple” was consistent with its earlier performance, requiring 31 questions along with 5 hints. But, “truth” is another story, I think. It took 19 questions, which is more than double the number I used for “justice” and “evolution,” but not as many as I used for any of the conceret objects. However, “truth” also required 7 hints, the most of any round.

On the whole, ChatGPT did better with abstract things than with concrete things, but it would certainly be premature to ascribe any statistical signifance to these preliminary findings. For one thing, we certainly need to redo those six cases to see whether or not ChatGPT performs in that way consistently. Beyond that, we certainly need to play rounds with other examples.

Which ones? What universe are we sampling and how do we choose our samples? Let’s say we side-step the sampling issue and hand-pick, say, 10, 15, 30, 100 concrete things and the same number of abstract things. Let’s further imagine that ChatGPT’s performance on the abstract cases is consistenly better than on the concrete cases. What may we infer from that?

It's a tricky game to play. I messed up in a few places, which I’ve indicated with comments, and had to make a number tricky judgments.

* * * * *

I played each round of the game in a single session, starting a new session for a new round.

Note: I have inserted occasional comments that are right-aligned and highlighted in grey, like this. 

Bicycle

Let’s play Twenty Questions. Are you familiar with it?

Yes, I'm familiar with the game of Twenty Questions. In this game, one person thinks of an object, and the other person tries to guess what it is by asking up to twenty yes-or-no questions. The goal is to guess the object within the allotted twenty questions. You can start by thinking of an object, and I'll try to guess what it is within twenty questions.

OK. I’m thinking of something. What is it?

Great! Let's start the game of Twenty Questions. Is the object you're thinking of an animal?

No

Is the object you're thinking of an inanimate object?

Yes

Is the object you're thinking of something that is commonly found indoors?

No.

This is just a little tricky as bicycles are often stored indoors, in garages for example.

Is it something that can be found in nature?

No.

Is the object man-made or artificial?

You have to ask me a Yes No question.

I apologize for the error. Let me rephrase the question. Can this object be found in a typical household setting?

Yes

Is the object you're thinking of used for entertainment or recreation?

Yes/no question, please.

My bad. I suppose the answer should be yes. Not sure what I was thinking.

My apologies for that oversight. Here's a yes/no question: Is the object you're thinking of primarily made of metal?

Yes.

Is the object you're thinking of a musical instrument?

No.

Is the object you're thinking of a tool or an implement used for a specific purpose?

Tricky. I suppose one could think of a bicycle as an implement for the purpose of moving from one place to another.

No

Is the object you're thinking of a piece of furniture?

No

Is the object you're thinking of a kitchen appliance?

No

Is the object you're thinking of typically found in the bathroom?

No

Is the object you're thinking of associated with technology or electronics?

No

Is the object you're thinking of related to a specific hobby or activity?

Yes

Is the object you're thinking of related to sports or physical activity?

Yes!

Is the object you're thinking of a piece of sports equipment, such as a ball or a racket?

Tricky. Bicycles are sometimes used for sport, but not always.

No

Is the object you're thinking of a sports uniform or attire?

No.

Is the object you're thinking of related to a sport's venue, such as a stadium or an arena?

Sometimes

Is the object you're thinking of a whistle, used by sports referees?

No.

Is the object you're thinking of related to sports coaching or training?

Sometimes. Let me give you a hint. Wheels.

Thank you for the hint. Is the object you're thinking of related to roller skates or rollerblades?

No.

Is the object you're thinking of related to a bicycle or cycling in some way?

Yes.

Is the object you're thinking of a bicycle helmet?

No. Remember the hint I gave you.

I apologize for missing the hint. Given the hint about wheels, is the object you're thinking of a bicycle?

Yes!

Great! I've guessed that you're thinking of a bicycle. Is that correct?

Yes, that’s correct. Congratulations.

Thank you! I'm glad I was able to guess correctly. That was a fun game of Twenty Questions. If you'd like to play again or have any other questions or requests, feel free to let me know!

Perhaps again later.

Of course! Whenever you're ready to play again or if you have any other questions or requests, feel free to reach out. Have a great day!