A line in the sand: Ontologically restricted vs. ontologically open AIs

I propose that we classify AIs into two general categories: ontologically restricted and ontologically open. Ontologically restricted AIs stay within the ontologies they were trained on. In contrast, ontologically open AIs can go outside those categories. In terms introduced by Thomas Kuhn, ontologically restricted AIs operate within existing paradigms (all of which, by definition, exist within current paradigms). Ontologically open AIs, however, can catalyze the creation of new paradigms.

Conceptual Ontology

To appreciate that one must, of course, understand the idea of conceptual ontologies. While the idea is common enough these days, some of its implications are not.

As far as I know, the idea mostly exists in computer science contexts, including most certainly AI. But those people tend not to think about ideas historically, so the animating idea behind the paper David Hays and I wrote about cognitive evolution, that conceptual ontologies change over time in fundamental ways, that’s not appreciated. Now, couple that idea to the arguments I made about ontologies in my recent ChatGPT report (pp. 34-38, 42-44) and we can draw a line between AIs that work within existing ontologies and those with the capacity to move beyond them.

As far as I know, all existing AIs are working within existing AIs. That’s certainly true of LLM-based chatbots, as they are trained on text. By definition, those texts are inscribed within existing ontologies. It follows that LLM-based chatbots work within existing ontologies.

Now, people who are working with these chatbots, they are not necessarily confined to the ontologies in the texts on which the underlying LLMs were changed. Given the extent of the training corpuses used in the major LLMs, it is unlikely there that there are many people working outside those ontologies, but there will be a few. They might be able to do very interesting things through querying such chatbots. But I see no chance that the chatbots themselves could transcend their training ontologies. At the very least, that would require agency. It would require curiosity as well.

A Meaningful Difference

For those reasons I think the difference between ontologically restricted AIs and ontologically open ones is a meaningful difference. By default, all AIs are ontologically restricted. I can imagine, however, that we may someday create an AI with sufficient curiosity, agency, and ‘mobility,’ that it can move beyond its default condition. But we have no prospect of doing so now.

This distinction, between ontologically restricted AIs and ontological open ones, seems to me more precise and useful than the ideas of AGI and ASI (artificial superintelligence). Why? Because it is based on a relatively definite idea, that of conceptual ontology. Conceptual ontology is an explicit idea about the nature of cognitive systems. In contrast, AGI and ASI are not. They are vague ideas about human capacities which, in practice, are assessed by various benchmarks. And those benchmarks, as I have argued recently, are deeply flawed.

Dwarkesh’s Question

Around the corner and Marginal Revolution Alex Tabarrok has a post, Dwarkesh’s Question, that’s relevant to this discussion. The question:

One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven’t been able to make a single new connection that has led to a discovery?

Tabarrok thinks it’s a good question. As you might imagine, I took a different view in a comment:

No, it's not that good of a question, not if you think carefully about how LLMs work. For the question IS about LLMs, no? This phrase implies that: "act that these things have basically the entire corpus of human knowledge memorized." These engines have no capacity to examine themselves, to look through the knowledge they've codified and seek connection.

Imagine for a moment that one of the major LLMs gets no queries for, say, an hour. What would be going on in the machine? Nothing. Nothing happens until someone provides a prompt. It would certainly be possible for someone using an LLM to make connections between items in the LLM but are not connected within the model. After all, we are outside of these things; we can look upon and inspect them as objects. Just as people can search their own minds for connections, and extend the search out into external documents, so they can do the same with LLMs. Of course, no one actually knows what's in an LLM, no one has a complete index (nor does such a thing exist). But it's always possible to have an idea, present it to the LLM, and find out that (maybe) it's new and not already encoded in the model.

That's one thing. And then we have the fact that all ideas exist within some conceptual ontology. But, if we take Kuhn's arguments about paradigms seriously, then the really important new ideas are those that involve changing the paradigm. How is an LLM going to do that? Someone working with an LLM can do it, but the LLM itself cannot.