LLMs, hallucinations, and language as cultural technology @3QD

I’ve got a new article at 3 Quarks Daily:

Of Grammar and Truth: Language Models and Norms, Truth and the World

I start with an obscure topic in linguistics, evidentials, and then move on to so-called hallucinations and into the Gopnik, Farrell, Underwood account of AI as cultural technology. I conclude the article by explaining how I got Claude to create the text and discussed the issues that raises for attribution.

The penultimate section is entitled: What Language Turns Out to Be: Mechanistic. But I never really explain that. I’m going to do that here.

Or rather I’m going to let Claude explain:

The success of modern chess programs and large language models shows that language and reasoning are mechanistic, but not in the familiar steam-engine sense of mechanism. These systems are better understood as machines with trillions of interacting parts, whose behavior emerges from distributed internal dynamics rather than from transparent, human-scale causal chains. Such mechanisms operate autonomously: once set in motion, they carry out sustained symbolic activity without continuous human or animal control. This autonomy is not accidental; it is the defining consequence of scale. Just as early steam locomotives violated pre-industrial ontologies by exhibiting self-propelled motion without life, contemporary computational systems violate inherited ontologies by exhibiting structured linguistic and cognitive behavior without minds. What we are confronting is not the end of mechanism, but the emergence of a new kind of mechanism—one that forces us to revise the categories by which we distinguish agency, control, and understanding.

We decided that steam-engine mechanisms are best called equilibrium machines while machines of a trillion parts are generative machines:

By equilibrium machines I mean mechanisms designed to settle into stable, repetitive behavior, minimizing deviation and surprise. These are the machines of the Industrial Revolution, and they underpin the worldview of Homo economicus. By generative machines I mean mechanisms maintained far from equilibrium, whose internal dynamics produce structured novelty and exploration. Language is the paradigmatic generative machine, and Homo ludens is the form of life that emerges when such machines become central rather than marginal.

The world of Homo economicus is organized around equilibrium mechanisms: machines designed to settle, repeat, and minimize deviation. These are the mechanisms of the Industrial Revolution, whose success shaped not only our technologies but our intuitions about causality, control, and value. Homo ludens inhabits a different world. Its characteristic institutions and practices arise from generative mechanisms—systems maintained far from equilibrium, whose internal dynamics support exploration, play, and the continual production of novelty. Human freedom does not stand opposed to such mechanisms; it depends on them.

This allows me to observe (in Claude’s words:

Human freedom and creativity are not opposed to mechanism. They are grounded in a special class of mechanisms—decoupled, autonomous mechanisms whose internal standards of coherence allow sustained activity independent of immediate worldly constraint. Language is paradigmatic of this class.

That is an idea I’ll be developing in my book, Play: How to Stay Human in the AI Revolution.

The Chinese Room – I got it! I see where it’s going, or coming from. (I think)

Bump to the head of the queue. I'm thinking about this stuff. Though I should say more, I don't find that intention is very useful in distinguishing between 'real' from 'artificial' intelligence. Where do we find intention in the brain or, for that matter, the whole organism? How would we create it in an artificial being? We haven't got a clue on either score. For some other thoughts, in a somewhat different but still related context, see this post on Stanley Fish and meaning literary criticism. 

* * * * *

 

John Searle’s Chinese room argument is one of the best-known thought experiments in the contemporary philosophy of mind and has spawned endless commentary. I read it when it appeared in Behavioral and Brain Science in 1980 and was unimpressed [1]. Here’s a brief restatement from David Cole’s entry in The Stanford Encyclopedia of Philosophy [2]:

Searle imagines himself alone in a room following a computer program for responding to Chinese characters slipped under the door. Searle understands nothing of Chinese, and yet, by following the program for manipulating symbols and numerals just as a computer does, he produces appropriate strings of Chinese characters that fool those outside into thinking there is a Chinese speaker in the room. The narrow conclusion of the argument is that programming a digital computer may make it appear to understand language but does not produce real understanding. Hence the “Turing Test” is inadequate. Searle argues that the thought experiment underscores the fact that computers merely use syntactic rules to manipulate symbol strings, but have no understanding of meaning or semantics. The broader conclusion of the argument is that the theory that human minds are computer-like computational or information processing systems is refuted. Instead minds must result from biological processes; computers can at best simulate these biological processes.

The whole thing seemed to me irrelevant because it didn’t address any of the ideas and models actually used in development computer simulation of mental processes. There was nothing in there that I could use to improve my work. The argument just seemed useless to me. For that matter, most of the philosophical discussion on this has seemed useless for the same reason; it’s conducted at some remote distance from the ideas and techniques driving the research.

I remarked on this to David Hays and he replied, that yes, the philosophers will say it can’t be done but the programs will get better and better. Not mind you, that Hays thought we were on the verge of cracking the human mind or, for that matter, that I think so now. It’s just that, well, this kind of argumentation isn’t helpful.

I still believe that – not helpful – but I’m beginning to think that, nonetheless, Searle had a point. A lot depends on just what “real understanding” is. The crucial point of his thought experiment is that there was in fact a mind involved, the guy (Searle’s proxy) “manipulating symbols and numerals just as a computer does” has a perfectly good mind (we may assume). But that mind is not directly engaged in the translation. It’s insulated from understanding Chinese by the layer of (computer-like) instructions he uses to produce the result that fools those who don’t know what’s happening inside the box.

The core issue is intentionality, an enormously important if somewhat tricky term of philosophical art. David Cole glosses it:

Intentionality is the property of being about something, having content. In the 19th Century, psychologist Franz Brentano re-introduced this term from Medieval philosophy and held that intentionality was the “mark of the mental”. Beliefs and desires are intentional states: they have propositional content (one believes that p, one desires that p, where sentences substitute for “p” ).

He quotes Searle as asserting:

I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. To put this point slightly more technically, the notion “same implemented program” defines an equivalence class that is specified independently of any specific physical realization. But such a specification necessarily leaves out the biologically specific powers of the brain to cause cognitive processes. A system, me, for example, would not acquire an understanding of Chinese just by going through the steps of a computer program that simulated the behavior of a Chinese speaker (p.17).

We’ll just skip over Searle’s talk of semantics as I have come to make a (perhaps idiosyncratic) distinction between semantics and meaning. Let’s just put semantics aside, but agree with Searle about meaning.

 

The critical remark is about “the biologically specific powers of the brain.” Brains are living beings; computers are not. Living beings are self-organized “from the inside” – something I explored in an old post, What’s it mean, minds are built from the inside? Computers are not; they programmed “from the outside” by programmers. But living beings are not self-organized in isolation. They are self-organized in an environment and it is toward that environment that they have intentional states.

Brains are made of living cells, each active from the time it emerged from mitosis. And so we have growth and learning in development, prenatal and postnatal. At every point those neurons are living beings. And those neurons, like all living cells, are descended from those first living cells billions of years ago.

Searle’s argument ultimately rests on human biology and a belief that life cannot be “programmed from the outside”. Let us say that I am deeply sympathetic to that view. But I cannot say for sure that a mind cannot be programmed from the outside. Moreover I note that Searle’s argument originated before the flowering of machine learning techniques in the last decade or so.

There is a sense in which those computers do in fact “learn from the inside”. Programmers do not write rules for recognizing cats, playing Go, or translating from one language to another. The machine is programmed with a general capacity for learning and it learns the “rules” of a given domain itself [3]. As a result, we don’t really know what the computer is doing. We can’t just “open it up” and examine the rules it has developed.

Will such technology evolve to the point where these systems have genuine intentionality? We don’t know. They’re along way from in now, but who knows?

* * * * *

[1] Searle, J., 1980, “Minds, Brains and Programs”, Behavioral and Brain Sciences, 3: 417–57. Preprint available online, http://cogprints.org/7150/1/10.1.1.83.5248.pdf

Searle has a brief 2009 statement of the argument online at Scholarpedia: http://www.scholarpedia.org/article/Chinese_room_argument

[2] Cole, David, "The Chinese Room Argument", The Stanford Encyclopedia of Philosophy (Winter 2015 Edition), Edward N. Zalta (ed.), https://plato.stanford.edu/archives/win2015/entries/chinese-room/

[3] For a good journalistic account of some of the recent work, see Gideon Lewis-Kraus, The Great A.I. Awakening, New York Times Magazine, December 14, 2016:

http://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html?&hp&action=click&pgtype=Homepage&clickSource=story-heading&module=second-column-region&region=top-news&WT.nav=top-news&_r=0

Mind or Machine?

I have, in various posts, expressed skepticism about the long-standing conversation about whether or not the mind is computational. I pay attention to the debate because I have a professional obligation to know something about it. But those discussions don’t tell me anything I can use in my intellectual work, whether it’s analyzing literary texts and movies, or thinking more generally about mind and culture. Those discussions rarely engage the with ideas and observations that one uses is more “practical” – as if analyzing “Kubla Khan” or King Kong were practical! – intellectual work.

It seems to me that those discussions are fundamentally metaphysical – well, duh! it’s philosophy of mind, no? – but likely ideological and even theological as well. We’ve got thinkers butting heads over fundamental assumptions, but not actually trying to figure anything out. In a way, machine is a proxy for man and all his works while mind is a proxy for God and mystery, or perhaps crass commerce vs. aristocratic noblesse oblige. Maybe both.

It’s not a problem that’s going to be solved. Perhaps, though, it will simply disappear. Spontaneous combustion.

I hope so.

"Philosophy doesn't do nuance" – Will computers ever be able, you know, to think?

Luciano Floridi in Aeon:

Philosophy doesn’t do nuances well. It might fancy itself a model of precision and finely honed distinctions, but what it really loves are polarisations and dichotomies. Internalism or externalism, foundationalism or coherentism, trolley left or right, zombies or not zombies, observer-relative or observer-independent, possible or impossible worlds, grounded or ungrounded … Philosophy might preach the inclusive vel (‘girls or boys may play’) but too often indulges in the exclusive aut aut (‘either you like it or you don’t’).

The current debate about AI is a case in point. Here, the dichotomy is between those who believe in true AI and those who do not. Yes, the real thing, not Siri in your iPhone, Roomba in your living room, or Nest in your kitchen (I am the happy owner of all three). Think instead of the false Maria in Metropolis (1927); Hal 9000 in 2001: A Space Odyssey (1968), on which Good was one of the consultants; C3PO in Star Wars (1977); Rachael in Blade Runner (1982); Data in Star Trek: The Next Generation (1987); Agent Smith in The Matrix (1999) or the disembodied Samantha in Her (2013). You’ve got the picture. Believers in true AI and in Good’s ‘intelligence explosion’ belong to the Church of Singularitarians. For lack of a better term, I shall refer to the disbelievers as members of the Church of AItheists. Let’s have a look at both faiths and see why both are mistaken. And meanwhile, remember: good philosophy is almost always in the boring middle.

Singulatarians:

...believe in three dogmas. First, that the creation of some form of artificial ultraintelligence is likely in the foreseeable future. This turning point is known as a technological singularity, hence the name. Both the nature of such a superintelligence and the exact timeframe of its arrival are left unspecified, although Singularitarians tend to prefer futures that are conveniently close-enough-to-worry-about but far-enough-not-to-be-around-to-be-proved-wrong.

Second, humanity runs a major risk of being dominated by such ultraintelligence. Third, a primary responsibility of the current generation is to ensure that the Singularity either does not happen or, if it does, that it is benign and will benefit humanity.

Here he hits pay dirt: "Singularitarianism is irrefutable because, in the end, it is unconstrained by reason and evidence." But, retorts the Singulatarian, it IS possible, no? Yeah, sure:

But this ‘could’ is mere logical possibility – as far as we know, there is no contradiction in assuming the development of artificial ultraintelligence. Yet this is a trick, blurring the immense difference between ‘I could be sick tomorrow’ when I am already feeling unwell, and ‘I could be a butterfly that dreams it’s a human being.’

And then there's that good old chestnut, exponential growth, as though it's somehow impossible for an exponential 'hockey stick' curve to level out into a sigmoid curve.

But, Floridi argues, their arch enemies, whom he dubs the AItheiests, are just as bad:

AI is just computers, computers are just Turing Machines, Turing Machines are merely syntactic engines, and syntactic engines cannot think, cannot know, cannot be conscious. End of story.

Quod erat demonstrandum.

NOT.

Here he references, among others, John Searle, whom I recently discussed, Searle almost blows it on computational intelligence, almost, but not quite [biology]. I give Searle points on his suggestion that specifically human biochemistry is essential for duplicating the causal powers of the human brains, but, as far as I can see, that has little or nothing to do with his famous Chinese Room argument, which is the center of his critique.

Searle almost blows it on computational intelligence, almost, but not quite [biology]

John Searle has long been a critic of the pretensions of artificial intelligence to, well, you know, intelligence. He’s perhaps best know for his Chinese room argument. To parody:

Some guy’s in a room. Knows English well but doesn’t speak a lick of Mandarin. But he’s got a flotilla of yellow pads on which there’s a blizzard of instructions in English and pseudo-code for writing Mandarin. A Chinese speaker slips some statement in Mandarin through a slot in the wall. The guy goes to work with his yellow pads and blue pencils and, in due course, scribbles some Mandarin on a slip of paper and sends it back out through the slot. And thus begins a convincing ‘conversation’ in Mandarin. But, really, our guy doesn’t know a jot of Mandarin.

In Searle’s terms what our guy is doing is all syntax, no semantics. And that’s what computers do, all syntax, but not a hint of semantics.

I wasn’t convinced back then – a lot of people weren’t – and I retain that old skepticism, though there are days when I think the argument might have some merit, under a particular interpretation.

Anyhow, I just came across a 2014 article in The New York Review of Books in which Searle takes on two recent books:

Luciano Floridi, The 4th Revolution: How the Infosphere Is Reshaping Human Reality, Oxford University Press, 2014.

Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, 
Oxford University Press, 2014.

Searle’s argument depends on understanding that both objectivity and subjectivity can be taken in ontological and epistemological senses. I’m not going to recount that part of the argument. If you’re curious what Searle’s up in this business to you can read his full argument and/or you can read the appendix to this post, where I’ve quoted a number of passages from a 1995 book in which Searle lays out matters with some care.

Searle sets up his argument by pointing out that, at the time Turing wrote his famous article, “Computing Machinery and Intelligence”, the term “computer” originally applied to people (generally women, BTW) who performed computations. The term was then transferred to the appropriate machines via the intermediary, “computing machinery”. Searle observes:

But it is important to see that in the literal, real, observer-independent sense in which humans compute, mechanical computers do not compute. They go through a set of transitions in electronic states that we can interpret computationally. The transitions in those electronic states are absolute or observer independent, but the computation is observer relative. The transitions in physical states are just electrical sequences unless some conscious agent can give them a computational interpretation.

This is an important point for understanding the significance of the computer revolution. When I, a human computer, add 2 + 2 to get 4, that computation is observer independent, intrinsic, original, and real. When my pocket calculator, a mechanical computer, does the same computation, the computation is observer relative, derivative, and dependent on human interpretation. There is no psychological reality at all to what is happening in the pocket calculator.

And that, believe it or not, is his argument. Oh he develops it, but really, that’s it, right there.

It seems rather like a semantic quibble that completely misses the substantive issue: can ‘intelligence’ and/or ‘consciousness’ be constructed from the kinds of circuits we use to build digital computers? Assume, for the sake of argument, that it can be done. Who the hell cares what we humans call it or attribute to it, the fact of the matter is that it’s now arguing the point with us. For all I know it might even argue that, no, it’s not intelligent; it’s all syntax, no semantics. Ain’t that a fine kettle of fish?

Return to Searle:

Except for the cases of computations carried out by conscious human beings, computation, as defined by Alan Turing and as implemented in actual pieces of machinery, is observer relative. The brute physical state transitions in a piece of electronic machinery are only computations relative to some actual or possible consciousness that can interpret the processes computationally.

See what I mean?

[And what if we choose to talk of computation in some sense other than that defined by Turing?]

Cheap criticism & cheap defense: Can machines think?

Searle’s Chinese room argument is one of the best-known thought experiments in analytic philosophy. The point of the argument as I remember it (you can google it) is that computers can’t think because they lack intentionality. I read it when Searle published in Brain and Behavioral Science back in the Jurassic Era and thought to myself: So what? It’s not that I thought that computers really could thing, someday, maybe – because I didn’t – but that Searle’s argument didn’t so much as hint at any of the techniques used in AI or computational linguistics. It was simply irrelevant to what investigators were actually doing.

That’s what I mean by cheap criticism.

But then it seems to me that, for example, Dan Dennett’s staunch defense of the possibility of computers thinking is cheap in the same way. I’m sure he’s read some of the technical literature, but he doesn’t seem to have taken any of those ideas on board. He’s not internalized them. Whatever his faith in machine thought is based on, it’s not based on the techniques investigators on the matter have been using or on extrapolations from those techniques. That makes his faith as empty as Searle’s doubt.

So, if these guys aren’t arguing about specific techniques, what ARE they arguing about? Inanimate vs. animate matter? Because it sure can’t be spirit vs. matter, or can it?

Why Philosophical Arguments about the Computational Mind Don’t Interest Me

And yet the idea of the computational mind does.

Philosophers have generated piles of arguments about whether or not or what way the mind is computational. I’ve read some of these arguments, but not many. They’re not relevant to my interests, which, as many of you know, are very much about the mind as computer. After all, in my major theoretical and methodological set piece, Literary Morphology: Nine Propositions in a Naturalist Theory of Form, the third proposition is “The form of a given work can be said to be a computational structure” and the fourth: “That computational form is the same for all competent readers.” I’m committed, and have been for years.

But the philosophical arguments, pro and con, have almost nothing to do any of the various models that have been posed and investigated through mathematical analysis and computer implementation. The philosophical arguments thus have no bearing on what interests me, the nuts and bolts of neuro-mental computation. For example, I read Searle’s (in)famous Chinese room argument when it was first published in 1980. Here’s how the Stanford Encyclopedia of Philosophy (SEP) summarizes it:

Searle imagines himself alone in a room following a computer program for responding to Chinese characters slipped under the door. Searle understands nothing of Chinese, and yet, by following the program for manipulating symbols and numerals just as a computer does, he produces appropriate strings of Chinese characters that fool those outside into thinking there is a Chinese speaker in the room. The narrow conclusion of the argument is that programming a digital computer may make it appear to understand language but does not produce real understanding. Hence the “Turing Test” is inadequate. Searle argues that the thought experiment underscores the fact that computers merely use syntactic rules to manipulate symbol strings, but have no understanding of meaning or semantics.

My response then, and now: And…? It simply doesn’t connect with anything you have to do get a model up and running.