Framing my discussion of The God Test, Part 1: Rorschach, reason, and whaling – [GT-3]

I’ve got to bite the bullet: I’m just going to have to go through a bunch of (preliminary) stuff before I can really engage with The God Test. My current target is to be in a position to publish a proper review of the book in 3 Quarks Daily for the week of August 9.

Rorschach Recap

I want start by recapping the Rorschach metaphor I introduced in the previous post, More on how I’m approaching The God Test – Rorschach! [GT-2]. What I like about it is that has a shape, there’s something there, but it’s not clear what. So we have little choice but to project onto it in order to (begin to) make sense of it.

First: It is a new kind of thing, an artifact we can converse with in an open-ended and natural way. The steam engine was the same kind of thing. It was an inanimate object that moved over the surface of the earth under its own power. Previously only animals (& humans as animals) had that power. So it becomes an iron horse. Just what are AIs? What’s their nature? That’s one thing.

Second: How it works is opaque. We know how to create large language models (LLMs), but we don’t know how they work. That’s new. We may not have understood the deep physics of the steam engine, but we certainly knew how they worked.

Third: We don’t know what they portend for the future. To some extent this is a function of the first two: How can we, how should we, interact. But it is also a function of the future, which is undetermined. We just don’t know.

Rhetorical force over reason

This is an argument I made in the first working paper I published after the release of ChatGPT in November of 2022: ChatGPT intimates a tantalizing future; its core LLM is organized on multiple levels; and it has broken the idea of thinking (February 6, 2023).

What do I mean by that, has broken the idea of thinking? Prior to ChatGPT it was obvious that humans could think and computers could not. [Yeah, I know, there’s Deep Blue defeating Kasparov in chess. That just changes the dates, not the argument.] The difference in performance was so obvious that the fact that we don’t really know how humans think wasn’t much of an issue. Now it is. Sure, we can still say that we can think and the AI’s can’t, but that’s just a line and without good explanations on both sides of the line, it seems a bit arbitrary, if not desperate.

I made a particular argument about Searles’ (in)famous Chinese Room thought experiment. I read it when it was first published in Brain and Behavioral Science in 1980. I wasn’t impressed. Why not? He didn’t say anything about any of the techniques used in AI or computational linguistics (CL). How could anyone possibly take that seriously?

He talked about intention, that’s how. Meaning requires intention and only living things can have intention, a remark he made at the end of the article. Without intention the most you get is syntax, but no meaning. Searle could get away with that because, in the first place, the concept of intention has a long history within philosophy – it has a subtle meaning, but that can wait for a later post – and so philosophers, his main audience, were comfortable with it. That’s one thing.

But there’s something more important, something that we can see only in retrospect, and that’s the simple fact computers very obviously could not translate from Chinese into English or into any other language. That difference carried tremendous weight. We don’t have a subtle behavioral difference between computers and humans that requires a subtle and sophisticated argument. To a first approximation, almost any argument would do. As far as I was concerned, “intention” was just a fancy word for something we don’t understand. But that’s not an argument anyone needs to take seriously. The behavioral distance is quite sufficient to carry the argument for those who insist that computers can’t and will never be able to think like humans.

Now the behavioral evidence has changed. Sure, differences remain, but the evidence is shifting. The old arguments remain and those who believed them still do so, but it’s getting harder. The need for explicit arguments grounded in explicit accounts of computers, and also brains, is growing.

Whaling and expertise

What’s an expert in machine learning and LLMs actually expert in? For some time now I’ve been arguing that investing in AI is like investing in a whaling venture where the captain and crew of the ship know all there is to know about the ship and how to handle it but know little or nothing about whales and their behavior and about navigating around the Cape Horn and in the South Pacific, where the whales live. What are the chances of that voyage being successful? Not very good.

The people who have created the current AI technology are like that captain and crew. The know how to sail the ship. But they don’t know much about language or cognition. They don’t actually know much about the human mind. Here my point is not about the fact that the models are opaque, but that human language and cognition are highly structured and they don’t believe that one needs to know (much of) anything about that not only to build AI but to make confident prediction about the future of AI.

Gary Marcus, Subbarao Kambhampati, and others have been consistently arguing that, yes, the current technology is remarkable, but we are going to have to adopt classical symbolic techniques if we are to fully develop the technology so that we have accurate and safe systems. Marcus is arguing from his knowledge of human language and cognition. As far as I can tell, Wright doesn’t take that seriously. I know that he had Marcus on his NonZero podcast, and that he lists Marcus in his acknowledgements, but that he doesn’t discuss Marcus’s ideas. I conclude that he doesn’t take that line of argument seriously.

That’s a mistake, but this is not the place to make my own arguments on this issue. My point is simply that expertise in AI is no generally construed to encompass knowledge of, expertise in, human cognition and language. I can’t see how that is going to work out well in the future.

[Note: If you’re curious about my views, on this subject, read the article linked in the first paragraph of this section. My views all over the place here at New Savanna, particularly around the work of the mathematician Miriam Yevick. Also, check out the experimental work I’ve done with LLMs.]

DeepSeek Q & A, John Searle, Proprietor

The Chinese Room again

The line between intelligence and imitation gets thinner and harder to discern as the imitation is done at a peta scale, and the jury is the one being imitated..

— Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) (@rao2z) January 3, 2022

AI and intellectual integration @3QD

I have a new article up at 3 Quarks Daily:

Western Metaphysics is Imploding. Will We Raise a Phoenix from The Ashes? [Catalytic AI], https://3quarksdaily.com/3quarksdaily/2024/02/western-metaphysics-is-imploding-will-we-raise-a-phoenix-from-the-ashes-catalytic-ai.html

It is about philosophy, though not philosophy as it is currently practiced as an academic discipline. I like it. In fact I like it a lot.

Why? Because it’s built on a number of articles I’d previously published in 3QD as well as other work I’d published about AI over the past year, ChatGPT in particular. When I finally posted it on Sunday afternoon, it felt good, really good. “Man, I’ve got something here,” said I to myself.

When I got up early Monday morning, so early one might call it late Sunday night, I looked at the article and started glancing through it. “Holy crap!” thought I to myself, “when people start reading this they’re going to think they’ve landed in one of those classic New Yorker essays that wander all over the place before getting to the point, if there is one. What happened?”

Those are two very different reactions: “I’ve got something here” vs. “Holy crap!” Conclusion: I’ve got some work to do.

I might as well begin here and now.

Meaning, intention, and AI

One of my friends remarked, “you are too smart for me.” I took that to be a polite and diplomatic way of saying that he figured there must be something there but he sure couldn’t find it. How’d I get from his remark to that interpretation? I can tell you want it didn’t involve: conscious, deliberate thought. I simply knew that’s what he was saying. I intuited the intention behind my friend’s words, an intention that I’ve subsequently verified.

Intentionality – closely related to but not quite the same as intention – is at the heart the classic argument against AI. As far as I know that argument was first articulated by Hubert Dreyfus back in 1969 or 71’, in that time frame, but is probably best-known from John Searle’s Chinese Room argument, which first appeared in 1980 in Behavioral and Brain Sciences. That argument has been refitted for the current era, perhaps most visibly by Emily Bender, who coined the phrase “stochastic parrot” to characterize the actions of Large Language Models (LLMs).

I accept that argument. The problem is, however, that it’s one thing to have made that argument at a time when AI systems responded to human input in a relatively simple and straightforward way, which was the case when Dreyfus and Searle made their arguments. Back then the argument supplied a fairly satisfying – at least to some people – account of why AI won’t work. Now, in the face of ChatGPT’s much more impressive performance, you are asking a lot more from that argument, more, I’ve argued elsewhere, more than it can reasonably deliver.

The issue here is the gap between our first-person experience of the machine and what the machine is actually doing. Back in Searle’s time the philosophical concept of intentionality was able to account for that gap, at least for some of those familiar with the concept. In the case of ChatGPT the nature of that gap is quite different. To a first approximation, our first-person experience is that we’re conversing with a person that has a strange name, ChatGPT. Some people have strange names and stilted discourse is not uncommon. If ChatGPT is in fact a person, then there is no gap to account for. We know, however, that ChatGPT is NOT a person. It’s a machine.

We are now faced with a HUGE gap. What’s the machine doing? We don’t know. The people who built these systems can’t tell us what they’re doing – a point I make in the first section of the article after the introduction, “Views about Machine Learning and Large Language Models.” They can’t even tell themselves what the machine is doing much less craft a simplified account, based on metaphors and analogies, for the rest of us. They know how the system builds an LLM and how it accesses the LLM, but they don’t know what’s going on inside the LLM itself, with its billions and billions of parameters.

That’s one thing. This business about bridging the game between first-person experience and what’s really going on, that’s a second thing. That’s a view of philosophy articulated by Peter Godfrey-Smith, which I discuss in the second part of the article, “Philosophy’s integrative role.” “Integrative” is the word he uses for that function that philosophy plays in the larger intellectual discourse. His argument is that philosophy has largely abandoned that role and that it needs to get back to it. My argument is that nowhere is that more important than in the case of artificial intelligence.

I spend the rest of the article making that point. First, I digress into a section entitled, “Tyler Cowen, Virtuoso Infovore,” where I also discuss Richard Macksey. Cowen has recently argued, in effect, that the very greatest economists, in addition to their specialized work within economics, have also performed that integrative role on behalf of the larger intellectual pubic. Then I get to the argument I’ve been chasing all along, “Artificial Intelligence as a catalyst for intellectual integration,” which you are welcome to read.

But I want to get back to my friend’s response to my article and say a few words about that.

Intention, intuition and deduction in “intelligence”

How did my friend arrive at that statement he made to me? I don’t know. But I’m guessing it was mostly by intuition rather than explicit deductive reasoning. He’d read the article and was puzzled, conjured up our relationship and, viola! out comes the statement, “you are too smart for me.” Simple as pie.

Could he have arrived at that statement through a process of rational deduction? Possibly. How might that have gone?

ONE: FACT: The article doesn’t make sense to me.

TWO: There are three possibilities: 1) It’s nonsense, or at least deeply flawed. 2) It’s fine but too abstract for me. 3) Some combination of the first two.

THREE: PREMIS: Bill’s a smart guy. CONCLUSION: It’s probably 2 or 3. What do I say?

FOUR: FACT: Bill’s a friend. THEREFORE: I’ll give him the benefit of the doubt and base my response on 2.

FIVE: PREMIS: The article is too abstract for me. PREMIS: I’m smart. FACT: Bill made the argument. THEREFORE: Bill must be very smart...

SIX: Here’s what I’ll say: “...you are too smart for me.”

As logical arguments go, it’s rather rickety. I would hate to have to formulate it in terms of formal logic. But you get the idea. Logically, it’s a tangled mess.

In the annoying matter of text books, I leave it as an exercise for the reader to make a similar argument about how I knew what my diplomatic friend was telling me.

I do not believe that ChatGPT is capable of anything like this, though, given that there’s been tons of fiction in its training corpus, containing millions and millions of lines of dialog, it might provide a passable simulacrum in this or that case. The situation will not change when the underling LLM has more parameters and has been trained on a larger dataset, assuming there’s one to be had. The limitation is inherent in the technology.

Critics like Gary Marcus argue that LLMs need to be augmented by the capacity for symbolic reasoning if they are to be truly intelligent, whatever that is. I agree. Symbolic reasoning will get you a lot, but not a whole hell-of-a-lot in the situation I’ve been discussing here. That pseudo-deduction I just went through, symbolic reasoning will get you the capacity to do that, but in even more detail.

On that basis I don’t expect that AI and ML systems will be able to handle the nuances of human interaction in the foreseeable future, if ever. We’ve come a long way, and we have a long way to go.

Further thoughts on the Chinese Room

In my tantalizing paper I all but declared Searle’s Chinese Room argument to be obsolete on the grounds that it no longer does any useful intellectual work. Let’s take another crack at that.

Back when Searle wrote that paper, 1980 and thereabouts, we really didn’t need an explanation for why computers couldn’t think. They’re not human, that seemed adequate. Yet somehow that didn’t seem quite satisfactory. Couldn’t a device with enough rules of the proper kind reason just like us? That was the challenged posed by cognitive science in general, AI in particular.

So, Searle concocted this cockamamie thought experiment to show why that ‘classical’ AI approach wouldn’t work. His idea was to isolate the intentions of the guy in the room from the relationship between the inputs to the room and the outputs generated by it. That relationship was governed entirely by the rules. And the guy in the room didn’t make the rules. He only applied them. Thus his intention is only to apply the rules and is isolated from the content of the rules.

The upshot of this thought experiment is a more refined answer to our question: Why can’t computers think? Why? Because they lack intentionality. What’s intentionality? Well, it’s something that biological organisms have. Oh. Oh? And just what does that tell us? “Not one whole fork of a lot” I thought back then. But it was something. If only we knew what intention was, something about the relationship of the organism to the environment in which it lives, no?

Still, it was something.

Now we’ve got large language models. And they do a much more convincing imitation of humans thinking and reasoning. But is it mere imitation? Or perhaps imitation isn’t the right word, as it implies intention, and they don’t have intention, do they? Let’s call it appearance: they exhibit a much more convincing appearance of humans thinking and reasoning.

Now, if we knew just what it is that humans do when we think, well then saying, they can’t think, would have explanatory value. As we don’t know how humans think the explanatory value of that statement approaches zero. And if we knew just what it is that these LLMs are doing, why then, then we wouldn’t worry about whether or not they’re thinking, would we?

Saying that they’re NOT thinking tells us (next to) nothing about what they ARE doing and tells us (next to) nothing about ourselves.

These labels – thinking, not thinking – are little but proxies for human and computer, AND we know that on extrinsic grounds. We don’t have to examine the behavior to make that determination. We can just look at the thing that’s producing the behavior and we know what kind of thing it is. But we’re NOT making that judgment on the basis of the behavior itself.

And that was the whole point of the so-called Turing Test, no? Behavior will tell. Well, it doesn’t, not in this case. 

Addendum: The Chinese Room is, in effect, a reply to the Turing Test. The Turing Test says, “behavior will tell.” The Chinese Room says, “no, it won’t.” Well, back in the days when behavior could tell, the Chinese Room, paradoxically, was a compelling argument. Now that behavior can’t tell, it’s next to worthless.

ChatGPT on intention: The Chinese Room and Lucy on the beach, plus comedians

In my first year at Johns Hopkins I took a course called, I believe, “Types of Philsophy.” One lecture in the course was organized around a thought experiment that was one of the strangest things I’d ever heard. It went like this: Some explorers come across a desert island in the middle of nowhere. As they get out of their Zodiac and step out on the beach they see some writing. I believe it was the Lord’s Prayer, but Wordsworth’s Lucy poem, “A slumber did my spirit seal,” has also been used in this little tale. The question is this: What do those words mean? Indeed, do they mean anything at all?

My friends and I were deeply puzzled. OK, but then how did those marks get there? What kind of phenomenon could possibly have produced them? Are we dealing with some weird highly improbably quantum state?

All of which is irrelevant. The philosophical point seems to have been that meaning resides in intention. Since no human scrawled those words on the beach, they can’t possibly mean anything. QED.

A decade and a half later I read Searle’s infamous Chinese room thought experiment, which dabbles in similar philosophical ideation. I didn’t much like that either, and have spent more than a little time worrying about it here at New Savanna.

So I decided to put the matter to ChatGPT, the March 14 version. First I asked it about the Chinese Room, the philosophical tale itself, standard objections, Searle’s replies. Then I prompted it with “What about Wordsworth on the beach?” For whatever reason ChatGPT thought we were still conversing about the Chinese Room and explicated it as a counter thought experiment proposed by Dan Dennett.

That, of course, was not at all what I had in mind. Rather than quize ChatGPT about it, which I figured was likely to spiral nowhere fast, I gave it a somewhat longer prompt that was quite explicit about “A slumber did my spirit seal.” I then asked it how various philosophers would respond to it: Searle, Dan Dennett, Maurice Merleau-Ponty, and Tyler Cowen – not technically a philosophy, but a smart guy with opinions on lots of things. 

Up to this point ChatGPT's remarks have been reasonable. But I'd be wary about using them in papers to be submitted for formal publication or, for that matter, in a philosophy course. If that's what you're up to, you might what to do a bit more reading.

I decided to end by throwing caution to the winds. I asked it for the response of various comedians: Victor Borge, Robin Williams, Jerry Seinfeld. For the last a gave it a more elaborate prompt: “Imagine you are Jerry Seinfeld. Tell a story about a philosopher explaining about finding the Lucy poem scrawled in the sand on a beach on a deserted island.” ChatGPT’s response was interesting. I thought it was very funny. But it made a mistake the Seinfeld would never have made. See if you can spot it; it’s not hard.

* * * * *

Can you tell me about Searle’s Chinese Room argument?

Searle’s Chinese Room argument is a thought experiment proposed by philosopher John Searle in 1980. The argument is intended to challenge the idea that computers or any other machine can truly understand language or have a mind.

The argument goes like this: Imagine a person who speaks only English and has no understanding of Chinese is locked in a room with a set of rules written in English that allow them to respond to written Chinese questions with appropriate Chinese responses. The person receives a set of Chinese characters as input and uses the rulebook to generate a corresponding output. From the outside, it would appear as though the person in the room understands Chinese and is able to respond appropriately. However, the person in the room does not actually understand the meaning of the Chinese characters or the questions being asked.

Searle argues that this scenario is analogous to how a computer processes information. A computer can manipulate symbols and generate output based on programmed rules, but it does not truly understand the meaning of the symbols or the information being processed. Therefore, Searle concludes that computers cannot have true understanding or consciousness, and that there must be something more to human cognition than just the ability to process information symbolically.

The Chinese Room argument has been widely debated in philosophy and cognitive science, with some critics arguing that it oversimplifies the nature of computer processing and the potential for machines to achieve true understanding. However, it remains a significant and influential critique of the idea that machines can truly replicate human cognitive abilities.

What are the standard objections to it?

There are several objections to Searle's Chinese Room argument, including:

1. The Systems Reply: This objection argues that the Chinese Room does not represent the full system of the human mind. According to this objection, the person in the Chinese Room is just one part of a larger system that includes the rulebook, the Chinese characters, and the person outside the room who is receiving the responses. The system as a whole could be said to understand Chinese, even if the individual parts do not.

2. The Robot Reply: This objection argues that the Chinese Room scenario assumes that the person in the room is only manipulating symbols without any physical interaction with the world. In reality, robots that interact with the physical world and have sensory input could develop true understanding of language and meaning.

3. The Brain Simulator Reply: This objection suggests that it is possible to simulate the functions of the brain with a computer program. If a program could accurately simulate the behavior of neurons and synapses, it could lead to a conscious computer that does understand language.

4. The Other Minds Reply: This objection argues that we cannot be sure that other humans truly understand language, since we have no direct access to their thoughts. Similarly, we cannot be certain that machines do not understand language just because they do not have the same kind of consciousness as humans.

5. The Intuition Reply: Some critics of the Chinese Room argument argue that it is based on a flawed intuition about the nature of understanding and consciousness. They argue that it is possible for a machine to achieve true understanding of language and meaning, even if it does not work the same way as the human mind.

How would Searle reply to them?

Searle has responded to some of these objections to his Chinese Room argument, including:

1. The Systems Reply: Searle argues that even if the entire system as a whole is said to understand Chinese, the individual components still do not. According to Searle, the argument is not about the entire system, but rather about the claim that a machine that processes symbols alone can truly understand language.

2. The Robot Reply: Searle argues that robots that interact with the physical world and have sensory input may be able to develop some level of understanding, but this does not change the fact that machines that process symbols alone cannot truly understand language.

3. The Brain Simulator Reply: Searle argues that even if a computer program could accurately simulate the behavior of neurons and synapses, it would still be a simulation and not true understanding. He maintains that understanding requires more than just the manipulation of symbols and the following of rules.

4. The Other Minds Reply: Searle argues that the claim that other humans do not truly understand language is a skeptical argument that is not relevant to the question of whether machines can understand language. He maintains that there is a fundamental difference between the way humans understand language and the way machines process symbols.

5. The Intuition Reply: Searle argues that the intuition behind the Chinese Room argument is based on our common-sense understanding of what it means to truly understand language. He maintains that the burden of proof is on those who claim that machines can achieve true understanding, and that so far there is no evidence to support this claim.

What about Wordsworth on the beach?

The "Wordsworth on the beach" objection is a thought experiment proposed by philosopher Daniel Dennett in response to Searle's Chinese Room argument. Dennett argues that the Chinese Room scenario is like Wordsworth on the beach - just as Wordsworth can see and appreciate the beauty of the beach without necessarily knowing the names of the different rocks, shells, and other objects he sees, a machine can understand the meaning of language without necessarily knowing the meaning of every word.

Dennett suggests that the Chinese Room scenario is too simplistic in assuming that understanding language requires knowing the meaning of every word in isolation. He argues that understanding language is a matter of recognizing patterns and connections between words and concepts, and that a machine that can recognize these patterns and connections can be said to understand language.

In response, Searle maintains that the Wordsworth on the beach objection is not relevant to his argument. He argues that the Chinese Room scenario is not about knowing the names of individual objects, but rather about the deeper understanding of the meaning and significance of language. Searle maintains that a machine that simply recognizes patterns and connections between words and concepts is not truly understanding language, but rather is simply processing symbols according to rules.

Intention, in the philosophical sense, and AI

Intention, in the philosophical sense, is a bit different from the common use of the term. It includes the common usage, but is more subtle. It is often glossed as “aboutness.” When you merely see something, or smell it, hear it, touch it, even think about it, and so forth, you have a mental state that is about that thing, whatever it is. You have an intentional stance toward it. You intend it.

Intention is at the crux of Searle’s famous Chinese Room argument about artificial intelligence. He argues that, no matter how subtle and sophisticated it may be, an AI system is unable to comprehend meaning. It is only syntactic. It may be able to pass whatever Turing test you throw at it, it's just a (philosophical) zombie. Why? Because it lacks intentionality.

When I first read Searle’s argument in 1980 I thought that it was something of a cop-out – I may still think that. “Intention” was just a filler for a whole bunch of still we don’t understand. Perhaps so.

But I don’t really want to argue that here. I want to talk about intention in the philosophical sense. This is something we need to develop step by step.

Let’s start with this simple diagram:

It represents the fact that the central nervous system (CNS) is coupled to two worlds, each external to it. To the left we have the external world. The CNS is aware of that world through various senses (vision, hearing, smell, touch, taste, and perhaps others) and we act in that world through the motor system. But the CNS is also coupled to the internal milieu, with which it shares a physical body. The net is aware of that milieu by chemical sensors indicating contents of the blood stream and of the lungs, and by sensors in the joints and muscles. And it acts in the world through control of the endocrine system and the smooth muscles. Roughly speaking the CNS guides the organism’s actions in the external world so as to preserve the integrity of the internal milieu. When that integrity is gone, the organism is dead.

Now consider this diagram, which I call an intention diagram:

It is obviously an elaboration of the previous diagram. At the middle and right we have some person named “Jill.” Her internal milieu is to the right while I’ve represented her central nervous system (CNS) in the middle. At the left we have the external world.

I’ve represented two things in the external world, a rose, and a person named “Jack” – though it could just as easily be a dog or a crow, whatever. In Jill’s CNS there is some representation of that rose, represented by a grey circle. Another grey circle represents Jack. We need not worry about the nature of either of these representations; each is no doubt complex, with that for Jack being more complex.

Let us imagine that Jill sees the rose, and sees Jack. Or perhaps she only smells the rose and is listening to Jack’s voice while looking elsewhere. Maybe she senses neither, but is only thinking of them. Whatever the case may be, she has an intentional attitude toward then; she is intending them. In the philosophical sense.

Those dotted red lines indicate Jill’s intentionality, her intentional attitudes. The do not correspond to any physical signal moving from the rose to Jill or from Jack to Jill. Such signals may exist, but they would have to be represented in some other way. Those intentional lines reflect (aspects of) Jill’s intentional relationship(s) to the world around her. They depend on the whole system, not on this or that discrete part. They represent aboutness.

A general comment concerning arguments about computers and brains

Arguments about whether or not computers will ever match the powers of the human mind have been around for a long time. As far as I can recall the first such argument I gave serious attention to was John Searle’s Chinese Room argument. I found it unsatisfactory as it didn’t address any of the computational mechanisms in use. I still find that bothersome.

And yet I don’t believe that one day computers will match or exceed the general capacities of the human mind. Of course, in many domains, they already exceed our capacities. We’re not talking about them. We’re talking about something called “general intelligence.”

It seems to me that, in the end, the arguments against computer intelligence get much, if not in fact most, of their force from the fact that they are currently inferior to humans and there is no obvious immediate prospect of them catching up. On the other hand, the arguments in favor of computer intelligence (catching up to or exceeding human intelligence) get much of their force from the fact that we cannot know the future. We have a much deeper understanding of the requirements for sending humans to Mars than we do of creating the (mythical) artificial general intelligence (AGI).

This is not a very encouraging state of affairs.

The Chinese Room – I got it! I see where it’s going, or coming from. (I think)

Bump to the head of the queue. I'm thinking about this stuff. Though I should say more, I don't find that intention is very useful in distinguishing between 'real' from 'artificial' intelligence. Where do we find intention in the brain or, for that matter, the whole organism? How would we create it in an artificial being? We haven't got a clue on either score. For some other thoughts, in a somewhat different but still related context, see this post on Stanley Fish and meaning literary criticism. 

* * * * *

 

John Searle’s Chinese room argument is one of the best-known thought experiments in the contemporary philosophy of mind and has spawned endless commentary. I read it when it appeared in Behavioral and Brain Science in 1980 and was unimpressed [1]. Here’s a brief restatement from David Cole’s entry in The Stanford Encyclopedia of Philosophy [2]:

Searle imagines himself alone in a room following a computer program for responding to Chinese characters slipped under the door. Searle understands nothing of Chinese, and yet, by following the program for manipulating symbols and numerals just as a computer does, he produces appropriate strings of Chinese characters that fool those outside into thinking there is a Chinese speaker in the room. The narrow conclusion of the argument is that programming a digital computer may make it appear to understand language but does not produce real understanding. Hence the “Turing Test” is inadequate. Searle argues that the thought experiment underscores the fact that computers merely use syntactic rules to manipulate symbol strings, but have no understanding of meaning or semantics. The broader conclusion of the argument is that the theory that human minds are computer-like computational or information processing systems is refuted. Instead minds must result from biological processes; computers can at best simulate these biological processes.

The whole thing seemed to me irrelevant because it didn’t address any of the ideas and models actually used in development computer simulation of mental processes. There was nothing in there that I could use to improve my work. The argument just seemed useless to me. For that matter, most of the philosophical discussion on this has seemed useless for the same reason; it’s conducted at some remote distance from the ideas and techniques driving the research.

I remarked on this to David Hays and he replied, that yes, the philosophers will say it can’t be done but the programs will get better and better. Not mind you, that Hays thought we were on the verge of cracking the human mind or, for that matter, that I think so now. It’s just that, well, this kind of argumentation isn’t helpful.

I still believe that – not helpful – but I’m beginning to think that, nonetheless, Searle had a point. A lot depends on just what “real understanding” is. The crucial point of his thought experiment is that there was in fact a mind involved, the guy (Searle’s proxy) “manipulating symbols and numerals just as a computer does” has a perfectly good mind (we may assume). But that mind is not directly engaged in the translation. It’s insulated from understanding Chinese by the layer of (computer-like) instructions he uses to produce the result that fools those who don’t know what’s happening inside the box.

The core issue is intentionality, an enormously important if somewhat tricky term of philosophical art. David Cole glosses it:

Intentionality is the property of being about something, having content. In the 19th Century, psychologist Franz Brentano re-introduced this term from Medieval philosophy and held that intentionality was the “mark of the mental”. Beliefs and desires are intentional states: they have propositional content (one believes that p, one desires that p, where sentences substitute for “p” ).

He quotes Searle as asserting:

I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. To put this point slightly more technically, the notion “same implemented program” defines an equivalence class that is specified independently of any specific physical realization. But such a specification necessarily leaves out the biologically specific powers of the brain to cause cognitive processes. A system, me, for example, would not acquire an understanding of Chinese just by going through the steps of a computer program that simulated the behavior of a Chinese speaker (p.17).

We’ll just skip over Searle’s talk of semantics as I have come to make a (perhaps idiosyncratic) distinction between semantics and meaning. Let’s just put semantics aside, but agree with Searle about meaning.

 

The critical remark is about “the biologically specific powers of the brain.” Brains are living beings; computers are not. Living beings are self-organized “from the inside” – something I explored in an old post, What’s it mean, minds are built from the inside? Computers are not; they programmed “from the outside” by programmers. But living beings are not self-organized in isolation. They are self-organized in an environment and it is toward that environment that they have intentional states.

Brains are made of living cells, each active from the time it emerged from mitosis. And so we have growth and learning in development, prenatal and postnatal. At every point those neurons are living beings. And those neurons, like all living cells, are descended from those first living cells billions of years ago.

Searle’s argument ultimately rests on human biology and a belief that life cannot be “programmed from the outside”. Let us say that I am deeply sympathetic to that view. But I cannot say for sure that a mind cannot be programmed from the outside. Moreover I note that Searle’s argument originated before the flowering of machine learning techniques in the last decade or so.

There is a sense in which those computers do in fact “learn from the inside”. Programmers do not write rules for recognizing cats, playing Go, or translating from one language to another. The machine is programmed with a general capacity for learning and it learns the “rules” of a given domain itself [3]. As a result, we don’t really know what the computer is doing. We can’t just “open it up” and examine the rules it has developed.

Will such technology evolve to the point where these systems have genuine intentionality? We don’t know. They’re along way from in now, but who knows?

* * * * *

[1] Searle, J., 1980, “Minds, Brains and Programs”, Behavioral and Brain Sciences, 3: 417–57. Preprint available online, http://cogprints.org/7150/1/10.1.1.83.5248.pdf

Searle has a brief 2009 statement of the argument online at Scholarpedia: http://www.scholarpedia.org/article/Chinese_room_argument

[2] Cole, David, "The Chinese Room Argument", The Stanford Encyclopedia of Philosophy (Winter 2015 Edition), Edward N. Zalta (ed.), https://plato.stanford.edu/archives/win2015/entries/chinese-room/

[3] For a good journalistic account of some of the recent work, see Gideon Lewis-Kraus, The Great A.I. Awakening, New York Times Magazine, December 14, 2016:

http://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html?&hp&action=click&pgtype=Homepage&clickSource=story-heading&module=second-column-region&region=top-news&WT.nav=top-news&_r=0

On finding Donkey Kong transistors in a MOS 6502 microprocessor chip – Whoops! the methods of the neurosciences have problems, no?

This is from June 2019. I'm bumping it to the top of the queue because I'm thinking about these things.

A couple of days ago I posted a conversation with Rodney Brooks on the limitations of the computing metaphor as a vehicle for understanding the brain. Brooks mentioned an article, "Could a Neuroscientist Understand a Microprocessor?". The point of the article is that if you attempt to understand a microprocessor using the same methods neuroscientists use to understand the brain you're going to come up with gibberish.

I've located that article along with an informal account of the work in The Atlantic. I conclude some some observations of my own.

* * * * *

Jonas E, Kording KP (2017) Could a Neuroscientist Understand a Microprocessor? PLoS Comput Biol 13(1): e1005268. https://doi.org/10.1371/journal.pcbi.1005268

Abstract

There is a popular belief in neuroscience that we are primarily data limited, and that producing large, multimodal, and complex datasets will, with the help of advanced data analysis algorithms, lead to fundamental insights into the way the brain processes information. These datasets do not yet exist, and if they did we would have no way of evaluating whether or not the algorithmically-generated insights were sufficient or even correct. To address this, here we take a classical microprocessor as a model organism, and use our ability to perform arbitrary experiments on it to see if popular data analysis methods from neuroscience can elucidate the way it processes information. Microprocessors are among those artificial information processing systems that are both complex and that we understand at all levels, from the overall logical flow, via logical gates, to the dynamics of transistors. We show that the approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor. This suggests current analytic approaches in neuroscience may fall short of producing meaningful understanding of neural systems, regardless of the amount of data. Additionally, we argue for scientists using complex non-linear dynamical systems with known ground truth, such as the microprocessor as a validation platform for time-series and structure discovery methods.

Author Summary

Neuroscience is held back by the fact that it is hard to evaluate if a conclusion is correct; the complexity of the systems under study and their experimental inaccessability make the assessment of algorithmic and data analytic technqiues challenging at best. We thus argue for testing approaches using known artifacts, where the correct interpretation is known. Here we present a microprocessor platform as one such test case. We find that many approaches in neuroscience, when used naïvely, fall short of producing a meaningful understanding.

* * * * *

Ed Yong, Can Neuroscience Understand Donkey Kong, Let Alone a Brain? The Atlantic, June 2, 2016. From the article:

The human brain contains 86 billion neurons, underlies all of humanity’s scientific and artistic endeavours, and has been repeatedly described as the most complex object in the known universe. By contrast, the MOS 6502 microchip contains 3510 transistors, runs Space Invaders, and wouldn’t even be the most complex object in my pocket. We know very little about how the brain works, but we understand the chip completely. [...]

Even though the duo knew everything about the chip—the state of each transistor and the voltage along every wire—their inferences were trivial at best and seriously misleading at worst. “Most of my friends assumed that we’d pull out some insights about how the processor works,” says Jonas. “But what we extracted was so incredibly superficial. We saw that the processor has a clock and it sometimes reads and writes to memory. Awesome, but in the real world, this would be a millions-of-dollars data set.”

Last week, the duo uploaded their paper, titled “Could a neuroscientist understand a microprocessor?” after a classic from 2002. It reads like both a playful thought experiment (albeit one backed up with data) and a serious shot across the bow. And although it has yet to undergo formal peer review, other neuroscientists have already called it a “landmark paper”, a “watershed moment”, and “the paper we all had in our minds but didn't dare to write”. “While their findings will not necessarily be surprising for a chip designer, they are humbling for a neuroscientist,” wrote Steve Fleming from University College London on his blog. “This kind of soul-searching is exactly what we need to ensure neuroscience evolves in the right direction.”

Five Observations

First, I've noted at various times that I find philosophical arguments about the human mind/brain and computing to be rather empty, mainly because they don't usefully engage the ideas actually used in investigating the mind or the brain. They don't provide pointers for doing better, whether in computational or other terms. I suspect that some of my humanist colleagues are attracted to these arguments because they don't want to entertain, even at a distance, any explicit account of mental operations. Some of them might balk at mind-body dualism as an explicit intellectual program, but they are, effectively, mind-body dualists and therefor mysterians as well. That is they want the mind to be shrouded in mystery. Is humanistic thought (in that view) such that it cannot in principle embrace or approach explicit accounts of the mind? If so, why? Is this a methodological or a metaphysical commitment (does it matter?)?

Second, if the computer metaphor isn't adequate, does it nonetheless have some role to play in understanding the mind? For instance, I've been arguing that language is the simplest thing humans do that involves computation. In this view computation is a very high-level brain process. That implies, of course, that we're going to need other concepts for understanding the brain – such as control theory and complex dynamics. [I note in passing basic that  the arithmetic computation we learn in primary school is a highly constrained and specialized form of language.]

Third, we do know quite a bit about biological mechanisms at the molecular and cellular levels. But we don't yet know how those processes "add up to" a brain, through we're working on it. See, for example, the OpenWorm project, which is an attempt to simulate the roundworm Caenorhabditis elegans at the cellular level. C. elegans has 959 cells, including 302 neurons and 95 muscle cells. Then we have the Blue Brain Project, which is an attempt to simulate a rodent brain at some neuronal level. What's going on in these simulations? I note that, while Searle said nothing about biology in his original formulation of his well-known Chinese Room argument (against the computational view of mind), he has some more recent observations in which he explicitly references biology, wondering how much of biological mechanism is "essential for duplicating the causal powers of the original."

Fourth, the upshot of Searle's Chinese Room argument is that computation is a purely syntactic process. It cannot encompass meaning. It lacks intention and so cannot be about anything. All of which is to say, it cannot connect with a world outside itself. A Universal Turing Machine certainly seems to be that kind of thing, doesn't it?

Fifth, it is nonetheless interesting and telling that computers give us a way simulating anything we can describe in sufficient detail of just the right kind. Including neurons and brains.

GPT-3: Waterloo or Rubicon? Here be Dragons

I've published a new working paper. Title above, download links, abstract, table of contents, and introduction below.

Download at:

• academia: https://www.academia.edu/43787279/GPT_3_Waterloo_or_Rubicon_Here_be_Dragons_Version_2
• SSRN: https://ssrn.com/abstract=3667608
• ResearchGate: https://www.researchgate.net/publication/343444766_GPT-3_Waterloo_or_Rubicon_Here_be_Dragons

---

GPT-3 is a significant achievement.

But I fear the community that has created it may, like other communities have done before – machine translation in the mid-1960s, symbolic computing in the mid-1980s, triumphantly walk over the edge of a cliff and find itself standing proudly in mid-air.

This is not necessary and certainly not inevitable.

A great deal has been written about GPTs and transformers more generally, both in the technical literature and in commentary of various levels of sophistication. I have read only a small portion of this. But nothing I have read indicates any interest in the nature of language or mind. That seems relegated to the GPT engine itself. And yet the product of that engine, a language model, is opaque. I believe that, if we are to move to a level of accomplishment beyond what has been exhibited to date, we must understand what that engine is doing so that we may gain control over it. We must think about the nature of language and of the mind.

That is what this working paper sets out to achieve, a beginning point, and only that. By attending to ideas by Adam Neubig, Julian Michael, and Sydney Lamb, and by extending them through the geometric semantics of Peter Gärdenfors, we can create a framework in which to understand language and mind, a framework that is commensurate with the operations of GPT-3. That framework can help us to understand what GPT-3 is doing when it constructs a language model, and thereby to gain control over that model so we can enhance and extend it.

It is in that speculative spirit that I offer the following remarks.

---

Abstract: GPT-3 is an AI engine that generates text in response to a prompt given to it by a human user. It does not understand the language that it produces, at least not as philosophers understand such things. And yet its output is in many cases astonishingly like human language. How is this possible? Think of the mind as a high-dimensional space of signifieds, that is, meaning-bearing elements. Correlatively, text consists of one-dimensional strings of signifiers, that is, linguistic forms. GPT-3 creates a language model by examining the distances and ordering of signifiers in a collection of text strings and computes over them so as to reverse engineer the trajectories texts take through that space. Peter Gärdenfors’ semantic geometry provides a way of thinking about the dimensionality of mental space and the multiplicity of phenomena in the world, about how mind mirrors the world. Yet artificial systems are limited by the fact that they do not have a sensorimotor system that has evolved over millions of years. They do have inherent limits.

Contents

0. Starting point and preview 1
1. Computers are strange beasts 4
2. No meaning, no how: GPT-3 as Rubicon and Waterloo, a personal view 8
3. The brain, the mind, and GPT-3: Dimensions and conceptual spaces 16
4. Gestalt switch: GPT-3 as a model of the mind 24
5. Engineered intelligence at liberty in the world 26

0. Starting point and preview

GPT-3 is based on distributional semantics. Warren Weaver had the basic idea in his 1949 memorandum, “Translation” (p. 8). Gerard Salton operationalized the idea in his work using vector semantics for document retrieval in the 1960s and 1970s (p. 9). Since then distributional semantics has developed as an empirical discipline. The last decade of work in NLP has seen remarkable, even astonishing, progress. And yet we lack a robust theoretical framework in which we can understand and explain that progress. Such a framework must also indicate the inherent limitations of distributional semantics. This document is a first attempt to outline such a framework, as such its various formulations must be seen as speculative and provisional. I offer them so that others may modify them, replace them, and move beyond them.

It started with a comment at a blog

On July 19, 2020, Tyler Cowen made a post to Marginal Evolution entitled “GPT-3, etc.” It consisted of an email from a reader who asserted, “When future AI textbooks are written, I could easily imagine them citing 2020 or 2021 as years when preliminary AGI first emerged,. This is very different than my own previous personal forecasts for AGI emerging in something like 20-50 years…” While I have my doubts about the concept of AGI – it’s too ill-defined to serve as anything other than a hook on which to hang dreams, anxieties, and fears – I think GPT-3 is worth serious consideration.

Cowen’s post has attracted 52 comments so far, more than a few of acceptable or even high quality. I made a long comment to that post. I then decided to expand that comment into a series of blog posts, say three or four, and then to collect them into a single document as a working paper. When it appeared that those three or four posts would grow to five or six I decided that I would issue two working papers. This first one would concentrate on GPT-3 and the nature of artificial intelligence, or whatever it is. The second would speculate about the future and take a quick tour of the past.

Here is a slightly revised version of the comment I made at Marginal Revolution. This paper covers the shaded material. The rest will be covered in the second paper.

Yes, GPT-3 [may] be a game changer. But to get there from here we need to rethink a lot of things. And where that's going (that is, where I think it best should go) is more than I can do in a comment.

Right now, we're doing it wrong, headed in the wrong direction. AGI, a really good one, isn't going to be what we're imagining it to be, e.g. the Star Trek computer.

Think AI as platform, not feature (Andreessen). Obvious implication, the basic computer will be an AI-as-platform. Every human will get their own as an very young child. They're grow with it; it’ll grow with them. The child will care for it as with a pet. Hence we have ethical obligations to them. As the child grows, so does the pet – the pet will likely have to migrate to other physical platforms from time to time.

Machine learning was the key breakthrough. Rodney Brooks’ Gengis, with its subsumption architecture, was a key development as well, for it was directed at robots moving about in the world. FWIW Brooks has teamed up with Gary Marcus and they think we need to add some old school symbolic computing into the mix. I think they’re right.

Machines, however, have a hard time learning the natural world as humans do. We're born primed to deal with that world with millions of years of evolutionary history behind us. Machines, alas, are a blank slate.

The native environment for computers is, of course, the computational environment. That's where to apply machine learning. Note that writing code is one of GPT-3's skills.

So, the AGI of the future, let's call it GPT-42, will be looking in two directions, toward the world of computers and toward the human world. It will be learning in both, but in different styles and to different ends. In its interaction with other artificial computational entities GPT-42 is in its native milieu. In its interaction with us, well, we'll necessarily be in the driver’s seat.

Where are we with respect to the hockey stick growth curve? For the last 3/4 quarters of a century, since the end of WWII, we've been moving horizontally, along a plateau, developing tech. GPT-3 is one signal that we've reached the toe of the next curve. But to move up the curve, as I’ve said, we have to rethink the whole shebang.

We're IN the Singularity. Here be dragons.

[Superintelligent computers emerging out of the FOOM is bullshit.]

* * * * *

ADDENDUM: A friend of mine, David Porush, has reminded me that Neal Stephenson has written of such a tutor in The Diamond Age: Or, A Young Lady's Illustrated Primer (1995). I then remembered that I have played the role of such a tutor in real life, The Freedoniad: A Tale of Epic Adventure in which Two BFFs Travel the Universe and End up in Dunkirk, New York.

While the portion of the comment to be elaborated in the next working paper is considerably longer than the portion being elaborated in this one, I do not expect that paper to be proportionately longer. This paper covered quasi-technical matters requiring fairly careful exposition. The next paper will go by more quickly and will, in sections, approach science fiction.

* * * * *

1. Computers are strange beasts – They’re obviously inanimate, and yet we communicate with them through language. The don’t fit pre-existing (19th century?) conceptual categories, and so we are prone to strange views about them.

2. No meaning, no how: GPT-3 as Rubicon and Waterloo, a personal view – Arguing from first principles it is clear that GPT-3 lacks understanding and access to meaning. And yet it produces very convincing simulacra of understanding. But common sense understanding remains elusive, as it did for old school symbolic processing. Much of common sense is deeply embedded in the physical world. GPT-3, as it currently functions is, in effect, an artificial brain in a vat.

3. The brain, the mind, and GPT-3: Dimensions and conceptual spaces – GPT-3 creates a language model by examining the distances and ordering of signifiers in a collection of text strings and computes over them so as to reverse engineer but the trajectories texts take through a high-dimensional mental space of signifieds. Peter Gärdenfors’ semantic geometry provides a way of thinking about the dimensionality of mental space and the multiplicity of phenomena in the world.

4. Gestalt switch: GPT-3 as a model of the mind – GPT-3 creates: 1) a model of a body of natural language texts, and only a model. 2) Those texts are the product of human minds. 3) Though the application of 2 to 1 we may conclude that GPT-3 is also a model of the mind, albeit a very limited one. 3 requires a Gestalt switch.

5. Engineered intelligence at liberty in the world – The “intelligence” in systems such as GPT-3 is static and reactive. To liberate and mobilize it we need to endow AI systems with mental models of the kind investigated in “old school” symbolic AI.

Once more into the Chinese Room

I’ve been crashing on my GPT-3 working paper and I had a new thought about Searle’s infamous Chinese room [1].

Yet if you would believe John Searle, no matter how rich and detailed the world model included in an AI, understanding would necessarily elude them. When I first encountered the Chinese room argument years ago my reaction was something like: interesting, but irrelevant. Why irrelevant? Because it said absolutely nothing about the techniques AI or cognitive science investigators used and so would provide no guidance toward improving that work. He did, however, have a point: If the machine has no contact with the world, how can it possibly be said to understand anything at all? All it does is grind away on syntax.

What Searle misses, though, is the way in which meaning is a function of relations among concepts, as I pointed out earlier (see [2]). It seems to me, however – and here I’m just making this up off the top of my head – we can think of meaning as having both a intentional aspect, the connection of signs to the world, and a relational aspect, the relations of signs among themselves. Searle’s argument concentrated on the former and said nothing about the latter.

What of the intentional aspect when a person is writing or talking about things not immediately present, which is, after all quite common? In this case the intentional aspect of meaning is not supported by the immediate world. Language use thus must necessarily be driven entirely by the relations of signifiers among themselves, Sydney Lamb’s point which we have already investigated [again, see it in [2]).

Have I at long last wrestled that pesky argument to the ground?

We'll see.

* * * * *

[1] I have written a number of blog posts about this argument. Here’s one of them: Another romp around Searle’s Chinese room, New Savanna, blog post, July 18, 2018, http://new-savanna.blogspot.com/2018/07/another-romp-around-searles-chinese-room.html. You can find others at the Searle link, which, however, contains other Searle posts as well, http://new-savanna.blogspot.com/search/label/Searle.

[2] New Savanna blog post, 2. The brain, the mind, and GPT-3: Dimensions and conceptual spaces, July 29, 2020, http://new-savanna.blogspot.com/2020/07/2-brain-and-gpt-3-part-1-dimensions-and.html.

Border Patrol: Arguments against the idea that the mind is (somehow) computational in nature

I’ve been through this before, the idea that arguments such as those by Dreyfus and Searle seem curious and empty to me.

Eye-hand coordination

But before I get to that I want to acknowledge what seems to me some remarkable work by researchers at OpenAI, a robotic hand that solves Rubik’s Cube. The cube algorithm is (old school) symbolic (as Gary Marcus points out) but visual perceptual and manipulation are achieved by (new school) neural networks. I think the visuo-manipulative work is wonderful.

But does it display intelligence? I don’t much care. I think it’s wonderful. And perhaps the most wonderful aspect is the interaction between visual perception on the one hand and hapsis (touch) and movement on the other. My instinct/intuition is to say that THERE’s where you’re going to get “intelligence”, whatever that is, from that intermodal coordination. Why? Because that interaction is just a bit more abstract than either perception or movement alone; it must take place in an abstract space that encompasses both but isn’t OF either.

Of course, back in the old school world of symbolic intelligence, the sensorimotor world of manipulation was peripheral to intelligence, which was about things like theorem proving, chess, and expert systems of scientific and technical knowledge. That began to change, I believe, in the 1970s. For one thing work on natural language forced researchers to integrate perception (of the auditory signal) with cognition (semantic meaning). And that’s when things began to collapse.

But we needn’t go into that here.

So what?

I remember reading Searle’s (in)famous Chinese Room argument when it came out in Behavior and Brain Sciences in, I believe 1980, and being both puzzled and unimpressed. By that time I’d been reading quite widely in computational linguistics and written a dissertation in which I made use of computational semantics in analyzing a Shakespeare sonnet and in discussing narrative order. Searle’s argument simply did not connect with any of the many concepts and techniques used in modeling thought. Understanding Searle’s argument wasn’t going to help us develop better computer models nor, for that matter, would it be of much use to psychologists and neuroscientists.

Just what was it good for?

I suspect the same is true for the older arguments of Hubert Dreyfus, which I’ve never read in full. Of course, where Searle was arguing within the Anglo-American analytic philosophical tradition, Dreyfus was a Heideggerian arguing within the Continental tradition. The intellectual style is different, but the result is the same.

At the moment I’ve been looking at an article Dreyfus published in 2007, “Why Heideggerian AI failed and how fixing it would require making it more Heideggerian” (Artificial Intelligence 171, 2007, 1137-1160). He reviews a half century of work in AI, including the line of research pioneered by Rodney Brooks in the 1980s, which he finds akin to an idea advanced by Merleau-Ponty, “that intelligence is founded on and presupposes the more basic way of coping we share with animals” (p. 1141) and that requires giving up on the notion of internal symbolic representations of the world. That, as far as I can tell, is what he means by Heideggerian AI. He goes on to critique other varieties of “pseudo Heidegerian AI” until he arrives at Walter Freeman’s “Merleau-Pontian” neurodynamics (pp. 1150 ff).

Of which he approves. Nor am I surprised at this. I’d read quite a bit of Freeman when I was working on my book about music, Beethoven’s Anvil, and adopted his neurodynamics as the basic framework in which to understand music as a medium of group interaction. I had quite a bit of correspondence with him and know he was talking to Dreyfus and, for that matter, knew he was interested in Continental philosophy. But, I wonder, how much did he actually owe to Continental philosophy?

I don’t know. Freeman’s neurodynamic approach was pretty mature by the time he had these conversations with Dreyfus. It’s possible that he’d been influenced by Continental thought early in his career, as I had been, but I don’t actually know that. As far as I know, Freeman did all the work: laboratory observation, mathematical analysis, and computational modeling. Continental philosophy came late to his game and perhaps more as a vehicle for presenting his ideas to a wider intellectual community, and in opposition to AI, than as a fundamental source of technical insight.

For that is what is required, technical insight. Did he get technical insight from early-career reading of Continental thought? I don’t know.

A bit about how I got here

While I have never really thought the brain/mind was computational top-to-bottom, at least I don’t think I did, I have long be fascinated by the insight computation affords us into the mind and currently believe that some aspects of the mind are irreducibly computational. But not the whole shebang, top-to-bottom.

And I was certainly influenced by Continental thought in my undergraduate years, Merleau-Ponty in particular. I studied his Phenomenology of Perception, not for any course, but on my own, underlining passages in two or three colors, making marginal annotations, and indexing key passages on the end pages. Perhaps that “inoculated” me against going whole-hog for a computational view of the mind.

I was also strongly influence by Lévi-Strauss, also, of course, a Continental thinker, but of a somewhat different style. As I was interested in literature, it was his thinking on symbolic systems and, above all, mythology, that captured my attention. I like his tables, diagrams, and pseudo-mathematical expressions. It seemed to me that if THAT’s what was going on in myth, then we need more of it. And that, in turn, led me to the cognitive sciences (by way of “Kubla Khan” [1]).

When I read The Savage Mind I was particular struck by what he called the totemic operator:

There’s no need to explain it. I just liked it.

Another romp around Searle's Chinese room

First Thoughts

I don't know quite what I think about that Chinese room. When I read it in Brain and Behavior Science I was puzzled. So what, thought I to myself, so what? At the time I'd been reading widely in computational linguistics, AI, and cognitive science, as was deep into computational semantics myself. Searle's argument didn't address any of the ideas or techniques discussed in that rather broad literature. There wasn't anything in his argument that was of much use to someone either trying to figure out how the mind works or trying to get a computer to do something deep and interesting with language. As far as I know, most arguments about computers and minds are like that.

Now, let us assume for the moment that we have computer systems that deliver high quality machine translation. By high quality I mean the translations are as good as the best human translators can produce. These translations are suitable for legal and literary purposes. When Shakespeare is back translated into the original Klingon, native Klingons are pleased with the result. In that case, however, Searle's argument still holds. But it would seem rather thin and unsubstantial.

Of course, we don't have such computer systems and I don't see any on the horizon. Surely we can make better systems than we've got. But Searle's argument has little or nothing in it that tells us what to do.

So, yeah, sure, computers can't think. They're not living organisms. I don't remember whether or not that was explicit in the argument I read in BBS, but I don't remember it. I found it in some of Searle's more recent versions. And I think he's right about that, that it takes a biological organism to think. And...?

Second Thoughts

So, in order to refresh myself I looked up the argument in the Internet Encyclopedia of Philosophy and found this summary of a 1990 version of the argument:

Besides the Chinese room thought experiment, Searle's more recent presentations of the Chinese room argument feature - with minor variations of wording and in the ordering of the premises - a formal "derivation from axioms" (1989, p. 701). The derivation, according to Searle's 1990 formulation proceeds from the following three axioms (1990, p. 27):

(A1) Programs are formal (syntactic).

(A2) Minds have mental contents (semantics).

(A3) Syntax by itself is neither constitutive of nor sufficient for semantics.

to the conclusion:

(C1) Programs are neither constitutive of nor sufficient for minds.

Searle then adds a fourth axiom (p. 29):

(A4) Brains cause minds.

from which we are supposed to "immediately derive, trivially" the conclusion:

(C2) Any other system capable of causing minds would have to have causal powers (at least) equivalent to those of brains.

whence we are supposed to derive the further conclusions:

(C3) Any artifact that produced mental phenomena, any artificial brain, would have to be able to duplicate the specific causal powers of brains, and it could not do that just by running a formal program.

(C4) The way that human brains actually produce mental phenomena cannot be solely by virtue of running a computer program.

On the usual understanding, the Chinese room experiment subserves this derivation by "shoring up axiom 3" (Churchland & Churchland 1990, p. 34).

My statements should be taken as applying to that argument; note, in particular, A2 and C2. Now I said that computational semantics was well-developed by 1980. Did in thereby mean that computers have mental contents (A2)? No. I meant more or less that they had, say, morphology and syntax, but they also had something else, something that could be called semantics. Obviously that's not what Searle means by semantics (A2). Computers would have to have the causal powers of brains (C2) in order to have a mind (A4). That's all OK.

But here I am with a computer system where I make a distinction between something I call syntax and something I call semantics. I know we can do better, but Searle's argument is of no help. Nor is it of any help if I'm psychologist or neuroscientist making observations about and constructing models of human behavior and brain activity.

Third Thoughts

Upon further reflection, first and second thoughts are not fully consistent. In my first thoughts I imagined a computer system capable of high quality machine translation. Was I imagining that it had the causal powers of brains (thereby allowing it to have semantics)? No, I wasn't really. I wasn't thinking about brains until the fourth paragraph, and then just barely so. Now, if in fact we could be able to build computers having the causal powers of brains, well then, Searle's argument would appear to be OK. But whatever we did to build such computers, we didn't get any help from Searle.

Searle almost blows it on computational intelligence, almost, but not quite [biology]

John Searle has long been a critic of the pretensions of artificial intelligence to, well, you know, intelligence. He’s perhaps best know for his Chinese room argument. To parody:

Some guy’s in a room. Knows English well but doesn’t speak a lick of Mandarin. But he’s got a flotilla of yellow pads on which there’s a blizzard of instructions in English and pseudo-code for writing Mandarin. A Chinese speaker slips some statement in Mandarin through a slot in the wall. The guy goes to work with his yellow pads and blue pencils and, in due course, scribbles some Mandarin on a slip of paper and sends it back out through the slot. And thus begins a convincing ‘conversation’ in Mandarin. But, really, our guy doesn’t know a jot of Mandarin.

In Searle’s terms what our guy is doing is all syntax, no semantics. And that’s what computers do, all syntax, but not a hint of semantics.

I wasn’t convinced back then – a lot of people weren’t – and I retain that old skepticism, though there are days when I think the argument might have some merit, under a particular interpretation.

Anyhow, I just came across a 2014 article in The New York Review of Books in which Searle takes on two recent books:

Luciano Floridi, The 4th Revolution: How the Infosphere Is Reshaping Human Reality, Oxford University Press, 2014.

Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, 
Oxford University Press, 2014.

Searle’s argument depends on understanding that both objectivity and subjectivity can be taken in ontological and epistemological senses. I’m not going to recount that part of the argument. If you’re curious what Searle’s up in this business to you can read his full argument and/or you can read the appendix to this post, where I’ve quoted a number of passages from a 1995 book in which Searle lays out matters with some care.

Searle sets up his argument by pointing out that, at the time Turing wrote his famous article, “Computing Machinery and Intelligence”, the term “computer” originally applied to people (generally women, BTW) who performed computations. The term was then transferred to the appropriate machines via the intermediary, “computing machinery”. Searle observes:

But it is important to see that in the literal, real, observer-independent sense in which humans compute, mechanical computers do not compute. They go through a set of transitions in electronic states that we can interpret computationally. The transitions in those electronic states are absolute or observer independent, but the computation is observer relative. The transitions in physical states are just electrical sequences unless some conscious agent can give them a computational interpretation.

This is an important point for understanding the significance of the computer revolution. When I, a human computer, add 2 + 2 to get 4, that computation is observer independent, intrinsic, original, and real. When my pocket calculator, a mechanical computer, does the same computation, the computation is observer relative, derivative, and dependent on human interpretation. There is no psychological reality at all to what is happening in the pocket calculator.

And that, believe it or not, is his argument. Oh he develops it, but really, that’s it, right there.

It seems rather like a semantic quibble that completely misses the substantive issue: can ‘intelligence’ and/or ‘consciousness’ be constructed from the kinds of circuits we use to build digital computers? Assume, for the sake of argument, that it can be done. Who the hell cares what we humans call it or attribute to it, the fact of the matter is that it’s now arguing the point with us. For all I know it might even argue that, no, it’s not intelligent; it’s all syntax, no semantics. Ain’t that a fine kettle of fish?

Return to Searle:

Except for the cases of computations carried out by conscious human beings, computation, as defined by Alan Turing and as implemented in actual pieces of machinery, is observer relative. The brute physical state transitions in a piece of electronic machinery are only computations relative to some actual or possible consciousness that can interpret the processes computationally.

See what I mean?

[And what if we choose to talk of computation in some sense other than that defined by Turing?]