As I noted at the beginning of the month the project began with a long comment posted to Marginal Revolution on July 19 [see below for a copy of that comment]. My original idea was to elaborate on that comment in a series of posts. It soon became clear, however, that things were going to more complicated.
On August 5 I issued the first working paper in the project, with the expectation that there would be a second one. That first paper is entitled, GPT-3: Waterloo or Rubicon? Here be Dragons. At that time I expected to issue a second working paper to cover the rest of the material from that original comment.
And then things became even more complicated. What happened is that I started thinking over the material in the first working paper and reading more about GPT-3. It was like when I first came to terms with topic models. The hardcore technical literature is a bit beyond me, but the surrounding explanatory material wasn’t doing it for me. For a couple of days I didn’t know whether I was coming or going.
A Plan, three more papers
Now things have cleared up. I think. At any rate I now have a plan, which is to issue not one, but three more working papers, two shorter ones and a longer one. The longer one will cover the rest of the material from the original comment below while the other two will go into greater depth on specific issues. This is the plan:
GPT-3: The Star Trek computer, and beyond
GPT-3: Bounding the Space, toward a theory of minds
Why GPT-X will fail in creating literature
I’ve been working on all three, but my current plan is to issue the future-oriented one – Star Trek computer – next, thereby covering the full scope of that original comment. I will issue the other two papers as they are ready.
But who knows, things may change. There’s no telling where a mind will go once it’s got the scent. Here’s brief notes on the other two working papers.
GPT-3: Bounding the Space, toward a theory of minds
This is really re-working and expanding on two sections from the first paper: 3. The brain, the mind, and GPT-3: Dimensions and conceptual spaces, and 5. Engineered intelligence at liberty in the world. I’ll be making sense out of this:
1. Symbolic AI: Construct a model of what the mind’s doing and run that model.
2. Machine learning: Construct a learning architecture (e.g. GPT-3), feed it piles of examples, and let it figure out what’s going on inside.
3. The question I’ve been getting to: What’s the world have to be like in order for 2 to work at all.
4. And 3 reflects back on 1: If THAT’s how the world is, what kind of (symbolic) model will produce output such that 2 will work.
And so forth
The third proposition is particularly important. That’s where the semantics of Peter Gärdenfors comes into play.
Why GPT-X will fail in creating literature
There’s a pro forma discussion of that issue: GPT-3 is not human, doesn’t have emotion, and is not creative. I suppose we could think of that as Commander Data’s problem, since he was forever fretting about it.
I suppose it’s true enough. But it doesn’t interest me. I have a much narrower and more specific issue in mind: GPT-X can’t do rhyme and neither will GPT-X. It’s a limitation that is inherent in the technology. Rhyme is a feature of how a text sounds, and the text base on which GPT-3 is built doesn’t have sound in it, nor is it at all obvious how that deficiency can be remedied.
If it can’t do rhyme, then it can’t do meter either. Nor can it do prose rhythm, which also depends, if not directly on sound, certainly on timing. Without these, GPT-X cannot do literature. At least it can’t do good literature, much less great literature. Oh, it can crank out wacky language by the bucket full, but that’s not what poetry is, and it can tell stories too. But stories are only a beginning point, not the end.
Think about it: Computers play the best chess in the world, Go too. But it’s not at all clear whether or not they’ll ever do anything more than mediocre literature. And that, I supposed, brings us back to the fact that computers aren’t human.
And they aren’t. They’re computers.
Appendix: The original comment
Here is a slightly revised version of the comment I made at Marginal Revolution. I’ve added the titles of the working papers to indicate their scope.
GPT-3: Waterloo or Rubicon? Here be Dragons
Yes, GPT-3 [may] be a game changer. But to get there from here we need to rethink a lot of things. And where that's going (that is, where I think it best should go) is more than I can do in a comment.
Right now, we're doing it wrong, headed in the wrong direction. AGI, a really good one, isn't going to be what we're imagining it to be, e.g. the Star Trek computer.
GPT-3: The Star Trek computer, and beyond
Think AI as platform, not feature (Andreessen). Obvious implication, the basic computer will be an AI-as-platform. Every human will get their own as an very young child. They're grow with it; it’ll grow with them. The child will care for it as with a pet. Hence we have ethical obligations to them. As the child grows, so does the pet – the pet will likely have to migrate to other physical platforms from time to time.
Machine learning was the key breakthrough. Rodney Brooks’ Gengis, with its subsumption architecture, was a key development as well, for it was directed at robots moving about in the world. FWIW Brooks has teamed up with Gary Marcus and they think we need to add some old school symbolic computing into the mix. I think they’re right.
Machines, however, have a hard time learning the natural world as humans do. We're born primed to deal with that world with millions of years of evolutionary history behind us. Machines, alas, are a blank slate.
The native environment for computers is, of course, the computational environment. That's where to apply machine learning. Note that writing code is one of GPT-3's skills.
So, the AGI of the future, let's call it GPT-42, will be looking in two directions, toward the world of computers and toward the human world. It will be learning in both, but in different styles and to different ends. In its interaction with other artificial computational entities GPT-42 is in its native milieu. In its interaction with us, well, we'll necessarily be in the driver’s seat.
Where are we with respect to the hockey stick growth curve? For the last 3/4 quarters of a century, since the end of WWII, we've been moving horizontally, along a plateau, developing tech. GPT-3 is one signal that we've reached the toe of the next curve. But to move up the curve, as I’ve said, we have to rethink the whole shebang.
We're IN the Singularity. Here be dragons.
[Superintelligent computers emerging out of the FOOM is bullshit.]
No comments:
Post a Comment