The concept of superintelligence just isn't that useful

Dwarkesh Patel interviews Ege Erdil and Tamay Besiroglu, co-founders of Mechanize, a startup dedicated to fully automating work. Before founding Mechanize, Ege and Tamay worked on AI forecasts at Epoch AI. [Dwarkesh is an angel investor.] The interview runs a bit over three hours and covers a lot of ground. This is the section on superintelligence, a useless concept if ever there was one.

Dwarkesh Patel 02:29:48

I get your argument that thinking about the economy-wide acceleration is more important than focusing on the IQ of the smartest AI. But at the same time, do you believe in the idea of superhuman intelligence? Is that a coherent concept in the way that you don’t necessarily stop at human level Go play and you just go way beyond it in ELO score? Will we get to systems that are like that with respect to the broader range of human abilities? And maybe that doesn’t mean they become God, because there’s other ASIs in the world. But you know what I mean, will there be systems with such superhuman capabilities?

Tamay Besiroglu 02:30:27

Yeah I mean I do expect that. I think there’s a question of how useful is this concept for thinking about this transition to a world with much more advanced AI. And I don’t find this a particularly meaningful or helpful concept.

I think people introduce some of these notions that on the surface seem useful, but then actually when you delve into them it’s very vague and kind of unclear what you’re supposed to make of this. And you have this notion of AGI which is distinguished from narrow AI in the sense that it’s much more general and maybe can do everything that a human can do on average. AI systems have these very jagged profiles of capability. So you have to somehow take some notion of average capabilities and what exactly does that mean, it just feels really unclear.

And then you have this notion of ASI, which is AGI in the sense that it’s very general but then it’s also better than humans on every task. And is this a meaningful concept? I guess it’s coherent. I think this is not a super useful concept, because I prefer just thinking about what actually happens in the world. And you could have a drastic acceleration without having an AI system that can do everything better than humans can do. I guess you could have no acceleration when you have an ASI that is better than humans at everything, but it’s just very expensive or very slow or something. So I don’t find that particularly meaningful or useful. I just prefer thinking about the overall effects on the world and what AI systems are capable of producing those types of effects.

Dwarkesh Patel 02:32:06

Yeah I mean one intuition pump here is: compare John von Neumann versus a human plucked from the standard distribution. If you added a million John von Neumanns to the world what would the impact on growth be as compared to just adding a million people from normal distribution?

Ege Erdil 02:32:25

Well I agree it would be much greater.

Dwarkesh Patel 02:32:27

Right. But then because of Moravec paradox-type arguments that you made earlier that evolution has not necessarily optimized us for that long along the kind of spectrum on which John von Neumann is distinguished from the average human. And given the fact that already within this deviation you have this much greater economic impact. Why not focus on optimizing on this thing that evolution has not optimized that hard on, further?

Ege Erdil 02:32:51

I don’t think we shouldn’t focus on that. But what I would say is, for example if you’re thinking about the capabilities of Go-playing AIs, then the concept of a superhuman Go AI, yeah, you can say that is a meaningful concept. But if you’re developing the AI, it’s not a very useful concept. If you just look at the scaling curve, it just goes up and there is some human level somewhere. But the human level is not privileged in any sense. So the question is, is it a useful thing to be thinking about? And the answer is probably not. Depends on what you care about. So I’m not saying we shouldn’t focus on trying to make the system smarter than humans are, I think that’s a good thing to focus on.

Dwarkesh Patel 02:33:31

Yeah I guess I try to understand whether we will stand in relation to the AIs of 2100 that humans stand in relation to other primates. Is that the right mental model we should have, or is it going to be a much greater familiarity with their cognitive horizons?

Tamay Besiroglu 02:33:49

I think AI systems will be very diverse, and so it’s not super meaningful to ask something about this very diverse range of systems and where we stand in relation to them.

Dwarkesh Patel 02:33:59

I mean, will we be able to cognitively access the kinds of considerations they can take on board? Humans are diverse, but no chimp is going to be able to understand this argument in the way that another human might be able to, right? So if I’m trying to think about my place, or a human’s place, in the world of the future, is a relevant concept of; is it just that the economy has grown a lot and there’s much more labor, or are there beings who are in this crucial way super intelligent?

Tamay Besiroglu 02:34:28

I mean there will be many things that we just will fail to understand, and to some extent there are many things today that people don’t understand about how the world works and how certain things are made. And then how important is it for us to have access or in principle be able to access those considerations?

And I think it’s not clear to me that that’s particularly important that any individual human should be able to access all the relevant considerations that produce some outcome. That just seems like overkill. Why do you need that to happen? I think it would be nice in some sense. But I think if you want to have a very sophisticated world where you have very advanced technology, those things will just not be accessible to you. So you have this trade-off between accessibility and maybe how advanced the world is. And from my point of view I’d much rather live in a world which has very advanced technology, has a lot of products that I’m able to enjoy, and a lot of inventions that I can improve my life with, if that means that I just don’t understand them. I think this is a very simple trade that I’m very willing to make.