Stephen Wolfram has a long ramble over at Edge, AI & The Future Of Civilization. Here's a three-paragraph thought experiment from the end:
Here's one of my scenarios that I'm curious about. Let's say there's a time when human consciousness is readily uploadable into digital form, virtualized and so on, and pretty soon we have a box of a trillion souls. There are a trillion souls in a box, all virtualized. We look at this box. In the box, there will be hopefully nice molecular computing, maybe it'll be derived from biology in some sense, but maybe not, but there will be all kinds of molecules doing things, electrons doing things. The box is doing all kinds of elaborate stuff.Then we look at the rock sitting next to the box. Inside the rock, there's all kinds of elaborate stuff going on, all kinds of electrons doing all kinds of things. We say, "What's the difference between the rock and the box of a trillion souls?" The answer will be that the box of trillion souls has this long history. The details of what's happening there were derived from the history of civilization and people watching videos made in 2015 or whatever. Whereas the rock came from its geological history, but it's not the particular history of our civilization.This question of realizing that there isn't this distinction between intelligence and mere computation leads you to imagine the future of civilization ends up being the box of trillion souls, and then what is the purpose of that? From our current point of view, for example, in that scenario, it's like every soul is playing video games basically forever. What's the endpoint of that?
It's the point about history, the contrast between geological and civilizational history, that interests me. Is in physically possible to put THAT kind of history into a (relatively small) box?