Scientology, Rationalists, and AI Doom

Over at Crooked Timber Henry Farrell has an interesting article, What OpenAI shares with Scientology. I believe, however, is that he’s really comparing Scientology with AI Doomer factions of the rationalist and effective altruist movements, which for all I know comprise a large fraction of the movement, than to OpenAI, though I’d guess that some significant percentage of OpenAI’s employees are Doomer rationalists and EAs and that the company as a whole has been influenced by those intersecting movements. For example, he says:

I’ve never read a text on rationalism, whether by true believers, by hangers-on, or by bitter enemies (often erstwhile true believers), that really gets the totality of what you see if you dive into its core texts and apocrypha. And I won’t even try to provide one here. It is some Very Weird Shit and there is really great religious sociology to be written about it.

That’s about a whole movement, one that has certainly influence OpenAI, but is also much larger than OpenAI. Both OpenAI and the Church of Scientology are corporations, but the rationalist and effective altruist movements are not. Corporations have administrative and coercive powers over their members that movements do not have, though organizations within movements may have such powers. I don’t know how many rationalists and EAs there are, but Wikipedia reports that there are estimated to be 40,000 Scientologists worldwide. There are also practitioners outside of the church in what’s called the Free Zone.

Scientology is an organized movement with 40K members worldwide. As an organized movement it has coercive power that Rationalists/AI Doom does not. And that’s why I think we need to recognize the distinction between OpenAI and the broader rationalist movement. OpenAI participates in the movement though its employees and some of its corporate beliefs and documents, but the movement is broader.

The upheaval at OpenAI is an instance of the influence this broader movement has and is having. How much is that broader movement with all that “Very Weird Shit” going to affect the development of AI technology? Given the effect that that technology has already begun to have and that it will have in the future, that’s a very important question. My guess is that the number of adherents of rationalism and effective altruism is currently smaller than the number of Scientologists. It seems to me, though, that they are potentially in a position to affect the wider evolution of culture and society in a way that Scientology is not.