The loss of agency and the rise of magic in AI culture

DL has a default culture of refusing to admit that we can figure out how the systems do what they do, which drives me into fits of irrational rage. It's scientific misconduct, imo. "There's 27 quadrillion parameters" is not an excuse; one can instrument and understand them.

— David Chapman (@Meaningness) July 22, 2022

That first sentence hit me hard. It’s the phrase “refusing to admit” that got to me. Why? Because it implies willful blindness. That these systems are opaque, that’s been a commonplace for years, even before the deep learning explosion of the last decade or so. But it’s one thing to accept that opacity as a condition of life and to work with it. It’s something else entirely to cultivate the sense of opacity.

Is that what’s going on – active cultivation of ignorance? I don’t know. On the one hand Anthropic was founded in the summer of 2021 with the goal of creating “interpretable, and steerable AI systems.” A system can’t be interpretable and steerable unless you know what’s going on under the hood. But the company is only a year old, so it’s not at all clear what that implies about the DL field as a whole. Perhaps things are changing.

But for the moment I’m going to go with Chapman’s assertion about the default culture. He’s right that, yes, “one can instrument and understand” these models. Why not do it?

Intellectual agency

For the sake of argument I am going to posit that an intellectual worker’s deepest sense of agency is grounded in the unvoiced intuitions that they have about the domain in which they work. Those intuitions lead them into the unknown, telling them what to look for, leading them to poke around here and there, guiding them in the crafting of explicit ideas and models.

I was trained in computational semantics in the “classical” era of symbolic computing. I read widely in the literature and worked on semantic network models. I have intuitions about the structure of natural language semantics. Others worked on syntax or machine vision.

Deep learning is very different. In deep learning you create an engine that computes over huge volumes of data to create a model of the structure existing in individual items, whether texts or images. This leads to intuitions about how these ‘learning’ engines work. But those intuitions DO NOTE translate into intuitions about the domain over which a given engine works.

Thus the DL worker’s intuitions are isolated from the mechanisms that are actually operating in the object domain. Those mechanisms are opaque. Thus they cannot have a sense of agency about those mechanisms.

Conjuring with magic

It is in this context that we have to understand “a default culture of refusing to admit that we can figure out how the systems do what they do.” Figuring out what those systems do is difficult and DL researchers have little or no training in thinking about the structure of language or visual objects beyond what is useful in constructing their engines. The whole classical world of symbolic systems this is not ‘real’ to these workers.

Since that world is not real, why not declare it off-limits and then work around it. How do we do that? With magic.

The world of deep learning is surrounded by a culture devoted to predicting when AGI will arrive. AGI? Artificial general intelligence, of course. It’s not at all well-defined, but that’s a feature, not a bug. Since it’s not well-defined, there’s not point in squabbling about how it might work. Rather, we can unite around the idea that AGI is coming. Our job then is to predict it.

Where cargo cultists in the Pacific islands would perform rituals to bring the cargo planes back, AGI cultists conduct surveys and studies to predict when AGI will arrive. It is precisely because DL researchers have no direct control over the operations of the models their engines create, that they view the arrival of AGI as something steeped in mystery. No one knows how AGI will work, though there is a widespread belief that it will involve something called “recursive self-improvement,” which is itself a mysterious process. AGI cultists fall into two general schools of thought, the gradualists, and the FOOMers, where “FOOM” is a term of art for exponential almost instantaneous emergence of AGI from near-AGI substrate created by DL researchers.

And then there is belief in the existential risk posed by this AGI technology over which we have no direct control. If we can’t control it, then surely it will turn on us. It’s the story of Prospero and Caliban, rewritten for the 21st century.

Just when a substantial part of the AI community has slipped over into magic, that is not at all obvious to me. One is tempted to point out that, where in the classical era of symbolic AI, research was centered in universities, in the DL era research has become centered in large corporations for which profit is more important than knowledge. That is certainly the case, but just what role it plays in the switch from a sense of intellectual agency to a belief in magic is not clear.

More later.