Henry Farrell on "The political economy of billionaire derangement"

Henry Farrell, The political economy of billionaire derangement, Programmable Mutter, July 15, 2026.

He opens with two somewhat different quotes about billionaires, Tyler Cowen more or less in favor of them and Tim O’Reilly deeply skeptical of them. He suggests we breed them together:

To be clear: this hybrid would be notably different to Tim’s argument, and likely actively objectionable to Tyler. I alone am to blame. But surprisingly, there is inadvertent support for some of its key elements from Peter Thiel, who is a surprisingly valuable chronicler of the conditions that give rise to billionaire derangement, as well as a living example of it (the Antichrist is here!).2 Here, I’m drawing both on lectures that Thiel gave at Stanford, as summarized by his amanuensis Blake Masters, and the resulting co-authored book, Zero To One. These sources have a lot to say about the connections between entrepreneurship, kingship, and personal eccentricity.

The conclusion I draw is that the forces that Tim identifies, combined with specific aspects of the political economy of Silicon Valley, help explain the derangement of certain Silicon Valley billionaires and their epigones. Old style princes were notorious for their tendencies to deranged behavior, which came not just from their inbreeding, but their power, and the unwillingness and inability of others to contradict or check them. So too, modern princes.

Not too long ago, many people, including Tyler, hoped that the advance of classical liberalism would go hand-in-hand with the growing power of tech billionaires, advancing both causes at once. Now, politically influential tech billionaires have visibly lost contact both with reality and with anything that could plausibly be described as classical liberal values. See the screen shotted quote above. Rather than making snide asides about billionaire derangement syndrome, it might be time for such people to confront what actual billionaire derangement means for the ideological straddle that they have relied on for so long. That is even more so, in a world where markets as well as market-makers are being devoured by the passions.

To summarize the particulars of my theory: Thiel’s lectures and book provide good, if incomplete evidence that the princely passions described by Hirschman didn’t disappear, but went underground. Commerce and power were fused into a new ideology of entrepreneurial virtù that became highly influential among the founder community in Silicon Valley. This combined with Silicon Valley’s (and popular culture’s) tendency to connect genius with eccentricity, not simply selecting people who seemed strange, but compounding their strangeness through self-reinforcing feedback loops. Finally, this all happened in an intimate social environment of founders and funders. Billionaires know each other and measure themselves and their success against those they consider peers, in a dense entangled clique that commingles high degrees of mutual influence with rivalries and jealousies.

Farrell then goes on to quote Thiel’s book at some length, analyzing while going along. And so:

In short then, Thiel - both as reported by Masters and in collaboration with him - suggests that Silicon Valley is a place where being a founder is tantamount to being a king. You have a greater chance of succeeding if you are weird, and if you succeed, your weirdness will likely feed upon itself in feedback loops of positive reinforcement. Finally, it is a closely interconnected social system where the key people are conjoined in a dense tangible clique. They observe each other all the time, and are observed by others for cues of what you need to do to become a made man. [...]

SpaceX perhaps marks a culmination of the kingly ideal, a moment in which, as Tim puts it, someone like Musk can “raise enormous amounts of capital while freeing himself from any restraints from those who provide it, so that he can spend the proceeds on Mars, humanoid robots, artificial intelligence or whatever next satisfies his ambition.” But it is also the moment in which someone who is visibly profoundly disturbed has briefly become the world’s first trillionaire. And it is one in which others want to copy him.

And then a bow to Keynes’s “animal spirits” in the market place:

I’ll finish by noting that this is just one aspect of a greater change. Deranged billionaires like Musk and Thiel are both partly responsible for this transformation, and notable symptoms of it. Still, they are not the whole of it. The reason that the SpaceX IPO temporarily succeeded, even though the numbers make no sense whatsoever, is that investment markets too are ruled by vibes. Tyler’s co-blogger, Alex Tabarrok, devoted years of his life to making the case for prediction markets, arguing that they would distill market wisdom into a more general source of knowledge on a multitude of topics. Now they are here, but all too often it’s spirits, not wisdom, that they seem to be distilling. Even more so for crypto. Matt Levine makes a joke about “Bleebzorx Tokens” to explain the worthlessness of memecoins, and someone else invents Bleebzorx Tokens to make a tidy profit.

There’s more at the link.

LLMs as cultural technologies: Four Views

Henry Farrell, Large language models are cultural technologies. What might that mean? Programmable Mutter, Aug. 18, 2025.

It’s been five months since Alison Gopnik, Cosma Shalizi, James Evans and myself wrote to argue that we should not think of Large Language Models (LLMs) as “intelligent, autonomous agents” paving the way to Artificial General Intelligence (AGI), but as cultural and social technologies. In the interim, these models have certainly improved on various metrics. However, even Sam Altman has started soft-pedaling the AGI talk. I repeat. Even Sam Altman.

So what does it mean to argue that LLMs are cultural (and social) technologies? This perspective pushes Singularity thinking to one side, so that changes to human culture and society are at the center. But that, obviously, is still too broad to be particularly useful. We need more specific ways of thinking - and usefully disagreeing - about the kinds of consequences that LLMs may have.

This post is an initial attempt to describe different ways in which people might usefully think about LLMs as cultural technologies. Some obvious provisos. It identifies four different perspectives; I’m sure there are more that I don’t know of, and there will certainly be more in the future. I’m much more closely associated with one of these perspectives than the others, so discount accordingly for bias. Furthermore, I may make mistakes about what other people think, and I surely exaggerate some of the differences between perspectives. Consider this post as less a definitive description of the state of debate than a one man presentation exchange that is supposed to reveal misinterpretations and clear the air so that proper debate can perhaps get going. Finally, I am very deliberately not enquiring into which of these approaches is right. Instead, by laying out their motivating ideas as clearly as I can, I hope to spur a different debate about when each of them is useful when and for which kinds of questions.

Gopnikism

I’m starting with this because for obvious reasons, it’s the one I know best. The original account is this one, by Eunice Yiu, Eliza Kosoy and Alison, which looks to bring together cognitive psychology with evolutionary theory. They suggest that LLMs face sharp limits in their ability to innovate usefully, because they lack direct contact with the real world. Hence, we should treat them not as agentic intelligences, but as “powerful new cultural technologies, analogous to earlier technologies like writing, print, libraries, internet search and even language itself.”

Behind “Gopnikism” lies the mundane observation that LLMs are powerful technologies for manipulating tokenized strings of letters. They swim in the ocean of human-produced text, rather than the world that text draws upon. Much the same is true, pari passu, for LLMs’ cousin-technologies which manipulate images, sound and video. That is why all of them are poorly suited to deal with the “inverse problem” of how to reconstruct “the structure of a novel, changing, external world from the data that we receive from that world.”

Interactionism

Interactionist accounts of LLMs start from a similar (but not identical) take on culture as a store of collective knowledge, but a different understanding of change. Gopnikism builds on ideas about how culture evolves through lossy but relatively faithful processes of transmission. Interactionism instead emphasizes how humans are likely to interpret and interact with the outputs of LLMs, given how they understand the world. Importantly for present purposes, cultural objects are more likely to persist when they somehow click with the various specialized cognitive modules through which human intelligence perceives and interprets its environment, and indeed are likely to be reshaped to bring them more into line with what those modules lead us to expect.

From this perspective, then, the cultural consequences of LLMs will depend on how human beings interpret their outputs, which in turn will be shaped by the ways in which biological brains work. The term “interactionism” stems from this approach’s broader emphasis on human group dynamics but by a neat coincidence, their most immediate contribution to the cultural technology debate, as best as I can see it, rests on micro-level interactions between human beings and LLMs.

Structuralism

I’ve recently written at length about Leif Weatherby’s recent book, Language Machines, which argues that classical structuralist theories of language provide a powerful theory of LLMs. This articulates a third approach to LLMs as cultural technologies. In contrast to Gopnikism, it doesn’t assume that culture’s value stems from its connection to the material world, and pushes back against the notion that we ought build a “ladder of reference” from reality on up. It also rejects interactionists’ emphasis on human cognitive mechanisms:     

A theory of meaning for a language that somehow excludes cognition—or at least, what we have often taken for cognition—is required.

Further:

Cognitive approaches miss that the interesting thing about LLMs is their formal-semiotic properties independent of any “intelligence.”

Instead of the mapping between the world and learning, or between the architecture of LLMs and the architecture of human brains, it emphasizes the mappings between large scale systems. The most important is the mapping between the system of language and the statistical systems that can capture it, but it is interested in other systems too, such as bureaucracy.

Language models capture language as a cultural system, not as intelligence. … The new AI is constituted as and conditioned by language, but not as a grammar or a set of rules. Taking in vast swaths of real language in use, these algorithms rely on language in extenso: culture, as a machine.

The idea, then, is that language is a system, the most important properties of which do not depend on its relationship either to the world that it describes or to the intentions of the humans who employ it.

Role play

Weatherby is frustrated by the dominance of cognitive science in AI discussions. The last perspective on cultural technology that I am going to talk about argues that cognitive science has much more in common with Wittgenstein and Derrida than you might think. Murray Shanahan, Kyle McDonell and Laria Reynolds’ Nature article on the relationship between LLMs and “role play” starts from the profound differences between our assumptions about human intelligence and how LLMs work. Shanahan, in subsequent work, brings this in some quite unexpected directions.

I found this article a thrilling read. Admittedly, it played to my priors. I first came across LLMs in early/mid 2020 thanks to “AI Dungeon,” an early implementation of GPT-2, which used the engine to generate an infinitely iterated role-playing game, starting in a standard fantasy or science fiction setting. AI Dungeon didn’t work very well as a game, because it kept losing track of the underlying story. I couldn’t use it to teach my students about AI as I had hoped, because of its persistent tendency to swivel into porn. But it clearly demonstrated the possibility of something important, strange and new.

There's much more at the link.

Needless to say, I am very sympathetic to this line of thinking. 

Cultural Technology, Old School (in Jersey City)

When tech CEOs are like grumpy ducklings

That's the title of Henry Farrell's interview with Le Grand Continent in Programmable Mutter.

Louis de Catheu - Something that is quite striking with the second Trump administration is the part taken by the Tech Bros in his entourage and his coalition. To you, what are the causes of this new alignment between the MAGA movement and Silicon Valley active bosses?

Henry Farrell - There are a number of areas where they share common interests. One thing, which is always very important to remember, is that they have common enemies. Many people on the Tech right are very much opposed to some of the measures that the Biden administration was associated with.

The Tech right also harks back to a particular vision of the politics of technology. [...] It more or less suggests: wouldn't it be incredible if we lived in a world in which we did not have to worry about government anymore?

That vision is very clearly part of the animating vision of Peter Thiel. [...] This will be a new world order in which technology founders will effectively be treated almost as the God Kings of these micro communities. People would be able to move back and forth between these communities according to Albert Hirschman’s logic of exit. [...]

The second thing that is important is a much more immediate set of political tensions, some of which are purely and simply due to people in the Silicon Valley right feeling that they did not get sufficient recognition from the Biden administration. [...]

The third thing is that there is an extreme hostility towards unions in Silicon Valley, which goes across the far right through to moderate left Democrat leaning people among the founders. The research of Neil Malhotra, David Broockman, and Gregory Ferenstein, suggests that in many ways Silicon Valley founders and funders are pretty left wing when it comes to a variety of social issues. They are more left wing than you might expect with respect to certain kinds of distributional issues, such as, being more favorable to basic income or to the distribution of welfare. But they are virulently hostile towards anybody who tries to tell them how to do their business. [...]

Opposition to woke is a significant part of the new way of thinking about things, but this is not because people have opposition to individual lifestyles, to people getting gender alteration surgery or any of these things. On principle, many people in Silicon Valley often tend to be radically libertarian when it comes to people's lifestyles. What they really objected to was the possibility that other people could tell them what they could or could not do to shape the workplace of their firms. These three forms of opposition helped to explain why you saw the Big Tech and Trump coming together.

Great Men:

You just spoke about God Kings and the need for respect. That make me think about your writings on the Silicon Valley canon. Some founders in Silicon Valley seem to have very specific views on history, the great men and their own place in history.

When I said that there was a lack of respect, I think that this involves changes in the broader conversation. Over the last number of years, Silicon Valley CEOs and venture capitalists have been treated with a significant degree of worship by the United States press — until around 2015-2016.

If Silicon Valley people wanted to pronounce on something, their grand statements were treated with enormous respect by the press. They had entourages, as if they were presidents. Mark Zuckerberg and other people were deemed by many to be world historical figures.

That, unsurprisingly changed the sense among Silicon Valley people about what their world historic role was. Here, one of the key essays for me is Marc Andreessen's piece, from 2011, called “Why Software is Eating the World,” which suggests that he and his colleagues are going to bring through a fundamental transformation in the way of things. [...]

When you talk to Silicon Valley people, you get the sense that they either read a lot or —which is in some ways nearly the same thing— they pretend to read a lot, because being well read is viewed as being part of the culture.

That is very different from 15 years ago. When my friend Aaron Swartz was alive and was still part of the debate, he used to be extremely grumpy about the lack of intellectual curiosity among Silicon Valley people. Now Silicon Valley people at least feel that they are obliged to pretend to read — and some of them actually do read. [...]

I think that the reason why Silicon Valley leaders read is because they are looking for models of how to act in the world as great men. [...]

We're in a world now where Silicon Valley people really began to identify with the idea that they were great men. This is not to say that great men are perfect, but they are people with very large virtues — and very large flaws as well. Silicon Valley founders were going to be world historical colossuses, bestriding the stage, reshaping the world in their image.

DOGE and its aftereffects:

First of all, there's DOGE's official goal, which is to achieve efficiency in the federal government. Indeed, there were probably a couple of people attached to DOGE for whom that was actually their primary goal.

A second goal was to effectively eliminate large chunks of the federal government. Of course, if you look at where DOGE acted, very often it was acting in those parts of the federal government, which were left coded rather than conservative coded. [...]

Finally, you could argue that a significant part of DOGE and efforts associated with it involved Elon Musk looking to position his own people in places of power. [...]

The first part, the efficiency part, I think never really took off in any very serious way. Perhaps under different Republicans or under Democrats, you might begin to see that take off again seriously. I don't think that was ever really a serious part of the project, as opposed to an ideological justification for it.

The second part, trying to rip away large chunks of the US government, I think is going to continue because it does not just involve DOGE, but is also about the people associated with Project 2025.

The final part - Musk installing his own people into the government - is obviously now unlikely to succeed. A lot of people are probably not going to continue on in the same way, or they will be asked to sign different loyalty oaths than the one that they felt that they were signing when they first came in. [...]

My personal theory is that if you really want to understand the Trump administration, read the autobiography of Cardinal de Retz or similar people who worked in these courts. The modern equivalents don’t have titles of nobility - perhaps they might like to - but they are engaged in the same squalid intrigues based around personal advancement, sex, rivalry. All of these things are going to be part of the Trump administration in a way that they were not part of the Biden administration — which of course had its own rivalries, but they tend to be much more technocratic ones.

AI and AGI:

We're now in a different world. It is very hard to say exactly how the Trump administration thinks about AI and AGI. It has not come out with any explicit documentation. But from what we can tell, this is much less of a coherent vision that the Biden administration had — whether you liked it or not — and much more a question of convergence between the interests of particular powerful companies and the national interest of the United States, which is defined in a somewhat flexible way.

The deals that were done in the Gulf a few weeks ago, or the apparent deal that has been done between the United States and China, which we have not seen the details of, suggest a much more flexible approach to AI. The United States still wants to be the country which really dominates the AI debate and discourse, but it also wants to take full advantage of the opportunities to do deals around AI, which will cement not simply US power, but also the power of particular people within the administration and of particular business interests which have strong connections with the administration. [...]

Throughout all of this we will see a perpetual opportunism in place of planning and strategy. I think that this is going to be a world of deal-making, rather than a world of the United States trying to fulfill a grand vision as it has in the past.

There's much more at the link.

Henry Farrell on science fiction, AI hype, and the need for usable futures

Henry Farrell, We need to escape the Gernsback Continuum, Programmable Mutter, July 2, 2025. The three opening paragraphs:

As I admitted at the time, my review essay on Adam Becker’s More Everything Forever: AI Overlords, Space Empires and Silicon Valley’s Crusade to Control the Fate of Humanity was a bit of a bait-and-switch. Becker’s book explains how the dreams of science fiction have shaped Silicon Valley’s dreams of technology in general. I deliberately made a comparison that was both narrower and more fantastical: emphasizing how debates over AGI resembled the dreams of Renaissance alchemy. In partial redress, here’s a more specific argument about the relationship between science fiction and Silicon Valley.

Again, it’s more a riff on Becker than a bald presentation of his argument, but the connections are much clearer, even if it isn’t quite the same argument that Becker makes. What Becker sees as rooted in 1950s and 1960s science fiction arguably goes back a few decades further: to the fusion of “scientifiction” and technocracy that happened in the early 1930s, right at the beginning of the so-called “Golden Age” of science fiction.

Silicon Valley is trapped in a new version of the Gernsback Continuum - a situation in which it is collectively haunted by the visions of an imaginary future of endless expansion that didn’t happen and never will. Our escape route, as Becker suggests, is to acknowledge the physical and social limits that we can’t escape, and to try to construct better futures within those limits.

Then we have a lot of this and that, about Becker's argument, about the Golden Age of science fiction, William Gibson, Hugo Gernsback, visions of the future, limitless energy, and Andreessen's techno-optimism. Then:

Equally, not all people who embrace this style of thinking are fascists, or even slightly fascist adjacent. If you read the liberal-leaning technocratic utopianism of Demis Hassabis or Dario Amodei, you’ll find the suggestion there too that technology - and in particular AI - is a limitless cornucopia of possibilities. See Hassabis’ WIRED interview a few weeks ago:

If everything goes well, then we should be in an era of radical abundance, a kind of golden era. AGI can solve what I call root-node problems in the world—curing terrible diseases, much healthier and longer lifespans, finding new energy sources. If that all happens, then it should be an era of maximum human flourishing, where we travel to the stars and colonize the galaxy. I think that will begin to happen in 2030.

Again, this leans on a particular reading of science fiction. The future that they aspire to is explicitly the future of Iain M. Banks’ Culture novels, in which vast intelligent AI Minds underpin a civilization in which people can do more or less whatever they want.

Amodei:

I think the Culture’s values are a winning strategy because they’re the sum of a million small decisions that have clear moral force and that tend to pull everyone together onto the same side. Basic human intuitions of fairness, cooperation, curiosity, and autonomy are hard to argue with, and are cumulative in a way that our more destructive impulses often aren’t. It is easy to argue that children shouldn’t die of disease if we can prevent it, and easy from there to argue that everyone’s children deserve that right equally.

Hassabis:

I would recommend “Consider Phlebas” by Iain Banks, which is part of the Culture series of novels. Very formative for me, and I read that while I was writing Theme Park. And I still think it’s the best depiction of a post-A.G.I. future, an optimistic post-A.G.I. future, where we’re traveling the stars and humanity reached its full flourishing.

On the one hand, this is obviously much more attractive, and far less sinister, than the Andreessen version. It’s an optimistic liberal bet on the boundless cornucopia of technology. Banks was a Scots socialist who detested authoritarianism with a passion.

On the other, Banks was making a much more complex and ambiguous argument than the version that Hassabis and Amodei present. The Culture is, to steal Ursula K. Le Guin’s term, an ambiguous utopia, in a universe that is emphatically not a mere backdrop for the playing out of the manifest destiny of mankind.

After some discussion of Iain Banks we get:

As Kim Stanley Robinson (another poet of hard physical limits) emphasizes in his recent work, we are almost certainly not going to be able to spread out among the stars. What we have on this planet is pretty well what we have. That doesn’t mean that we can’t do much better than we are doing. We can achieve greater forms of material abundance. But pretending that the hard problems simply don’t exist, or that they will be solved by some magical technology that is right around the corner, is a recipe for the embrace of fascism at the worst, and starry-eyed ingenuous optimism at the best. As I’ve written before, we need usable futures. But to get there, as Becker argues, we need to discard the imagined futures of a past world whose aspirations and understanding of the world are a very poor fit with the present that we find ourselves in.

Amen. Usable futures. That's what I'm up to:

• Kisangani 2150: Homo Ludens Rising, A Working Paper
• Chapter Outline: Homo Ludens Rising: A Manifesto for the Fourth Arena

Cryptocracy: Crypto Is Good for Trump but Bad for America

Dan Davies and Henry J. Farrell, Crypto Is Good for Trump but Bad for America, NYTimes, May 23, 2025.

This has been a good week for America’s crypto interests. The Genius Act, which legitimates a kind of cryptocurrency called stablecoins, advanced in the Senate, and on Thursday, President Trump held a gala dinner for the top 220 holders of his $Trump memecoin.

That doesn’t mean it’s been a good week for America. Stablecoins, as their name suggests, are crypto assets guaranteed by other assets like the U.S. dollar. Mr. Trump and his sons created one called USD1 through their cryptocurrency company, World Liberty Financial. Digital currencies like stablecoins are bad enough when they could potentially be used for political self-dealing. The potential problems they pose to the mainstream financial system go deeper and are much more concerning.

Their advocates claim that they will enhance U.S. financial power — Mr. Trump said that stablecoins would “expand the dominance of the U.S. dollar.”

They are likely instead to undermine it, fostering scams and sanctions evasion, generating financial risk and perhaps even allowing another currency to supplant the dollar in global trade. [...]

Crypto interests want to break down the boundary between cryptocurrencies and regulated finance by integrating stablecoins into the regular U.S. financial system. That would allow them to go back and forth between the swashbuckling world of crypto, where cryptocurrencies swing wildly in value and you can gamble on the latest meme, and the rule-bound world of regulated finance, with assets and bank accounts protected by the Securities and Exchange Commission and Federal Deposit Insurance Corporation.

After some argument they conclude:

Stablecoins were supposed to leverage dollars to stabilize the chaotic universe of crypto. Instead, they seem set to infect the dollar-dominated financial system with the unique combined chaos of crypto and Mr. Trump.

There's more at the link.

Large AI models are cultural and social technologies

Henry Farrell, Alison Gopnik, Cosma Shalizi, and James Evans, Large AI models are cultural and social technologies, Science, 13 Mar 2025, Vol 387, Issue 6739 pp. 1153-1156, DOI: 10.1126/science.adt9819

Abstract: Debates about artificial intelligence (AI) tend to revolve around whether large models are intelligent, autonomous agents. Some AI researchers and commentators speculate that we are on the cusp of creating agents with artificial general intelligence (AGI), a prospect anticipated with both elation and anxiety. There have also been extensive conversations about cultural and social consequences of large models, orbiting around two foci: immediate effects of these systems as they are currently used, and hypothetical futures when these systems turn into AGI agents—perhaps even superintelligent AGI agents. But this discourse about large models as intelligent agents is fundamentally misconceived. Combining ideas from social and behavioral sciences with computer science can help us to understand AI systems more accurately. Large models should not be viewed primarily as intelligent agents but as a new kind of cultural and social technology, allowing humans to take advantage of information other humans have accumulated.

Here's an ungated version.

Henry Farrell: Biden moves to control global AI

Henry Farrell, America’s plan to control global AI, Programmable Mutter, Jan. 15. 2025.

The idea is to use export controls to restrict the selling and use of to achieve two U.S. policy goals. The first is its desire to keep the most advanced AI out of the grasp of China, for fear that China will use strong AI to undermine U.S. security. The second is its desire to allow some degree of continued access to semiconductors and AI in most countries, to mitigate the anticipated shrieks of protest from big U.S. firms that don’t want to see their export markets disappear.

Hence, this highly complex plan involves controlling access to the advanced semiconductors that are used to train advanced AI models, as well as the model ‘weights’ themselves. The plan continues to very sharply restrict China’s and some other countries’ access to highly advanced semiconductors [...] It allows a much more liberal regime of exports without much in the way of controls to a small group of ‘Tier 1’ countries - important allies and other friendlies such as Norway and Ireland. Finally, there is a large intermediary zone of other countries, including some traditional U.S. allies, that will be allowed access to U.S. semiconductors, but under complex restrictions.

The whole shebang “is intended to cement U.S. power over information technology over the longer term” and depends on “five distinct bets; two on technology, and three on politics.” The technology bets are on 1) scaling and 2) AGI. The political bets are on the 3) effectiveness of export controls, 4) organizational capacity, and 5) politics. I want to comment on 1 and 2 and give you bit of Farrell on 5.

Scaling

The most straightforward bet behind this policy is that the “scaling hypothesis” is right. That is, (a) the more computer power is applied to training AI, the more powerful it will be, and (b) access to the most advanced parallel processing semiconductors is essential to building cutting edge AI models. If this is so, then the U.S. has a possible trump card. U.S. based and dependent companies like Nvidia and AMD, that design the cutting edge semiconductors that are used for training AI, have a considerable advantage over their competitors. China and other U.S. rivals and adversaries have no equivalent producers, and are obliged to rely on the inferior chips that they can make themselves, or that the U.S. allows them access to.

If this bet is right, then the U.S. indeed potentially possesses a chokehold that might allow it to shape the world’s AI system, selectively providing access to those countries and companies that it favors, while denying access to those it does not. Controlling the chips used for training, while restricting the export of AI weights, will allow it to shape what other countries do.

There is, however, some possible evidence suggesting that the relationship between chips and scaling is more complicated than the US might like.

Farrell goes on to mention DeepSeek, a powerful Chinese LLM “that it has trained a frontier AI model without access to the most advanced semiconductors.” Beyond that, I just don’t think that scaling alone is the key to the kingdom. As Gary Marcus, Yann LeCun (just search on the names) and others have been arguing, we need new architectures.

AGI

As you know, I think the term itself (artificial general intelligence) as all but meaningless. AGI’s about as real as the Holy Grail and likely springs from similar psycho-cultural desires.

Farrell notes:

One other belief, which is quite widespread among people in the U.S. national security debate as well as many in Silicon Valley, is that we are on the verge of real AGI - ‘artificial general intelligence.’ In other words, we are about to witness a moment where there will be a vast leap forward in the ability of AI to do things in the world, creating self reinforcing dynamics where those with strong AI are going to be capable of creating yet stronger AI and so on in a feedback loop. This then implies that short term AI superiority over the next couple of years might lead into a long term strategic advantage.

Farrell is skeptical:

Here, for example, Arvind Narayanan and Sayash Kapoor argue that we should be skeptical about the hype that is bubbling out right now from inside the big AI companies.

Industry leaders don’t have a good track record of predicting AI developments. … There are some reasons why we might want to give more weight to insiders’ claims, but also important reasons to give less weight to them. … there’s a huge and obvious reason why we should probably give less weight to their views, which is that they have an incentive to say things that are in their commercial interests, and have a track record of doing so.

There is a lot more in Narayanan and Kapoor’s article, about the specifics of what is happening right now, as we (perhaps) move from one model of AI development to another. I find their arguments compelling - your own mileage may of course vary.

Yes, great things will one day be possible, but not as long as the techbros keep leading us down the path of scaling up LLMs and forms of deep learning. We need new architectures and that’s going to require some fundamental research, research that won’t happen as long as scaling sucks up all the resources, financial, technological, and intellectual.

Politics

None of this will happen if the Trump administration doesn’t want it to. And there are clearly Republicans who are listening to industry protests, and promising to do what they can to get the plan reversed. A lot of people are speculating that the plan is dead on arrival.

That may be premature. One plausible interpretation is that the Biden people are trying to create facts on the ground that will bolster China hawks in the incoming administration, who want strong technology restrictions, so that they have a greater chance of prevailing over the people who want to let technology rip. And that might perhaps work!

It isn’t just the foreign policy people who want sharp restrictions on China. It is also some important people in the AI debate. Pottinger is probably not going to be coming back in (he demonstrated Insufficient Loyalty to the Beloved Leader in the days surrounding January 6 2021) but his co-author, Amodei reflects a general hawkish turn among many people in Silicon Valley. [...]

I don’t feel particularly confident in making any predictions about what the Trump administration will do. I am not the person you ought turn to for accurate gossip about who has influence among the people who are about to take power. But I don’t see any unambiguous signals (yet) that the one side or the other has the upper hand in the internal arguments.

There’s much more at the link.

AI and collaboration [superintelligence]

Over at Marginal Revolution Tyler Cowen has posted a paragraph from an interview with the mathematician, Terence Tao:

With formalization projects, what we’ve noticed is that you can collaborate with people who don’t understand the entire mathematics of the entire project, but they understand one tiny little piece. It’s like any modern device. No single person can build a computer on their own, mine all the metals and refine them, and then create the hardware and the software. We have all these specialists, and we have a big logistics supply chain, and eventually we can create a smartphone or whatever. Right now, in a mathematical collaboration, everyone has to know pretty much all the mathematics, and that is a stumbling block, as [Scholze] mentioned. But with these formalizations, it is possible to compartmentalize and contribute to a project only knowing a piece of it. I think also we should start formalizing textbooks. If a textbook is formalized, you can create these very interactive textbooks, where you could describe the proof of a result in a very high-level sense, assuming lots of knowledge. But if there are steps that you don’t understand, you can expand them and go into details—all the way down the axioms if you want to. No one does this right now for textbooks because it’s too much work. But if you’re already formalizing it, the computer can create these interactive textbooks for you. It will make it easier for a mathematician in one field to start contributing to another because you can precisely specify subtasks of a big task that don’t require understanding everything.

One of the regulars at Marginal Revolution, rayward, posted this comment:

Less collaboration? "It (AI) will make it easier for a mathematician in one field to start contributing to another because you can precisely specify subtasks of a big task that don’t require understanding everything."

Lawyers (I'm one) know a little about a lot not a lot about a little; thus, they are dependent on collaboration. Over my career many of the projects referred to me came from other lawyers, and vice versa. In the process of collaborating, the other lawyers learn a little from me and I learn a little from them, and hopefully the client is better for it.

I'm no economist (as Geithner liked to remind people), but my impression is that they work in silos, intentionally insulating themselves from outside influences: economics is very much driven by a certain way of defining and addressing a problem, reflected in the various "schools" of economics such as the Austrian School or the Keynesian School). Collaboration in this setting would be equivalent to MTG collaborating with AOC: it ain't happening. Sure, law at the highest level (e.g., the Supreme Court) is ideological, but in the real world of solving real problems for actual clients, it's not.

So which is it: will AI make economists (and others) more or less likely to collaborate?

Here’s how I replied to rayward:

Interesting. And that's the issue that this project raises for me: What kinds of projects & enterprises can be collaborative and which cannot? As I recall the Higgs boson paper from the super-collider had over a thousand signatures. That's a very large scale enterprise. In contrast, just about everything in literary criticism, the discipline I'm trained in, is done by a single person. Some disciplines lend themselves to collaboration, some do not. I suspect that AI will increase the range of collaborative work. And that's where we get the real superintelligence.

What do we know about the characteristics of projects & enterprises that make the amenable to collaboration or resistant to it?

Below the asterisks I have appended a passage from a recent post, How smart could an A.I. be? Intelligence in a network of human and machine agents.

* * * * *

So, let us think in terms of problem-solving by networks of specialized solvers. Some of those solvers are human, but some will be machines. Such man-machine problem-solving networks are ubiquitous in the modern world and they solve problems well-beyond the capacity of individual humans. They aren’t what most AI experts have in mind when they talk about superintelligence, but it’s not clear to me that we can simply ignore such networks in these discussions. They are, after all, how many very important problems get solved.  Henry Farrell and Cosma Shalizi have made this argument in The Economist (here’s an ungated and somewhat longer version, and here as well, where it is followed by a brief discussion).

I assume that such man-machine networks will proliferate in the future. Some of the nodes in these networks will be machines and some will be humans. The question of AGI then becomes:

Will there ever come a time when the tasks of every node in such problems-solving networks can be executed by a computer system that is as capable as any human?

Note that it is possible that some tasks will require manipulation of the physical world that is of such a nature that humans are better at it than any machine. Would we say that the existence of such nodes is evidence only of physical skill, but not of intelligence?

The question of machine superintelligence would then become:

Will there ever come a time when we have problem-solving networks where there exists at least one node that is assigned to a non-routine task, a creative task, if you will, that only a computer can perform?

That’s an interesting question. I specify non-routine task because we have all kinds of computing systems that are more effective at various tasks than humans are, from simple arithmetic calculations to such things solving the structure of a protein string. I fully expect the more and more systems will evolve that are capable of solving such sophisticated, but ultimately routine, problems. But it’s not at all obvious to me that computational systems will eventually usurp all problem-solving tasks.

How smart could an A.I. be? Intelligence in a network of human and machine agents

This continues the line of thinking I began with Intelligence, A.I. and analogy: Jaws & Girard, kumquats & MiGs, double-entry bookkeeping & supply and demand, which was focused specifically on analogical thinking. I now want to consider thinking more generally.

The general problem with thinking about AGI (artificial general intelligence) and superintelligence is that the idea of intelligence itself is vague. We’ve got the general idea that intelligence is the ability to solve a wide range of problems in a wide range of environments, which is a rather vague notion. There is another notion, independent of that, that conceives of intelligence as being to cognitive performance as horsepower is to engine performance. Conceived this way intelligence is a scaler quantity. That’s convenient, but not very convincing. Still...

Let’s start with that second idea. One corollary I’ve seen here and there is that a superintelligent AI would be to us as we are to, say, a mouse, or a bird, a fish, whatever animal you choose. The point seems to be that the intelligence “ceiling” of animals is fixed by their biology and is well below the intelligence ceiling of humans. And so it is with humans and a Superintelligent AI.

But is it actually the case that the intelligence ceiling of humans is fixed by human biology? Newton is able to solve problems that are beyond Aristotle, and Aristotle is able to solve problems that are beyond that of the most skilled hunter-gatherer. What is more, a merely competent college undergraduate in the current world is able to learn Newton’s concepts and methods and solve the same problems that Newton. That same college undergraduate can even solve problems beyond Newton’s competence. Why? Because physics did not stop with Newton. Our college undergraduate will have learned some of that more advanced physics and therefore have problem-solving capacities beyond those of Newton.

We have no reason believe that the biological aspect of human intelligence has increased over time. But there is a cultural aspect, and that has changed. Human intelligence is not fixed in the way that animal intelligence is. David Hays have published a series of articles about this process; the central article is The Evolution of Cognition (1990). In that article we also suggested that there is no reason to believe that the process has come to a halt. Cultural evolution seems to be ongoing.

The long-term evolution of human culture suggests that human intelligence is not properly conceived of as a function some biologically given computational capacity, for that biological capacity seems to have remained constant while our ability to solve problems has increased enormously. The way in which that capacity is organized would seem to be important – which is the foundation of the article Hays and I made. I note further, and this is not something that Hays and I discussed directly, that as the human capacity for problem-solving has increased, that capacity has become more and more a collective one. To a first approximation, every adult in a hunter-gatherer society possesses the full inventory of that society’s knowledge – though we have to allow for differences between male and female knowledge and some specialized knowledge for shamans and story-tellers. That changes with more advanced forms of social organization where knowledge becomes specialized. Knowledge has become very specialized indeed in our current world. Any number of problems now require interaction among diverse teams of specialists.

So, let us think in terms of problem-solving by networks of specialized solvers. Some of those solvers are human, but some will be machines. Such man-machine problem-solving networks are ubiquitous in the modern world and they solve problems well-beyond the capacity of individual humans. They aren’t what most AI experts have in mind when they talk about superintelligence, but it’s not clear to me that we can simply ignore such networks in these discussions. They are, after all, how many very important problems get solved. Henry Farrell and Cosma Shalizi have made this argument in The Economist (here’s an ungated and somewhat longer version, and here as well, where it is followed by a brief discussion).

I assume that such man-machine networks will proliferate in the future. Some of the nodes in these networks will be machines and some will be humans. The question of AGI then becomes:

Will there ever come a time when the tasks of every node in such problems-solving networks can be executed by a computer system that is as capable as any human?

Note that it is possible that some tasks will require manipulation of the physical world that is of such a nature that humans are better at it than any machine. Would we say that the existence of such nodes is evidence only of physical skill, but not of intelligence?

The question of machine superintelligence would then become:

Will there ever come a time when we have problem-solving networks where there exists at least one node that is assigned to a non-routine task, a creative task, if you will, that only a computer can perform?

That’s an interesting question. I specify non-routine task because we have all kinds of computing systems that are more effective at various tasks than humans are, from simple arithmetic calculations to such things solving the structure of a protein string. I fully expect the more and more systems will evolve that are capable of solving such sophisticated, but ultimately routine, problems. But it’s not at all obvious to me that computational systems will eventually usurp all problem-solving tasks.

Remember, that even as we’re developing ever more capable AI systems, we are also developing more sophisticated modes of human problem solving. It’s not at all obvious that machines will necessarily out-run us. Take a look at the analogy paper I linked in the first paragraph for something to think about in this context. In particular, take a look at my remarks about epistemological independence near the end of the discussion of the analogy between double-entry bookkeeping and supply and demand. For that matter, my remarks on ring-composition in this piece are worth thinking about as well.

More later.

Henry Farrell on democracy [it's messy]

Sean Carroll interviews Henry Farrell:

Democracy as problem solving:

0:06:11.0 Sean Carroll: ... So let’s start with this idea of democracy and other kinds of institutions in a more or less theoretical sense. Probably the thing that is gonna guide the conversation the most is the paper that you wrote with Cosma Shalizi on cognitive democracy. And I take it that the idea is to think about democracy as a way of making decisions and to compare it with other institutional ways of making decisions. Is that fair?

0:06:48.8 Henry Farrell: That’s right. So we can think about markets, we can think about democracy, we can think about hierarchy, we can think about all of these different modes of problem-solving. And what I mean by problem-solving here is something like the following: So imagine that there are a number of us and we have some shared problem in common that we would like to solve. But either A, it is a complex problem that the solution is not immediately and readily apparent; or B, we have strong disagreements about how to solve it. I think that is where we can start to begin to get some traction on which modes of governance, whether these be markets, whether this be democracy, whether this be hierarchy, are better or worse at solving individual problems.

0:07:30.9 HF: And so I think that the argument that Cosma and I have made there, and I think we would probably modify it some. I think we’re a little bit too enthusiastic about democracy as against its competitors in that piece, but I think that the argument that we make is that the ability of democracy to solve problems is underestimated because people don’t pay attention to the ways of which democracy forces people, ideally, with very different perspectives, very different wants to work together. And sometimes, from the very diversity of our goals and our wants, information emerges and the democratic process works best when it is actually able to harness that process, and to turn that information into something useful and something actionable to help us solve problems.

Democracy is not an optimization problem & how we find our goals:

0:17:18.0 HF: … I was listening to a podcast a couple of weeks ago. This was the Quanta Podcast with Steven Strogatz. And he was interviewing Moon Duchin, who is a mathematician at Tufts. And she said, and I think this is absolutely right, that democracy is not an optimisation problem. And this is, I think, a temptation that a lot of people with engineering or mathematical backgrounds have, is to think in a certain sense, “Well, why is it that people disagree? If we could just come up with some sweet engineering solution, everybody would realise, and we would find some kind of optimal solution”. There are a number of reasons why this is not true.

0:17:58.9 HF: One of those is what you’ve already touched upon, which is that we often figure out what our goals are in the process of actually seeking after them. And this is an argument that political theorists such as John Dewey have made at length. So I think that the best articulation of this is a wonderful, albeit, I think also flawed book that Dewey wrote back in the 1920s called The Public and Its Problems, where he conceives of democracy as effectively a means of trying to figure out these broader problems that we have and to… But in the process of discovery, which both involves ordinary members of the public and experts. Ordinary members of the public understand how the problem affects them in their lives, and experts perhaps understand the more subtle causal chains that mean that…

0:18:47.1 HF: Public has a shared problem. So first of all, a public identifies itself, and then the public tries to create the means through which it can actually address the problem. But the implication of all of this, and Dewey’s a pragmatist, is that this is going to be a never-ending process of discovery where, in trying to solve the problems, the public will, of course, figure out that some of its goals are appropriate and some of its goals widely misconceived the nature of the problem and so, there’s going to be an endless process of revision, and that is something that is crucial and important.

0:19:21.1 HF: The other part of this which is something that Dewey, I think, is bad at, is understanding how it is that people, in a democratic society, have very different goals. We are tossed together in American society, and I think this is part of the issue that we have at the moment with polarisation with people who have very, very different goals, very different understandings of what the shared outcome ought to be, what our goals ought to be, what kinds of values ought to guide those goals. And so democracy then becomes a process of group warfare, to some degree, when it works badly, or group accommodation where you work together and you sometimes figure out messy and painful and agonising solutions, which don’t please ever anybody, but which, nonetheless, sort of allow you to continue on together in relative peace while addressing some of the problems that everybody can agree are problems or whatever.

0:20:15.9 HF: And so this is a really messy process, but it also is not an optimisation process; this is much more like a… So if you think about it in slightly more abstract terms, you’re searching across a rugged solution space where you do not know what the solutions are.

Democracy's dynamic advantage:

0:22:55.3 HF: That if you think about how democracy works versus, for example, systems such as autocracy, so you can think about them as drawing upon possible solutions to problems that a society faces. And here, again, I’m abstracting away all of these difficult and complex problems about, “Do we agree what a solution is? Do we even agree on what a problem is?” etcetera, etcetera. But if you start from that kind of metaphor, then you can reasonably see that some solutions, which are likely to pass muster in a democratic space because they are to the benefit of the collective majority, are probably going to get locked in an autocratic situation or in a democracy where there are extreme power disparities because some of these solutions, even if they are overall beneficial for the society, are going to be uncongenial to the powerful minority the elite was in that society.

0:23:50.0 HF: And so what that suggests then is that under reasonable circumstances, we can say that democracy is likely to have a dynamic advantage, vis-à-vis totalitarian systems, and its ability to search this landscape for possible solutions for problems that pop up. And we think about this primarily in terms of problems of institutional change, finding new rules because there’s a literature there that we want to talk to. And one could also say the more equal democracies, ipso facto, and holding all things equal, are probably going to be better at doing this, at searching for good solutions than less equal democracies precisely because it is less likely that elites are going to be able to block these solutions from being adopted in a more general way.

Henry Farrell on weaponized interdependence & regulating the internet [Tyler Cowen interview]

A day later: I've had a chance to sleep on this conversation. My initial impression remains: one if Tyler's best. Why? I keep looking for hints of new ways of thinking about the world in-the-large, but I rarely find much. This conversation has hints, particularly concerning the relationships between large platform media companies and the nation-states in which they deploy their platforms. 

* * * * *

Tyler Cowen converses with (Crooked Timber's) Henry Farrell on this that and the other, including the internet:

COWEN: Now, we’re chatting in early October, and there were two huge stories today, both about weaponized interdependence. The first is that the European Union seems to be insisting that Facebook take down material posted on internets outside the European Union. What’s your view? Is that a good decision or a bad decision?

FARRELL: Well, when I’m looking at this stuff, I try to wear my hat —

COWEN: But I’m interviewing Henry Farrell, the person. Right?

FARRELL: Okay, let me back up and say two things. First of all, if I’m to think about this as a political scientist under my political science hat, what I would say is that this is something which is inevitable. And this is something which we’re going to see more and more of, which is that the internet for a long period — there was a kind of an equilibrium in which the United States managed to persuade everybody that self-regulation and platforms looking after their own business was more or less okay for most purposes.

That equilibrium has broken down. It has broken down in the US. It has broken down in Europe, and it has broken down in authoritarian societies in particular. So I think we’re going to see more and more efforts to try and use the platform companies effectively as a means of extending reach into other jurisdictions and imposing universal-type restrictions on what they can or cannot do.

And now:

If I want to think about this as an individual, what I would say is that I’m not at all happy with what the European Union is doing in specific here, but I do think that there needs to be much greater regulation of platform companies. And in the absence of the United States, I think that the European Union is the most plausible actor which has got the regulatory clout and the willingness to do so in a manner which is at least somewhat compatible with broad liberal norms.

So my basic attitude will be to push back against a specific decision but to say that if the choice is between the United States not regulating or perhaps regulating — perhaps that will be different under a possible Warren administration — and authoritarian states, that I would much prefer to have a robust European Union than any of the other actors that I can think of that have the clout and the ability to try and bring the platform companies to heel.

The second shoe drops:

COWEN: The second big story from today is from the world of basketball. As you probably know, Houston Rockets general manager Daryl Morey tweets in favor of the Hong Kong protests. It seems both the Rockets and the NBA forced him to delete the tweet and basically retract the statement. Who is in the wrong here? Daryl Morey, the Rockets, the NBA, China, everyone? What do you think?

FARRELL: Again, looking at this from my political science hat, this kind of stuff has been happening for a long, long time, but not so much to the United States. There’s a book by Blackwill and Harris, which came out with the Council on Foreign Relations a couple of years ago, which looks at how China has been imposing this kind of pressure against a variety of smaller countries, especially with respect to the Dalai Lama. And when China says jump, companies do tend to jump.

And of course, we’ve seen Hollywood for the last number of years has been quite sensitive to the Chinese market, most recently as exemplified by the decision to remove, I think it was the Taiwanese flag from the remake of Top Gun from the back of Tom Cruise’s jacket because this was perceived as being possibly provocative. So this is pretty standard stuff, and this tells us something about the way that businesses operate.

You can see the way that businesses operate globally as having both great benefits and great weaknesses. If one wants to think about the standard story about how interdependence, at least under some circumstances, creates peace because businesses have an incentive to try and create peaceful and happy relations between nations in order to secure their commerce, I think that there’s something to it, although as my work with Abe on weaponized interdependence suggests, there also are some clear limits to that as well.

But the obverse of this is that business is obviously in the business of making profits, and when there are clear political risks associated to business doing certain kinds of things, by and large, businesses are not going to be especially courageous. And this is particularly likely to be true of big firms, which I think are more likely to come under this kind of pressure.

One saw this already with regards to Hong Kong. There have been that airways chief executive who effectively got ousted. I’m trying to remember the company, but I can’t.

I think that this is, in a sense, if you want to outsource a lot of our decision-making about culture, about the ways that things ought to be done to business, this is what you’re going to expect, for better or for worse.

The New Interdependence

Henry Farrell and Abraham Newman, Domestic Institutions Beyond the Nation State: Charting the New Interdependence Approach (PDF)

Tim Büthe and Walter Mattli. The New Global Rulers: The Privatization of Regulation in the World Economy. Princeton NJ: Princeton University Press, 2011, 301pp.

Elliot Posner. The Origins of Europe’s New Stock Markets. Cambridge MA: Harvard Univer sity Press, 2009, 240pp.

Kal Raustiala. Does the Constitution Follow the Flag? The Evolution of Territoriality in American Law. New York: Oxford University Press, 2009, 313pp.

Abstract: What is the relationship between domestic and international politics in a world of economic interdependence? This essay discusses and organizes an emerging body of scholarship, which we label the new interdependence a pproach, addressing how transnational interactions shape domestic institutions and global politics in a world of economic interdependence. This literature makes three important contributions.

First, it examine s how domestic institutions affect the ability of political actors to construct the rules and norms governing interdependent relations and thus offer a source of asymmetric power. Second, it explores how interdependence alters domestic political institutions through processes of diffusion, transgovernmental coordination and extraterritorial application and in turn change s the national institutions med iating internal debates on globalization. Third, it studies the shifting boundaries of political contestation through which sub state actors affect decision making in foreign jurisdictions.

Given the importance of institutional change to the new interdependence agenda, we suggest several instances where historical institutionalist tools might be exploited to address these transnational dynamics, in particular mechanisms of cross national sequencing and sub state actor change strategies. As globalization continues, it will be ever more difficult to examine national trajectories of institutional change in isolation from each other. Equally, it will be difficult to understand international institutions without paying attention to the ways in which they both transform and are transformed by domestic institutional politics. While not yet cohering as a single voice, we believe the new interdependence approach offers an innovative agenda that holds tremendous promise for both comparative and International Relations research.

Symposium on Graeber’s Debt

Crooked Timber is running a symposium on David Graeber’s Debt: The First 5000 Years. Contributions so far:

• Seminar on David Graeber’s Debt: The First 5000 Years – Introduction, Chris Bertram
• The unmourned death of the double coincidence, John Quiggan
• The World Economy is not a Tribute System, Henry Farrell
• Debt Jubilee or Global Deleveraging, Barry Finger
• The end of debt, John Quiggan
• The Return of Grand Narrative in the Human Sciences, Neville Morley
• The Dangers of Pricing the Infinite, Malcolm P. Harris, on student debt

All are worth reading, as are many of the comments. I’ll end with the last paragraph from Bertram’s introduction:

Does Graeber find in utopian and democratic resistance to the Axial empires an historic precedent for the Occupy movement to emulate? Perhaps our best possibilities lie not in grand schemes of societal transformation but in developing the “baseline communism” and the democratic instincts that persist even in the heart of modern capitalism. The anarchist writer Colin Ward used a phrase from Ignazio Silone – “the seed beneath the snow” – to make a similar idea vivid. We cannot take the beast on in a direct assault, and nor should we, but we can work together to develop a more human society within the nooks and crannies of the commercial one.

Sounds a bit like a plug for the Transition Movement, which originated in England and has since spread around the world.