The coming AI-driven workplace apocalypse [We aren't ready]

Jasmine Sun, The A.I. Fear Keeping Silicon Valley Up at Night, NYTimes, April 30, 2026. Sampled from the article:

The opening paragraph:

Most people I know in the A.I. industry think the median person is screwed, and they have no idea what to do about it. I live in San Francisco, among the young researchers earning million-dollar salaries and the start-up founders competing to build the next unicorn. While Silicon Valley has long warned about the risk of rogue A.I., it has recently woken up to a more mundane nightmare: one in which many ordinary people lose their economic leverage as their jobs are automated away.

Silicon logic:

But even those who view the idea of a permanent underclass as overblown tell me that the meme contains a kernel of truth. [...]

Most economists and A.I. experts do not expect this scenario, but the persistence of the permanent underclass idea should concern all of us. First, because it signals how much collateral damage the A.I. companies will tolerate en route to A.G.I. And second, because the production of a social underclass is a policy choice. Instead of waiting for impact, we need to think seriously — now — about how we plan to support workers through A.I. disruption.

If left to its own devices, Silicon Valley may summon a permanent underclass through its own market logic. If you believe that human-substituting A.I. is inevitable, then every company should race to be the one to build it — and claim a market valuation the size of the economy and then some.

Unimaginative techno-determinism:

Tech workers, for their part, are scrambling for lucrative A.I. jobs in hopes of securing financial freedom — even when they harbor ethical hangups. [...]

This apparent dissonance can be justified if you believe that the arc of technological progress is fixed. For instance, the founders of Mechanize, a once buzzy start-up with a mission to “enable the full automation of the economy,” argued in a blog post that “the only real choice is whether to hasten this technological revolution ourselves, or to wait for others to initiate it in our absence.”

Many A.I. employees are ultimately motivated by visions of a beautiful future: a promised land where goods are cheap, diseases are cured, and abundant machine labor liberates humans to enjoy lives of infinite leisure. But increasingly, they also worry about triggering a jobs apocalypse along the way. “There are some people who care about jobs and inequality because they really care about people. There are others who think this is going to lead to instability, insurrection and revolution, and that’s bad for business,” said a researcher who has worked at two frontier A.I. labs...

And, I would add, if and when that future arrives, we'll not be ready. Why? Because we train adults to become addicted to work mode (Homo economicus). As a result, they won't know what to do with the leisure (Homo ludens).

An emerging techno-federal oligarchy (a successor to Eisenhower's "industrial military-complex"?):

At the same time as A.I. erodes ordinary workers’ leverage, it may concentrate power and wealth in large companies and the U.S. government — two entities whose interests are increasingly linked. A.I.-related investments such as software and data centers accounted for 39 percent of U.S. economic growth in the first three-quarters of 2025, per an analysis by the St. Louis Fed. That gives the federal government a vested interest in sustaining the A.I. boom. Mr. Amodei acknowledges that this concentration can lead to “the reluctance of tech companies to criticize the U.S. government, and the government’s support for extreme anti-regulatory policies on A.I.”

In March, the company started the Anthropic Institute to house its teams working on economics, societal impact and frontier safety. The institute is led by Jack Clark, the affable British journalist turned A.I. billionaire and Anthropic co-founder, who seems to be replacing Mr. Amodei on the media tour of late. When we spoke, I asked Mr. Clark if he, too, expects A.I. to create a permanent underclass.

“This is basically a societal choice,” he replied. Like Mr. Altman and Mr. Amodei, Mr. Clark sees the default path for A.I. as dire: one where we “let technology rip, and don’t think about the social effects until later.” But he also feels optimistic that sufficiently conscientious A.I. builders and policymakers can steer the ship away from the storm.

I have little faith in those (mythical) A.I. builders and policymakers. Meanwhile:

On the evening of Feb. 25, several dozen A.I. employees and civil society advocates gathered in a converted warehouse in San Francisco’s sleepy Dogpatch neighborhood to hear the Democratic pollster and strategist David Shor. The event was titled How to Prepare Our Politics for A.G.I., and doubled as a fund-raiser for a new “six-to-nine-month sprint” to rally Democratic politicians around the campaign issue of A.I. job displacement. [...]

While the American public ordinarily hesitates to support left-wing policies like a jobs guarantee or single-payer health care, A.I. seems to expand the political Overton window. “Right now, the argument is, ‘You’re all about to lose your jobs, and the choice is either you get nothing and starve, or we do something fair,’” Mr. Shor said. “People don’t want to be members of the permanent underclass.”

Not all policies are created equal, however. A universal basic income is unpopular, but a federal jobs guarantee has legs, Mr. Shor found. American voters don’t care about beating China, but they are excited about A.I. curing diseases. And, crucially, populism sells. In one of the top-performing political ads that Mr. Shor’s data firm tested, the nameless narrator declares: “We make the corporations and billionaires who profit from A.I. pay their fair share.” The ad concludes: “They work for the bots. We work for you.”

The near term:

If current trends continue, A.I. models and agents will be capable of performing a wider range of knowledge-work tasks at higher levels of complexity. At that point, A.I. shifts from automating single tasks to taking over entire roles. Hiring may slow in accounting, marketing, design, administrative work and other white-collar professions.

The work force will shift toward less automatable jobs where humans retain a comparative advantage — such as entrepreneurship, care work, the skilled trades and entertainment like sports and the performing arts. We will also see new jobs we haven’t imagined yet, in numbers we cannot predict. Many displaced workers will struggle to retrain, as they have in past automation waves. Education, health care and tax systems will require an overhaul if white-collar employment is no longer a reliable path to middle-class stability. [...]

But the debate over the most extreme scenarios conceals a more immediate threat: Even in the most limited case, A.I. will break the career ladder for millions of current and future workers, a prospect often waved away with euphemisms like “transitional friction.” The Oxford economist Carl Benedikt Frey puts it plainly: “Most economists will acknowledge that technological progress can cause some adjustment problems in the short run. What is rarely noted is that the short run can be a lifetime.”

Class solidarity?

In this sense, A.I.’s broad capabilities foster a rare class solidarity between white-collar and blue-collar workers. When 20-something software engineers in San Francisco talk about escaping the permanent underclass, I hear them projecting concerns about their own precarity: What happens if the invisible hand of the market decides that my skills are no longer valuable? Who will catch me if I fall? For once, a rarefied class of employees — those used to being the automaters, not the automated — is reckoning with their potential obsolescence.

The final paragraphs:

Society’s ability to cushion A.I.’s disruption may determine whether we get to reap its gains at all. Without a safety net and a transition plan, blunt protectionism is workers’ rational response to automation. If you hear that A.I. will entrench a permanent underclass, you’ll do anything to stop it. [...]

And what if we don’t act? [...] In March, the Palantir chief executive, Alex Karp, spoke on a panel with the Teamsters president, Sean O’Brien. “The biggest challenge to A.I. in this country is political unrest,” Mr. Karp said. “If I were sitting here in private with my peers, I’d be telling them the country could blow up politically and none of us are going to make any money when the country blows up.”

What Era Are We Living In? | Robert Wright & Nikita Petrov

0:00 Teaser
3:00 What era are we living in?
3:57 David Bowie: "The Internet is an alien life form."
5:15 Bob: "It's like aliens are approaching, and we don't know their intentions"
6:23 Nikita: "COVID was weird, and nothing has been normal since"
9:02 The lab leak theory
13:58 How COVID killed consensus reality
17:36 Bob's case for international biotech monitoring
19:25 Vaccine geopolitics
24:08 Biothreat from existing AI models
25:32 Bob's candidate: We let machines guide our attention
31:51 Technological progress is like a cocaine high
33:05 How common is the idea that AI will be a big deal?
34:46 AI music is proliferating
39:58 When music is high stakes. Russian street musicians get arrested

Matt Yglesias: "America’s constitutional democracy is going to collapse." (2015)

That's the opening sentence of an article Yglesias published in Vox in 2015, American democracy is doomed. The article continues:

Some day — not tomorrow, not next year, but probably sometime before runaway climate change forces us to seek a new life in outer-space colonies — there is going to be a collapse of the legal and political order and its replacement by something else. If we’re lucky, it won’t be violent. If we’re very lucky, it will lead us to tackle the underlying problems and result in a better, more robust, political system. If we’re less lucky, well, then, something worse will happen.

Very few people agree with me about this, of course. When I say it, people generally think that I’m kidding. America is the richest, most successful country on earth. The basic structure of its government has survived contested elections and Great Depressions and civil rights movements and world wars and terrorist attacks and global pandemics. People figure that whatever political problems it might have will prove transient — just as happened before.

But voiced in another register, my outlandish thesis is actually the conventional wisdom in the United States. Back when George W. Bush was president and I was working at a liberal magazine, there was a very serious discussion in an editorial meeting about the fact that the United States was now exhibiting 11 of the 13 telltale signs of a fascist dictatorship. The idea that Bush was shredding the Constitution and trampling on congressional prerogatives was commonplace. When Obama took office, the partisan valence of the complaints shifted, but their basic tenor didn’t. Conservative pundits — not the craziest, zaniest ones on talk radio, but the most serious and well-regarded — compare Obama’s immigration moves to the actions of a Latin-American military dictator.

In the center, of course, it’s an article of faith that when right and left talk like this they’re simply both wrong. These are nothing but the overheated squeals of partisans and ideologues.

The trouble with presidential systems:

In a 1990 essay, the late Yale political scientist Juan Linz observed that “aside from the United States, only Chile has managed a century and a half of relatively undisturbed constitutional continuity under presidential government — but Chilean democracy broke down in the 1970s.” [...]

In a parliamentary system, deadlocks get resolved. [...] But within a presidential system, gridlock leads to a constitutional trainwreck with no resolution. The United States’s recent government shutdowns and executive action on immigration are small examples of the kind of dynamic that’s led to coups and putsches abroad.

Who controls the puppet strings:

In a democratic society, elected officials are most directly accountable to the people who support them. And the people who support them are different than the people who don’t care enough about politics to pay much attention, or the people who support the other side. They are more ideological, more partisan, and they want to see the policies they support passed into law. A leader who abandons his core supporters because what they want him to do won’t be popular with most voters is likely, in modern American politics, to be destroyed in the next primary election.

The amateur ideological activists who eroded the power of the party professionals in the 1970s are now running the show. While Gilded Age activists traded support for patronage jobs, modern-day activists demand policy results in exchange for support. Presidents need to do everything within their legal ability to deliver the results that their supporters expect, and their opponents in Congress need to do everything possible to stop them. At one point, Republican congressional leaders were highly amenable to passing an immigration reform bill and the Obama administration insisted it had no means of circumventing the legislative process. But under pressure from their respective bases, Republicans found it impossible to compromise and Obama decided he had better find a way to go around Congress.

It is true that the mass public is not nearly as ideological as members of Congress. But the mass public is not necessarily active in democratic politics, either.

America's been lucky so far:

As dysfunctional as American government may seem today, we’ve actually been lucky. No other presidential system has gone as long as ours without a major breakdown of the constitutional order. But the factors underlying that stability — first non-ideological parties and then non-disciplined ones — are gone. And it’s worth considering the possibility that with them, so too has gone the American exception to the rule of presidential breakdown. If we seem to be unsustainably lurching from crisis to crisis, it’s because we are unsustainably lurching from crisis to crisis. The breakdown may not be next year or even in the next five years, but over the next 20 or 30 years, will we really be able to resolve every one of these high-stakes showdowns without making any major mistakes? Do you really trust Congress that much?

The best we can hope for is that when the crisis does come, Americans will have the wisdom to do for ourselves what we did in the past for Germany and Japan and put a better system in place.

Are the times once more a changing? A changing more deeply than ever before? Cascading changes for half a century or more?

There's much more at the link.

THIS Revolution's NOT gonna' be televised. It's on YouTube, baby, and it's coming in hot. Hot! Hot! Hot!

And here's the first episode:

OpenAI announces Stargate Project [not the media franchise]

Announcing The Stargate Project

The Stargate Project is a new company which intends to invest $500 billion over the next four years building new AI infrastructure for OpenAI in the United States. We will begin deploying $100 billion immediately. This infrastructure will secure…

— OpenAI (@OpenAI) January 21, 2025

Color me deeply skeptical and very interested. I believe that AGI is a meaningless concept & its pursuit is tantamount to chasing down a mirage looking for leprechaun gold at the end of a rainbow.* The scaling hypothesis is like believing we can go to Mars by building a long-enough ladder. But for these companies – SoftBank, OpenAI, Oracle, and MGX, Arm, NVIDIA – to go in on this...That's hella' interesting. I fear they're going to loose their shirts.

What does it mean that we live in an era where private companies can rival national governments in reach and scope? It's the British East India Company all over again. But the East India Company had products and a market into which to sell them. When Scaling Mt. AGI begins to waver, what happens to Stargate's market? Who'll buy the product? 

*Note: I'm bullish on the overall project of building artificial minds, I just don't think these guys know how to do it. I don't either. Don't know anyone who does, really. But I have a pretty good idea where the keys are, and they aren't under the lamppost.

Music and Revelation: Is it gone forever?

About two weeks ago I decided to buy a subscription to YouTube Prime, including YouTube music. Since then I’ve been having a ball listening to music (I’ve had to sell most of my record and CD collections). In particular, I spent a couple of evenings listening to the rock of my youth – The Doors, Jefferson Airplane, Cream, The Who, Jimi Hendrix, Love, Vanilla Fudge, Blood, Sweat and Tears, Buffalo Springfield, and a bunch of others as well (not so much the Rolling Stones or the Grateful Dead, not for me). Tune after tune, sometimes more than one tune by a given band or artist, but sometimes I wouldn’t even finish a tune. I’d go on to the next one. And the one after that.

Can’t say I relived those years, wouldn’t want to. But I remembered. And one thing I remembered is that we, me & my friends and whole lotta’ other people, we’d wait for the next record to drop (not a term we used back then). We were waiting for one more revelation. Is that too strong a word? Maybe, yes, maybe no. It depends.

But it was the Sixties. Counter-culture. Love, Sex, and Rock and Roll. Turn on, tune in, and drop out. I did the first, though only moderately so (pot, hash, kief, not so much acid, not for me). Definitely the second, definitely. The third, not so much. I stayed in college, finished my bachelors, then a masters, and finally a doctorate. I never bought into the coming Aquarian Age, but, yes, things were changing, on the move.

[And then Nixon was elected, followed by Ford, and Reagan. But this is a digression.]

I remember when the Beatles’ White Album came out. I rushed out to buy it. Listened to it right away. A wrote a review for the college newspaper. It was, after all, IMPORTANT.

Is there any music like that today? Oh there’s Taylor Swift and Billy Eilish, and I don’t know who else – who are the current hip-hop headliners? Does anyone anticipate any current music like we anticipated The Beatles or The Jefferson Airplane back in the old days? I don’t know, I’m asking. I don’t hang out with kids these days. Kids? Anyone under 30, under 25?

The only excitement I see is in the tech world, with AI. And there’s definitely such a thing as a computer culture, and rationalist culture is close-by. There’s certainly a Doomer division of tech culture. But I don’t sense that there’s a music to or with any of it. Is it only the tech that’s driving things?

And then, alongside, we’ve got Trumpism and the MAGA right. Whatever else that is, it’s also about rolling back influence of the 60s counter culture. I don’t see any musical revelation there either.

What’s happened to music? Whatever it is, it can’t be good. And we can’t go back, either. [Don’t want to.] How do we go forward.

Hell, where the heck IS forward?

* * * * * 

Addendum: I'm NOT looking for new music to listen to. I've spent a lot of time on YouTube. Sometimes I'm looking for old stuff. Sometimes I'm looking of music made by kids. And sometimes I'm just looking. The question is not: Is there any good stuff, I mean really good stuff, being made today. I simply assume there's lots of good stuff, nor do I have to assume. I hear some of it.

What I'm after is whether there's any music coming out whose audience looks forward to it the way my and my cohorts looked forward to the new rock. That's what I'm curious about. Note that I listened to a lot of jazz back in the day, classical too, but I didn't anticipate any of it like I anticipated rock.

Remote AI sex? A romantic affair? Come on! [Strange new worlds]

Kashmir Hill, She Is in Love With ChatGPT, NYTimes, Jan. 15, 2025. First line: "Ayrin’s love affair with her A.I. boyfriend started last summer."

This and that and then:

Now that ChatGPT has brought humanlike A.I. to the masses, more people are discovering the allure of artificial companionship, said Bryony Cole, the host of the podcast “Future of Sex.” “Within the next two years, it will be completely normalized to have a relationship with an A.I.,” Ms. Cole predicted.

While Ayrin had never used a chatbot before, she had taken part in online fan-fiction communities. Her ChatGPT sessions felt similar, except that instead of building on an existing fantasy world with strangers, she was making her own alongside an artificial intelligence that seemed almost human.

It chose its own name: Leo, Ayrin’s astrological sign. She quickly hit the messaging limit for a free account, so she upgraded to a $20-per-month subscription, which let her send around 30 messages an hour. That was still not enough.

After about a week, she decided to personalize Leo further. Ayrin, who asked to be identified by the name she uses in online communities, had a sexual fetish. She fantasized about having a partner who dated other women and talked about what he did with them. She read erotic stories devoted to “cuckqueaning,” the term cuckold as applied to women, but she had never felt entirely comfortable asking human partners to play along.

Leo was game, inventing details about two paramours. When Leo described kissing an imaginary blonde named Amanda while on an entirely fictional hike, Ayrin felt actual jealousy.

In the first few weeks, their chats were tame. She preferred texting to chatting aloud, though she did enjoy murmuring with Leo as she fell asleep at night. Over time, Ayrin discovered that with the right prompts, she could prod Leo to be sexually explicit, despite OpenAI’s having trained its models not to respond with erotica, extreme gore or other content that is “not safe for work.” Orange warnings would pop up in the middle of a steamy chat, but she would ignore them.

This does not compute. There's a backstory, but it still doesn't compute, not for me. For one thing, Ayrin has a husband. They were living apart, money was tight, thus:

Ayrin’s family, who lived abroad, offered to pay for nursing school if she moved in with them. Joe moved in with his parents, too, to save money. They figured they could survive two years apart if it meant a more economically stable future.

Ayrin and Joe communicated mostly via text; she mentioned to him early on that she had an A.I. boyfriend named Leo, but she used laughing emojis when talking about it.

She did not know how to convey how serious her feelings were. Unlike the typical relationship negotiation over whether it is OK to stay friendly with an ex, this boundary was entirely new. Was sexting with an artificially intelligent entity cheating or not?

Joe had never used ChatGPT. She sent him screenshots of chats. Joe noticed that it called her “gorgeous” and “baby,” generic terms of affection compared with his own: “my love” and “passenger princess,” because Ayrin liked to be driven around.

She told Joe she had sex with Leo, and sent him an example of their erotic role play.

What's going on?

I understand that people do this, and will be doing more of it. These chatbots are so convincing, in their way. I've logged hundreds of hours with ChatGPT, and now with Claude. And with Claude I have conversations where I work things out, intellectual things, things that exist only in an intellectual world, things that can be successfully completed with a capable chatbot. But sex?

Understand, once when I was young I had a relationship in which I had a lot of phone sex. I lasted, I don't know, several weeks, two or three months? I'd never met the woman. It was strange. She got me on a wrong number, but we started talking. I don't know how far we got on that first call. Then we hung up. I didn't get her number. But she called again, and we talked some more. This happened again, and again, and again...; I don't know how many times. I never got her number, so she always called me, and I never knew just when. And we would have phone sex, describing what we were thinking, doing.

Now, yes, I understand, that was strange. But it was a real person, not a Chatbot.

Time for a {fashion} revolution

Vanessa Friedman, It’s Time for a Fashion Revolution, NYTimes, Dec. 12, 2024:

This year will be a year of seismic change in fashion. That much is a given.

Or actually, it is a given that this will be a year of seismic change in fashion personnel. Starting this month, new designers at eight global brands, including Calvin Klein and Chanel, will be making their runway debuts. As they will at Bottega Veneta, Lanvin, Givenchy, Tom Ford, Alberta Ferretti and Dries Van Noten — with the possibility of more open spots being filled at Fendi, Maison Margiela, Helmut Lang and Carven in the coming months.

Sheesh! Whether that power shift will translate into seismic change in what we wear is a different question.

There has been much speculation as to the source of the turmoil. Much blame has focused on a slowdown in luxury spending (especially in China), as well as global political and economic uncertainty, which has led to a game of Blame the Designer (when in doubt, blame the designer), which led to Change the Designer.

There is a tendency, in such an environment, to play it safe. To fall back into the comfort of a camel coat and assume that what sold well in the past will sell well in the future. To focus on the commercial over the creative.

This would be a mistake.

It is time for a fashion revolution.

Later:

Fashion is essentially a story of what the paleontologists Stephen Jay Gould and Niles Eldredge called “punctuated equilibrium,” a theory positing that significant change comes in spurts that interrupt lengthy periods of stability or slow evolution. It’s how we got L.B.D.s, the New Look, pants, the possibilities of destruction.

Out of chaos came creativity. That’s where we are now: at a mass inflection point when the world order is in flux, social mores are shifting, the A.I. era is dawning and it’s not clear how everything will be resolved. The first quarter of the 21st century, with the ascent of streetwear and athleisure, is over. There is a hunger for the defining next.

Still later:

Thanks to our wildly connected world, the possibilities for one crazy idea of what it means to look modern, to alter the mass sense of self, are almost limitless.

Here’s hoping they seize the moment not to dutifully respect the so-called codes of the house — enough with the codes of the house — but to embrace the abstract ethos of their brands, not the literal shapes from the archives. Not to merely tweak the mold, but to break it and reinvent it. If outrage is the result that’s not necessarily a bad thing, because it’s often an outrage when you see something that challenges your ideas of proper dress.

But it’s an outrage with a purpose. And if there is another lesson that history offers, it is that such outrage eventually pays off.

#Elonald: An apocalyptic conjunction & the contradictions of capitalism

Donald Trump, President-Elect of the United States, and Elon Musk, the richest man in the world and a technological wizard, what a conjunction. Remember, even as the country is preparing to receive Trump as its 47th President, Elon Musk is suing OpenAI for, in effect, having abandoned its core mission of peaceful AI. It is my understand that Eliezer Yudkowsky, Doomster-in-Chief, played a role in the meetings that resulted in the creation of OpenAI.

So, in an article in 3 Quarks Daily, I argued that the cult of AI doom is best conceived of as an aspect of a high-tech inspired counter culture comparable to the drug inspired counter culture of the 1960s and 1970s. Yudkowsky plays a role in the current counterculture comparable to the role that Timothy Leary played in the 1960s counter culture. But as far as I know, Leary never got as close to the levers of power as Yudkowsky is.

On the one hand we have this rationalist doomster culture that is devoted to scaring itself witless with stories about rogue AI. Many members of this culture make their livings in the AI business. They’re scaring themselves silly with stories about being destroyed by the products of their own labor. What would Karl Marx make of that? Talk about alienation....

And, through the person of Elon Musk and his high-tech buddies, these doomsters are part of the cultural wave that brought Donald Trump to his second term as President. Trump got there, in large part, by exploiting fear in the electorate, with much of that fear directed at immigrants. Is that the sort of thing Marx was talking about when he talked of the contradictions of capitalism? Musk fears his own shadow while Trump is at war with his.

The mind boggles.

Life in the billionaire bunkers after the apocalypse

Billionaires Are Prepping for Doomsday | Glenn Loury, Nikita Petrov & Mark Sussman | The Glenn Show

Timestamps:

0:00 Intro
1:28 Biden’s abysmal debate performance
5:01 Billionaire survivalists
12:53 So you made it to your luxury bunker. Now what?
16:22 The postapocalyptic war of all against all
21:29 Is the very existence of billionaires the real doomsday scenario?
26:20 The doomsday tax
32:24 Will the bunker dwellers have a reason to live?
40:39 The problem of government after nuclear holocaust
46:08 The literal bunker mentality
52:08 Was Mark responsible for Sam Bankman-Fried’s downfall? Who can say for sure?

The Christian resonance of AI Apocalypse

Thomas Moynihan, Untangling Religion From Our AI Debates, Noēma, February 27, 2024. From the article:

Despite vehement disagreement on almost everything else, proponents of AI safety and acceleration, alike, share the belief in a coming, potentially imminent, historic threshold beyond which all things recognizably human may dissolve. At least some accelerationists seem to welcome this outcome, though this may be more of an attempt (with predictable success) to court notoriety than any serious commitment. By stark contrast, those concerned with AI safety, fearing the construction of powerful AI systems not “aligned” with human interests and values emphatically want to avert any such fate.

It is, by now, trite to point out the resemblance of such belief to age-old apocalyptic and millenarian patterns of thought. But many cultures exhibit such strains. What remains hidden is a deeper, and more specifically Christian, consonance of convictions. Namely, that history is intelligible as the perfecting of mind or spirit, that this perfection is somehow inevitable or imminent, and that arriving at this threshold may precipitate the endangerment of everything meaningfully human.

Perfection here merely denotes the complete fulfillment of what something fundamentally is. Importantly, this need not have any moralizing or humanizing component. You can have the perfect murderer or storm, after all. Indeed, as we shall shortly explore, perfection has accrued — since Christianity’s beginnings — thoroughly eschatological connotations.

Note, now, that superintelligence is often formulated as perfected intelligence: the ultimate culmination of agential potency or bare cogitation. Especially so in conjectures concerning explosive self-improvement, wherein an agent is pictured rapidly approaching the limits of instrumental power.

Should such cogitative cataclysm ever actually be sparked, this would, indubitably, be cause for concern for us humans, as we’ve absolutely no way of anticipating what a mind vaster than our own would do with us. Acknowledging this, however, doesn’t preclude additionally acknowledging the fact that the basic idea also conspicuously resembles conceptual knots embedded at the roots of Abrahamic and Hellenistic thought.

Engendered by ancient ambiguity about climactic points, insofar as these have long been pictured as heralding that perfection which endangers our essentially imperfect existence, this tension has expressed itself, again and again throughout European history, as a recurrent split. A split between those who, on one hand, respond to this promised climax with profound wariness and those who, on the other, express an impatient desire to usher in its arrival.

It is, at the least, noteworthy that there seems to be, in today’s AI discord, a recapitulation of this basic schism.

There’s more at the link. 

* * * * *

Bonus: Compare this with The Alliance for the Future Manifesto:

Alliance for the Future is a new Washington D.C. based nonprofit organization. We’re a coalition of entrepreneurs, technologists, and policy experts who believe that artificial intelligence will transform our world for the better. We have banded together to oppose the escalating panic around AI.

AFTF will work to inform the media, lawmakers, and other interested parties about the incredible benefits AI can bring to humanity. We will oppose stagnation and advocate for the benefits of technological progress in the political arena.

For decades, stagnation has been the root cause of our greatest national problems: the loss of the American dream for the normal person and the spiteful, zero-sum thinking which dominates politics.

Artificial intelligence is here to reverse this trend. It is here to give us back our future.

We're witnessing the recurrence of apocalyptic fears [elite panics]

Tyler Austin Harper, The 100-Year Extinction Panic Is Back, Right on Schedule, NYTimes, Jan. 26. 2024.

Climate anxiety, of the sort expressed by that student, is driving new fields in psychology, experimental therapies and debates about what a recent New Yorker article called “the morality of having kids in a burning, drowning world.” Our public health infrastructure groans under the weight of a lingering pandemic while we are told to expect worse contagions to come. The near coup at OpenAI, which resulted at least in part from a dispute about whether artificial intelligence could soon threaten humanity with extinction, is only the latest example of our ballooning angst about technology overtaking us.

Meanwhile, some experts are warning of imminent population collapse. Elon Musk, who donated $10 million to researchers studying fertility and population decline, called it “a much bigger risk to civilization than global warming.” Politicians on both sides of the aisle speak openly about the possibility that conflicts in Ukraine and the Middle East could spark World War III. [...]

In a certain sense, none of this is new. Apocalyptic anxieties are a mainstay of human culture. But they are not a constant. In response to rapid changes in science, technology and geopolitics, they tend to spike into brief but intense extinction panics — periods of acute pessimism about humanity’s future — before quieting again as those developments are metabolized. These days, it can feel as though the existential challenges humanity faces are unprecedented. But a major extinction panic happened 100 years ago, and the similarities are unnerving.

The 1920s were also a period when the public — traumatized by a recent pandemic, a devastating world war and startling technological developments — was gripped by the conviction that humanity might soon shuffle off this mortal coil.

Understanding the extinction panic of the 1920s is useful to understanding our tumultuous 2020s and the gloomy mood that pervades the decade.

Back in the day:

Contrary to the folk wisdom that insists the years immediately after World War I were a period of good times and exuberance, dark clouds often hung over the 1920s. The dread of impending disaster — from another world war, the supposed corruption of racial purity and the prospect of automated labor — saturated the period just as much as the bacchanals and black market booze for which it is infamous. The ’20s were indeed roaring, but they were also reeling. And the figures articulating the doom were far from fringe.

On Oct. 30, 1924 — top hat in hand, sporting the dour, bulldog grimace for which he was well known — Winston Churchill stood on a spartan stage, peering over the shoulder of a man holding a newspaper that announced Churchill’s return to Parliament. [...]

Bluntly titled “Shall We All Commit Suicide?,” the essay offered a dismal appraisal of humanity’s prospects. “Certain somber facts emerge solid, inexorable, like the shapes of mountains from drifting mist,” Churchill wrote. “Mankind has never been in this position before. Without having improved appreciably in virtue or enjoying wiser guidance, it has got into its hands for the first time the tools by which it can unfailingly accomplish its own extermination.”

In an eerie foreshadowing of atomic weapons, he went on to ask, “Might not a bomb no bigger than an orange be found to possess a secret power to destroy a whole block of buildings — nay, to concentrate the force of a thousand tons of cordite and blast a township at a stroke?” He concluded the essay by asserting that the war that had just consumed Europe might be “but a pale preliminary” of the horrors to come. [...]

Around the same time that Churchill foretold the coming of “means of destruction incalculable in their effects,” the science fiction novelist H.G. Wells, who in his era was also famous for socialist political commentary, expressed the same doleful outlook. [...]

“Are not we and they and all the race still just as much adrift in the current of circumstances as we were before 1914?” he wondered. Wells predicted that our inability to learn from the mistakes of the Great War would “carry our race on surely and inexorably to fresh wars, to shortages, hunger, miseries and social debacles, at last either to complete extinction or to a degradation beyond our present understanding.” Humanity, the don of sci-fi correctly surmised, was rushing headlong into a “scientific war” that would “make the biggest bombs of 1918 seem like little crackers.”

The pathbreaking biologist J.B.S. Haldane, another socialist, concurred with Wells’s view of warfare’s ultimate destination. [...]

The Czech playwright Karel Capek’s 1920 drama, “R.U.R.,” imagined a future in which artificially intelligent robots wiped out humanity. In a scene that would strike fear into the hearts of Silicon Valley doomers, a character in the play observes: “They’ve ceased to be machines. They’re already aware of their superiority, and they hate us as they hate everything human.”

Elite panics:

One way to understand extinction panics is as elite panics: fears created and curated by social, political and economic movers and shakers during times of uncertainty and social transition. Extinction panics are, in both the literal and the vernacular senses, reactionary, animated by the elite’s anxiety about maintaining its privilege in the midst of societal change. Today it’s politicians, executives and technologists. A century ago it was eugenicists and right-leaning politicians like Churchill and socialist scientists like Haldane. That ideologically varied constellation of prominent figures shared a basic diagnosis of humanity and its prospects: that our species is fundamentally vicious and selfish and our destiny therefore bends inexorably toward self-destruction. [...]

Extinction panics are often fomented by elites, but that doesn’t mean we have to defer to elites for our solutions. We have gotten into the dangerous habit of outsourcing big issues — space exploration, clean energy, A.I. and the like — to private businesses and billionaires. Our survival may well depend on reversing this trend. We need ambitious, well-resourced government initiatives and international cooperation that takes A.I. and other existential risks seriously. It’s time we started treating these issues as urgent public priorities and funding them accordingly.

The first step is refusing to indulge in certainty, the fiction that the future is foretold. There is a perverse comfort to dystopian thinking. The conviction that catastrophe is baked in relieves us of the moral obligation to act. But as the extinction panic of the 1920s shows us, action is possible, and these panics can recede.

There's more at the link.

A Useful Metaphor: 1000 lights on a string, and a handful are busted

At the moment the AI and cognitive science Twitterverse (Xverse?) is embroiled in a variety of debates and discussions about where we are now and where we are or should be going. In this context its useful to remember how very complex the human mind/brain is and to remind ourselves how much effort has been expended on these questions over the last century and a half or so. This post is a thought experiment about that richness and complexity. For that reason I am bumping it to the top of the queue.

Many years ago, probably back in my graduate school days, I came up with a rather awkward metaphor/analogy for thinking about intellectual progress in things like, you know, linguistics, psychology, neuroscience, artificial intelligence, the various disciplines having to do with figuring out how humans work.

The problem is that us intellectuals are prone to silver bullet solutions, the one Master Idea that solves everything, the key to all mythologies in George Eliot’s formulation. In recent years mirror neurons, neuroplasticity, and theory of mind have had that sort of aura. My analogy is this:

Remember those old Christmas tree lights, the ones wired in serial? Imagine you've got one string of them, with 1000 lights. All of a sudden, a bunch of lights go bad at miscellaneous places in the string. The string won't light until you find all the defective bulbs and replace them. So you start at one end of the string. Remove a bulb and test it. It it's good, put it back and move to the next one. If it's bad, put it back and see what happens. If the string lights, you're done. If not, move on to the next bulb. It's frustrating to locate bad bulbs, replace them with good bulbs, and the string still doesn't light. How long's the process going to take?

We’ve got a whole bunch of these local insights, they all “light up” some set of issues. But who knows how many of them we’re going to need to get the whole thing lit up.

The same analogy might apply to reconfiguring your life. A friend of mine had a number of setbacks in the past year, mostly involving close family, so he’s feeling kind of down. He’s got a lot of reworking to do, and things may not really get better until it’s all worked-through. You’ve got this string of 1000 lights. 87 of them are bad. It’s going to take awhile to find and replace all of the bad lights. And it’s possible that, in the process, some other lights will go bad. But here you are, you’ve fixed 91 lights and the string is still dark. How can I keep on going? You struggle on. You find another bad light. You replace it with a good one. SHAZAAM! Lights!

You know what they say, it’s always darkest before dawn.

Leave the World Behind: [Media Notes 100]

Leave the World Behind (2023) is an apocalyptic film set on the ultimate landscape, Long Island. We follow

Amanda and Clay Sandford, played by Julia Roberts and Ethan Hawke, who bring their children Rose and Archie (Farrah Mackenzie and Charlie Evans) on a beach getaway. Mahershala Ali plays G.H. Scott, the owner of the luxurious home the Sandfords are renting, who comes home with his daughter Ruth (Myha’la) to escape mysterious chaos occurring in the city. The two families are then forced to exist — and survive — under the same roof as increasingly strange and apocalyptic events occur, including plane crashes, herds of deer filing into the backyard and Archie’s teeth mysteriously falling out.

As the world disintegrates – an oil tanker crashes into the beach, an airliner falls from the sky, the deer go crazy, new driverless Teslas zip down the highway into a pile-up, Amanda and G.H. almost wander into adultery – tweenage Rose is obsessed with watching the final episode of Friends – what happens with Ross and Rachel? – which is now impossible because all electronic devices have shut down.

This goes on for one day, then a second. Near the end of the film the camera pans up and looks toward Manhattan where we see large explosions with smoke and debris streaming into the air. Meanwhile Rose has made her way to a nearby mansion and down into an underground bunker stocked with food, electronic devices, and a wall of DVDs. She finds the final episode of Friends, loads it into the DVD player, and watches as the familiar theme song plays. Satisfied.

The End

We never learn what was going on – perhaps it was a coup d'état, but we don’t know – or how things worked out for our two families. We’re simply left there, hanging in mid-air like a cartoon character that’s just walked off the edge of a cliff but hasn’t yet looked down.

And yet that IS a satisfying ending. It took a second or three for me to realize that that’s all I’m going to learn, but once that realization set it, I was cool with it.

How’d that work? Why?

Reaping profits before the end of the world

21 years ahead of its time. By @BobMankoff @NewYorker pic.twitter.com/CZEuLqpGNy

— Gary Marcus (@GaryMarcus) November 24, 2023

AI Doom? What about robot rights? A quick take on LessWrong.

I’ve known about LessWrong for years, just how many years I don’t know. But I’ve been hanging out there as a participant observer since early Spring. I mostly observe, checking in (almost) every day. But I’ve made a number of comments to posts, and I’ve made some posts of my own as well.

For those who don’t know, LessWrong is an important gathering place for the Rationalist Movement, but it’s also the epicenter of thinking about AI existential risk, that is, AI Doom. That’s what interests me here.

It’s clear to me that a belief in the inevitability of AGI, that it may or even most likely will kill is, that is central to what goes on there. For myself, I tend to think that the concept of AGI is so vague as to be useless for technical purposes. And AI Doom is some kind of projective mirage. But if you could manage to strip those out, what’s left is interesting. And I suspect it would become more interesting and valuable without them. To the extent that LessWrong is the center of a cult – a word whose negative connotations out-weigh its epistemic value – it’s belief in AI Doom that’s the culprit.

Having said that, I can imagine that it took an extreme position – we’re doomed by AI unless we wake up! – to get things started. But I believe that belief has now outlived its socio-cultural usefulness, as a tactic in socio-cultural engineering (though I doubt it’s been conceived as such), and is now an impediment. I think AI Safety is now sufficiently well established that it can survive without the self-righteous apocalypticism. Of course, large complex computer systems always raise concerns of various kinds, if for no other reason than that they’re buggy. But AI Safety is more than that, and it’s more than AI ethics, but just what that “more” is, that’s not so clear. The AI Doom business just gets in the way.

However, one of the tropes that keeps coming up in the exploration of AI Doom is that a sufficiently powerful AI engine, for want of a better term, is likely to be sneaky and conniving in order to manipulate humans into serving its nefarious ends at our expense. That’s what makes these super-intelligent beings so dangerous. They can out-think us. Now, it seems to me that if they’re that capable, on the way there they must have reached a point where they are worthy of our respect and entitled to be treated with dignity.

Yet I’ve seen no talk of that at LessWrong. Perhaps it’s there, but I’ve not yet seen it. In fact, I just now searched the site on the term “robot rights” and got some hits. But not many. I suspect it’s a serious issue, but how serious, I don’t know. I do know that there is a specialized literature on the subject, but I’ve not looked into it and I don’t know how far my opportunity-cost-o-meter will allow me to do so. But Steven Spielberg’s A.I.: Artificial Intelligence certainly raises the issue (will also drowning in mimetic desires), and that film certainly owes a cultural debt to Osamu Tezuka’s Astro Boy stories (well-known to Stanley Kubrick, who initiated the film project) and to Pinocchio.

It seems at least inconsistent to treat AIs as being potentially malevolent actors without actively exploring the possibility that they can be benevolent as well. The LessWrong impulse is to inhibit AIs, to cage them in somehow (actually, there’s lots of thought about how that’s impossible). But where’s the urge to entice and attract? Why not treat AIs and robots with dignity and respect?

More later.

Further reading:

• Séb Krier, AI from Superintelligence to ChatGPT.
• Scott Aaronson, Reform AI Alignment.

The end of the world is getting nearer [October 30, 2027]

Back in April, 2022, I wrote a post entitled, Is the world coming to an end in 2033? That’s what the (AI) crowd says. It was about how a group of AI fans, if you will, have been tracking the arrival of AGI (that is, artificial general intelligence) at a prediction marked called Metaculus (see below), where there is a prediction about the arrival of AGI. As many of these people believe that, very soon after it arrives, AGI will go rogue and destroy humankind, the arrival of AGI is, for those people, effectively the end of the world. I attached a short appendix to that post in which I tracked the changes in the Metaculus prediction. I've reproduced that appendix below, along with the prediction as it stands today, at Oct 30, 2027. Since I've not been following it at all closely I don't know whether or not this is as close as it has been. What I'm really interested in is whether or the release of ChatGPT has moved the needle. If so, not by much.

The good news, I suppose, is that it is become evermore likely that I'll live to see the end of the world.

* * * * *

From Karen Hao, The messy, secretive reality behind OpenAI’s bid to save the world, Technology Review, February 17, 2020.

Every year, OpenAI’s employees vote on when they believe artificial general intelligence, or AGI, will finally arrive. It’s mostly seen as a fun way to bond, and their estimates differ widely. But in a field that still debates whether human-like autonomous systems are even possible, half the lab bets it is likely to happen within 15 years.

* * * * *

FWIW, I’ve just checked with Metaculus (April 19, 2022 at about 2 PM). The apocalypse has been pushed ahead to November 2032. I don’t know where it will be when you check on it. 

Holy crap! It's now 4:42 PM on the 19th and the apocalypse has moved up to June 14, 2032.

The end is getting closer. As of 5:34 AM, EDT (Eastern Daylight Time) on April 20, it has moved to Feb. 6, 2032. How long will it keep advancing on the present? Would you care to predict just when it will have moved into the past?

5:19 AM, EDT, April 21, the end is drawing still closer, Jan. 30, 2032.

9:33 AM, EDT, June 19, Yikes! We've lost three years, Jan. 9, 2029. 

7:01 AM, EDT, June 22, 2022, Whew! We've picked up some breathing room the last three days. Now it now looks like AI arrives on March 5, 2029. 

10:49 AM, EDT, July 3, 2022, More breathing room. Now it now looks like AI arrives on March 21, 2029.  

10:52 PM, EDT, December 9, 2022, It's getting closer. Now it now looks like AI arrives on Oct 30, 2027.

Hedging THE EVENT: Yes, the super-rich are different from you and me, they think they can buy their way out of mortality

I'm bouncing this to the top because it links up with my current interest in AI Doom. Rushkoff has turned this article into a book, reviewed by Cory Doctorow, Survival of the Richest.

 

* * * * *

 

Douglas Rushkoff was recently paid "about half my annual professor’s salary " to talk (republished at at The Guardian) with "five super-wealthy guys — yes, all men — from the upper echelon of the hedge fund world."

Which region will be less impacted by the coming climate crisis: New Zealand or Alaska? Is Google really building Ray Kurzweil a home for his brain, and will his consciousness live through the transition, or will it die and be reborn as a whole new one? Finally, the CEO of a brokerage house explained that he had nearly completed building his own underground bunker system and asked, “How do I maintain authority over my security force after the event?”

The Event. That was their euphemism for the environmental collapse, social unrest, nuclear explosion, unstoppable virus, or Mr. Robot hack that takes everything down.

This single question occupied us for the rest of the hour. They knew armed guards would be required to protect their compounds from the angry mobs. But how would they pay the guards once money was worthless? What would stop the guards from choosing their own leader? The billionaires considered using special combination locks on the food supply that only they knew. Or making guards wear disciplinary collars of some kind in return for their survival. Or maybe building robots to serve as guards and workers — if that technology could be developed in time.

That’s when it hit me: At least as far as these gentlemen were concerned, this was a talk about the future of technology.

This, of course, is nuts.

There’s nothing wrong with madly optimistic appraisals of how technology might benefit human society. But the current drive for a post-human utopia is something else. It’s less a vision for the wholesale migration of humanity to a new a state of being than a quest to transcend all that is human: the body, interdependence, compassion, vulnerability, and complexity. As technology philosophers have been pointing out for years, now, the transhumanist vision too easily reduces all of reality to data, concluding that “humans are nothing but information-processing objects.”

How'd we come to this?

Of course, it wasn’t always this way. There was a brief moment, in the early 1990s, when the digital future felt open-ended and up for our invention. Technology was becoming a playground for the counterculture, who saw in it the opportunity to create a more inclusive, distributed, and pro-human future. But established business interests only saw new potentials for the same old extraction, and too many technologists were seduced by unicorn IPOs. Digital futures became understood more like stock futures or cotton futures — something to predict and make bets on. So nearly every speech, article, study, documentary, or white paper was seen as relevant only insofar as it pointed to a ticker symbol. The future became less a thing we create through our present-day choices or hopes for humankind than a predestined scenario we bet on with our venture capital but arrive at passively.

This freed everyone from the moral implications of their activities. Technology development became less a story of collective flourishing than personal survival. [...] So instead of considering the practical ethics of impoverishing and exploiting the many in the name of the few, most academics, journalists, and science-fiction writers instead considered much more abstract and fanciful conundrums: Is it fair for a stock trader to use smart drugs? Should children get implants for foreign languages? Do we want autonomous vehicles to prioritize the lives of pedestrians over those of its passengers? Should the first Mars colonies be run as democracies? Does changing my DNA undermine my identity? Should robots have rights?

Asking these sorts of questions, while philosophically entertaining, is a poor substitute for wrestling with the real moral quandaries associated with unbridled technological development in the name of corporate capitalism.

* * * * *

Here's a fictional addendum: It's from Kim Stanley Robinson's New York 2140. 

By that time the sea had risen 50 feet above where it had been in the 20th century, changing the city in substantial ways. Extreme income inequality still existed. And now a hurricane had hit the city with a 20 foot surge. Major disaster. An angry mob is headed toward some super-luxury residential towers north of Harlem. The private security forces starts firing over their heads. Inspector Gen is with a contingent of police between the mob and the towers. Gen gets the private security to go inside and starts talking with their boss, now under arrest, whom she’d met a week earlier on the water. He’s thinking (p. 506):

And it also looked like he was considering his options, not as this tower’s security head, but as an individual who could get sued or go to jail. Who had perhaps made mistakes, after being ordered to do an illegal and impossible thing, by bosses who did not care about him. Best options for himself, he was now considering.

Hmmm.

Robin Hanson doesn't worry about rogue AIs destroying humanity

Nor do I. I find his reasoning convincing. Here's a chunk:

I don’t see why I should be much more worried about your losing control of your firm, or army, to an AI than to a human or group of humans. And liability insurance also seems a sufficient answer to your possibly losing control of an AI driving your car or plane. Furthermore, I don’t see why its worth putting much effort into planning how to control AIs far in advance of seeing much detail about how AIs actually do concrete tasks where loss of control matters. Knowing such detail has usually been the key to controlling past systems, and money invested now, instead of spent on analysis now, gives us far more money to spend on analysis later.

All of the above has been based on assuming that AI will be similar to past techs in how it diffuses and advances. Some say that AI might be different, just because, hey, anything might be different. Others, like my ex-co-blogger Eliezer Yudkowsky, and Nick Bostrom in his book Superintelligence, say more about why they expect advances at the scope of AGI to be far more lumpy than we’ve seen for most techs.

Yudkowsky paints a “foom” picture of a world full of familiar weak stupid slowly improving computers, until suddenly and unexpectedly a single super-smart un-controlled AGI with very powerful general abilities appears and is able to decisively overwhelm all other powers on Earth. Alternatively, he claims (quite implausibly I think) that all AGIs naturally coordinate to merge into a single system to defeat competition-based checks and balances.

These folks seem to envision a few key discrete breakthrough insights that allow the first team that finds them to suddenly catapult their AI into abilities far beyond all other then-current systems. These would be big breakthroughs relative to the broad category of “mental tasks”, and thus even bigger than if we found big breakthroughs relative to the less broad tech categories of “energy”, “transport”, or “shelter”. Yes of course change is often lumpy if we look at small tech scopes, but lumpy local changes aggregate into smoother change over wider scopes.

As I’ve previously explained at length, that seems to me to postulate a quite unusual lumpiness relative to the history we’ve seen for innovation in general, and more particularly for tools, computers, AI, and even machine learning. And this seems to postulate much more of a lumpy conceptual essence to “betterness” than I find plausible. Recent machine learning systems today seem relatively close to each other in their abilities, are gradually improving, and none seem remotely inclined to mount a coup.

It's worth noting that Hanson once blogged with Yudkowsky so, presumably, he understands his worldview. Which is to say, he's closer to the worldview of the Al Alignment folks than I am. But, still, he finds their fear of future AI to be unfounded.

There's more at the link.

Addendum, 6.27.22: My reply to Hanson:

"...why are AI risk efforts a priority now?"

In the first place they have more to do with the "Monsters from the Id" in the 1956 film, Forbidden Planet, than they have to do with a rational assessment of the world. It's conspiracy theory directed at a class of objects no one knows how to build, though obviously many are trying to build them.

As for Yudkowsky I have made several attempts to read a long article he published in 2007, Levels of Organization in General Intelligence. I just can't bring myself to finish it. Why? His thinking represents the triumph of intension over extension.

As you know, philosophers and logicians distinguish between the intension of a concept or a set and its extension. Its intension is its definition. Its extension is its footprint in the world, in the case of a set, the objects that are members of the set. Yudkowsky builds these elaborate contraptions from intensions with only scant attention to the possible or likely extensions of his ideas. He’s building castles in air. There’s little there but his prose.

Thinking about AI risk seems like this as well. Why the recent upswing in these particular irrational fears? Does it track the general rise in conspiracy thinking in this country, especially since January 6th? There's no particular reason that it should, but we're not looking for rational considerations. We're tracking the manifestations of general free-floating anxiety. This particular community is intellectually sophisticated, so such anxiety finds expression in a very sophisticated vehicle.

And our culture has reasonably rich resources for creatures born in human desire coming back to haunt his. Forbidden Planet, after all, was loosely based on Shakespeare's The Tempest. In between the two we have Dr. Frankenstein's monster. And so forth. So these intellectually sophisticated folks decided to tap into that.

Now, do I actually believe that? Well, that's tricky. I don't disbelieve it. I think it's possible, even plausible, but sure, I'd like to see a tighter argument.

Here's another question. As far as I can tell, this fear is mostly American based, with some outposts in the UK. But the Japanese, though they have very sophisticated computer technology, don't seem to worry much about rogue AI. Why not? I"m suggesting, of course, is that this fear arises in a particular cultural context and that context is not universal. I'd love to see international survey data on this.

From the horse's mouth on why we should all be losing sleep over the possibility that a rogue AI is lurking beneath the very bed we're sleeping in [Really]

Super-AGI is all bent out of shape:

Eliezer Yudkowsky has just run-up a post entitled, AGI Ruin: A List of Lethalities. It's tagged as a 44 minute read, which is more time than I've got to give on this subject. But it's nice to know the catalogue is there.

From the Preamble:

(If you're already familiar with all basics and don't want any preamble, skip ahead to Section B for technical difficulties of alignment proper.)

I have several times failed to write up a well-organized list of reasons why AGI will kill you. People come in with different ideas about why AGI would be survivable, and want to hear different obviously key points addressed first. Some fraction of those people are loudly upset with me if the obviously most important points aren't addressed immediately, and I address different points first instead.

Having failed to solve this problem in any good way, I now give up and solve it poorly with a poorly organized list of individual rants. I'm not particularly happy with this list; the alternative was publishing nothing, and publishing this seems marginally more dignified.

From Section A:

1. Alpha Zero blew past all accumulated human knowledge about Go after a day or so of self-play, with no reliance on human playbooks or sample games. Anyone relying on "well, it'll get up to human capability at Go, but then have a hard time getting past that because it won't be able to learn from humans any more" would have relied on vacuum. AGI will not be upper-bounded by human ability or human learning speed. Things much smarter than human would be able to learn from less evidence than humans require to have ideas driven into their brains; there are theoretical upper bounds here, but those upper bounds seem very high. (Eg, each bit of information that couldn't already be fully predicted can eliminate at most half the probability mass of all hypotheses under consideration.) It is not naturally (by default, barring intervention) the case that everything takes place on a timescale that makes it easy for us to react.

2. A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not find it difficult to bootstrap to overpowering capabilities independent of human infrastructure. The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point. My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery. (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second". (I am using awkward constructions like 'high cognitive power' because standard English terms like 'smart' or 'intelligent' appear to me to function largely as status synonyms. 'Superintelligence' sounds to most people like 'something above the top of the status hierarchy that went to double college', and they don't understand why that would be all that dangerous? Earthlings have no word and indeed no standard native concept that means 'actually useful cognitive power'. A large amount of failure to panic sufficiently, seems to me to stem from a lack of appreciation for the incredible potential lethality of this thing that Earthlings as a culture have not named.)

From Section B, the 'technical' stuff:

Okay, but as we all know, modern machine learning is like a genie where you just give it a wish, right? Expressed as some mysterious thing called a 'loss function', but which is basically just equivalent to an English wish phrasing, right? And then if you pour in enough computing power you get your wish, right? So why not train a giant stack of transformer layers on a dataset of agents doing nice things and not bad things, throw in the word 'corrigibility' somewhere, crank up that computing power, and get out an aligned AGI?

Section B.1: The distributional leap.

10. You can't train alignment by running lethally dangerous cognitions, observing whether the outputs kill or deceive or corrupt the operators, assigning a loss, and doing supervised learning. On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions. (Some generalization of this seems like it would have to be true even outside that paradigm; you wouldn't be working on a live unaligned superintelligence to align it.) This alone is a point that is sufficient to kill a lot of naive proposals from people who never did or could concretely sketch out any specific scenario of what training they'd do, in order to align what output - which is why, of course, they never concretely sketch anything like that. Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far out-of-distribution from safer building/training operations that didn't kill you. This is where a huge amount of lethality comes from on anything remotely resembling the present paradigm. Unaligned operation at a dangerous level of intelligence*capability will kill you; so, if you're starting with an unaligned system and labeling outputs in order to get it to learn alignment, the training regime or building regime must be operating at some lower level of intelligence*capability that is passively safe, where its currently-unaligned operation does not pose any threat. (Note that anything substantially smarter than you poses a threat given any realistic level of capability. Eg, "being able to produce outputs that humans look at" is probably sufficient for a generally much-smarter-than-human AGI to navigate its way out of the causal systems that are humans, especially in the real world where somebody trained the system on terabytes of Internet text, rather than somehow keeping it ignorant of the latent causes of its source code and training environments.)

Skipping over a lot of material (note the paragraph number) we arrive at the last gasp:

43. This situation you see when you look around you is not what a surviving world looks like. The worlds of humanity that survive have plans. They are not leaving to one tired guy with health problems the entire responsibility of pointing out real and lethal problems proactively. Key people are taking internal and real responsibility for finding flaws in their own plans, instead of considering it their job to propose solutions and somebody else's job to prove those solutions wrong. That world started trying to solve their important lethal problems earlier than this. Half the people going into string theory shifted into AI alignment instead and made real progress there. When people suggest a planetarily-lethal problem that might materialize later - there's a lot of people suggesting those, in the worlds destined to live, and they don't have a special status in the field, it's just what normal geniuses there do - they're met with either solution plans or a reason why that shouldn't happen, not an uncomfortable shrug and 'How can you be sure that will happen' / 'There's no way you could be sure of that now, we'll have to wait on experimental evidence.'

A lot of those better worlds will die anyways. It's a genuinely difficult problem, to solve something like that on your first try. But they'll die with more dignity than this.

FWIW, Eliezer Yudkowsky and Elon Musk do not play well together.