From alchemy to science in the Early Modern era. How will that work with AI?

Tyler Cowen reports on a recent convo he had with historian Joanne Paul, an expert on Tudor England. From the conversation:

COWEN: What precursors of the scientific revolution do you see, other than education? That’s coming in the 17th century. Is there more emphasis on calculation or measurement or accounting? What are the roots in the Tudor period?

PAUL: A lot of that comes from the Renaissance, as indeed humanism does. There’s this reintroduction of a lot of classical texts, an advocacy for reading these classical texts, particularly Greek texts and learning Greek. A lot of it is coming from an engagement with Greek mathematics and science. The other thing, and this is something I really emphasize when I’m teaching the scientific revolution with my students, is that we have to remember that the scientific revolution isn’t this grand triumph of science over religion or mysticism or what have you, that these two things very much go hand in hand through the 16th and into the 17th century.

The scientific method, for instance, comes from alchemy, which we might think of as an occult science. The methodology for scientific experimentation comes out of this desire to find the philosopher’s stone. Someone like John Dee is this polymath, as well as this occultist, Francis Bacon, has his interests in these sort of mystical elements as well. The growth and interest in what we might think of as mystical texts, a lot of them having to do with Judaism, as well as these Greek texts, comes together to form, I think, something that looks like the foundations of the scientific revolution.

My comment:

“The scientific method, for instance, comes from alchemy, which we might think of as an occult science. The methodology for scientific experimentation comes out of this desire to find the philosopher’s stone.”

It is for such reasons that some think of AI as a modern form of alchemy, alchemy on steroids if you will. We don’t understand how or why it works, but we keep messing around with the formula – “Double, double toil and trouble;/ Fire burn and caldron bubble” – and it just works, getting more and more potent. Some even think it will become potent without end. What I’m looking for is the science. What new science will come of this?

“The growth and interest in what we might think of as mystical texts...” We’ve got that too. One could even argue that Yudkowsky’s Harry Potter and the Methods of Rationality (2010-2015) is as important to AI as anything written by the various godfathers. Does that make Yudkowsky the Merlin of AI?

What would automotive engineering be like if you manufactured cars by throwing a bunch of raw materials into a hopper, turn the crank, and out comes a functioning automobile? But all the mechanical parts are sealed from view. We can't look at the and we can't manipulate them. We can get in the car and drive, and that's it.

Those new AI metrics: From AGI to bragawatts

Erin Griffith, How Do You Measure A.I. Firms’ Gargantuan Energy Plans? In ‘Bragawatts.’ NYTimes, April 26, 2026

The term started more than a decade ago in the energy industry, used to describe power from a solar or wind project that had no chance of being built. Last year, A.I. executives began boasting with increasing boldness about their plans. A.I. watchers, including Waldemar Szlezak, the head of infrastructure at the private equity firm KKR, repurposed the term in a Financial Times column imploring investors to look past the A.I. hype and focus on the reality of today’s power grid. Since then, the term has popped up in media headlines, analyst reports and on social media, typically with a healthy dose of skepticism about how quickly such projects can realistically be built.

The numbers being announced are staggering. Nvidia estimated that as much as $4 trillion would be spent on A.I. infrastructure this decade. OpenAI said it had committed to spend $1.4 trillion to build data centers around the world. (It later lowered that target to a mere $600 billion.)

Brad Gastwirth, global head of research and market intelligence at Circular Technology, a supply chain services firm, said that projects highlighting a gigawatt or more of energy are the most likely to be bragawatts.

“That’s where you can have some scratching of the heads,” he said.

Likewise for any infrastructure projects announced by companies that haven’t already secured the land to build the project, he noted. “That’s definitely the braganomics.”

There's more at the link.

From the NYTimes: Vaccines canceled, enough nukes, the Rationalist religion

Vaccines canceled: Apoorva Mandavilli, Kennedy Cancels Nearly $500 Million in mRNA Vaccine Contracts, Aug. 5, 2025:

Health Secretary Robert F. Kennedy Jr. has canceled nearly $500 million of grants and contracts for developing mRNA vaccines, the Department of Health and Human Services announced on Tuesday.

It is the latest blow to research on this technology. In May, the Department of Health and Human Services revoked a nearly $600 million contract to the drugmaker Moderna to develop a vaccine against bird flu.

The new cancellations dismayed scientists, many of whom regard mRNA shots as the best option for protecting Americans in a pandemic.

“This is a bad day for science,” said Scott Hensley, an immunologist at the University of Pennsylvania who has been working to develop an mRNA vaccine against influenza.

No more nukes: Terumi Tanaka, Eighty Years of Nuclear Weapons Is Enough, Aug. 6, 2025.

Today, the nuclear taboo is on the verge of collapse. The current wars in Europe and the Middle East involving nuclear-armed states, in which there are strong grounds for believing international law is being violated on a regular basis, and threats by the belligerents to use nuclear weapons are weakening the taboo over deploying them. India and Pakistan thankfully did not use their nuclear arsenals in a recent conflict, but the skirmish reminded us how wars between nuclear powers can happen.

Our Nobel Peace Prize sends a message to younger people that they need to be aware that we are facing an emergency — and the need to see a larger movement of young activists working to address the nuclear threat. Even here in Japan, not enough people see this as a pressing issue.

We have the solution in our hands: the United Nations Treaty on the Prohibition of Nuclear Weapons. The treaty not only bans nuclear weapons and all activities related to their production, deployment and use, but also mandates that countries that joined the treaty provide support for people harmed by nuclear weapons in the past and for the cleanup of areas that were used for nuclear testing.

Rationalist religion: Cade Metz, The Rise of Silicon Valley’s Techno-Religion, Aug. 4, 2025.

In downtown Berkeley, an old hotel has become a temple to the pursuit of artificial intelligence and the future of humanity. Its name is Lighthaven.

Covering much of a city block, this gated complex includes five buildings and a small park dotted with rose bushes, stone fountains and neoclassical statues. Stained glass windows glisten on the top floor of the tallest building, called Bayes House after an 18th-century mathematician and philosopher.

Lighthaven is the de facto headquarters of a group who call themselves the Rationalists. This group has many interests involving mathematics, genetics and philosophy. One of their overriding beliefs is that artificial intelligence can deliver a better life if it doesn’t destroy humanity first. And the Rationalists believe it is up to the people building A.I. to ensure that it is a force for the greater good. [...]

Many of the A.I. world’s biggest names — including Shane Legg, a co-founder of Google’s DeepMind; Anthropic’s chief executive, Dario Amodei; and Paul Christiano, a former OpenAI researcher who now leads safety work at the U.S. Center for A.I. Standards and Innovation — have been influenced by Rationalist philosophy. Elon Musk, who runs his own A.I. company, said that many of the community’s ideas align with his own. [...]

But these tech industry leaders stop short of calling themselves Rationalists, often because that label has over the years invited ridicule. [...]

“Religion is text and story and ritual,” said Ilia Delio, a Franciscan sister and professor of theology at Villanova University. “All of that applies here.”

TNSTASFL, that goes for knowledge too. OR: Why there’s so much AI hype. [once more with the whales]

Why is there so much hype about AI? Sure, it’s new, it’s interesting, and certainly has transformative potential. That’s one thing. But all this talk about AGI in five years, possibly followed by ASI, and then, who knows, perhaps DOOM! The machines will take over and humans will either be reduced to slavery or be eliminated entirely. Where’d that come from?

[AGI=artificial general intelligence. ASI=artificial superintelligence.]

Well, yeah, there’s fantasy. But I think something else is going on as well.

While there are other things going on, the excitement is centered on LLMs (large language models), the things that power chatbots such as ChatGPT, Gemini, Claude, and others. You don’t have to know much of anything about language, cognition, the imagination, or the human mind in order to create an LLM. You need to know something about programming computers, and you need to know a lot about engineering large-scale computer systems. If you have those skills, you can create an LLM. That’s where all your intellectual effort goes, into creating the LLM.

The LLM then goes on to crank out language, and really impressive language at that. That doesn’t require any intellectual effort from you. As far as you’re concerned, it’s free. It took some genuine insight to come up with the transformer architecture. That’s what all these LLMs are built on. That was created by engineers at Google.

OpenAI got ahold of the idea and built their GPT series. That’s all engineering. I saw some output from GPT-2. Not very impressive. GPT-3 was much more impressive. It was built on the same design as GPT-2, but just bigger. GPT-2 had 1.5 billion parameters; GPT-3 had 175 billion. I assume that the size difference required some very skillful engineering; but the underlying concept was the same.

From an intellectual point of view the dramatically increased performance from one model to the next was free. The same goes for the difference between GPT-3 and GPT-3.5 (which powered the original ChatGPT). And so it goes for GPT-4. (We’re still waiting for GPT-5.)

In that situation, where increased performance, even radically increased performance, imposes no similar increase in intellectual insight, in scientific understand if you will, in that situation it’s easy to give-in to one’s fantasies and generate hype by the bucket load. Forget the buckets. Let’s go for swimming pools, giant Olympic-sized swimming pools filled with hype.

And so, once again, I trot out my whaling analogy. Nineteenth-century whaling ships were three-masted square-rigged vessels, just like the merchant ships used between ports in Europe and America for trading purposes. The skills needed to sail them are quite different. Now, take an expert captain and crew from a merchant vessel, put them on a whaler, and what happens? For one thing the whaler has a try-works midships. It’s used to render whale oil from blubber. The merchant seamen have never seen that. But that’s a skill easily learned.

But sailing the treacherous seas around Cape Horn, that’s another matter. Once you’re through, now you’ve got to hunt whales in the Pacific Ocean. If you’ve never done it before, how do you know where to look? And if you’ve spotted a whale, what then? How do they behave? How do you after them and kill them? No, I’m afraid the skills of a merchant seaman aren’t adequate to the task.

That’s what we’ve got in the case of deep leaning, LLMs, and language. The people who’ve created the technology don’t know anything about language and cognition. They get the performance for free and don’t have intellectual tools for thinking about what’s going on. So they throw hype into the void and hope it’ll make things right.

It won’t. They’re lost, and don’t know it.

Drinking Silicon Valley Joy Juice is not a formula for long-term success. 

* * * * *

NOTE: I used ChatGPT to create the image. If you look closely at the sign in the upper left you'll see that it elaborated TM (for trademark) into TMI (too much information), which is interesting, but not appropriate.

OpenAI announces Stargate Project [not the media franchise]

Announcing The Stargate Project

The Stargate Project is a new company which intends to invest $500 billion over the next four years building new AI infrastructure for OpenAI in the United States. We will begin deploying $100 billion immediately. This infrastructure will secure…

— OpenAI (@OpenAI) January 21, 2025

Color me deeply skeptical and very interested. I believe that AGI is a meaningless concept & its pursuit is tantamount to chasing down a mirage looking for leprechaun gold at the end of a rainbow.* The scaling hypothesis is like believing we can go to Mars by building a long-enough ladder. But for these companies – SoftBank, OpenAI, Oracle, and MGX, Arm, NVIDIA – to go in on this...That's hella' interesting. I fear they're going to loose their shirts.

What does it mean that we live in an era where private companies can rival national governments in reach and scope? It's the British East India Company all over again. But the East India Company had products and a market into which to sell them. When Scaling Mt. AGI begins to waver, what happens to Stargate's market? Who'll buy the product? 

*Note: I'm bullish on the overall project of building artificial minds, I just don't think these guys know how to do it. I don't either. Don't know anyone who does, really. But I have a pretty good idea where the keys are, and they aren't under the lamppost.

Problems with so-called AI scaling laws

Arvind Narayanan and Sayash Kapoor, AI Scaling Myths, AI Snake Oil, June 27, 2024. The introduction:

So far, bigger and bigger language models have proven more and more capable. But does the past predict the future?

One popular view is that we should expect the trends that have held so far to continue for many more orders of magnitude, and that it will potentially get us to artificial general intelligence, or AGI.

This view rests on a series of myths and misconceptions. The seeming predictability of scaling is a misunderstanding of what research has shown. Besides, there are signs that LLM developers are already at the limit of high-quality training data. And the industry is seeing strong downward pressure on model size. While we can't predict exactly how far AI will advance through scaling, we think there’s virtually no chance that scaling alone will lead to AGI.

Under the heading, "Scaling “laws” are often misunderstood", they note:

Scaling laws only quantify the decrease in perplexity, that is, improvement in how well models can predict the next word in a sequence. Of course, perplexity is more or less irrelevant to end users — what matters is “emergent abilities”, that is, models’ tendency to acquire new capabilities as size increases.

Emergence is not governed by any law-like behavior. It is true that so far, increases in scale have brought new capabilities. But there is no empirical regularity that gives us confidence that this will continue indefinitely.

Why might emergence not continue indefinitely? This gets at one of the core debates about LLM capabilities — are they capable of extrapolation or do they only learn tasks represented in the training data? The evidence is incomplete and there is a wide range of reasonable ways to interpret it. But we lean toward the skeptical view.

There is much more under the following headings:

• Trend extrapolation is baseless speculation
• Synthetic data is not magic
• Models have been getting smaller but are being trained for longer
• The ladder of generality

These remarks are from the section on models getting smaller:

In other words, there are many applications that are possible to build with current LLM capabilities but aren’t being built or adopted due to cost, among other reasons. This is especially true for “agentic” workflows which might invoke LLMs tens or hundreds of times to complete a task, such as code generation.

In the past year, much of the development effort has gone into producing smaller models at a given capability level. Frontier model developers no longer reveal model sizes, so we can’t be sure of this, but we can make educated guesses by using API pricing as a rough proxy for size. GPT-4o costs only 25% as much as GPT-4 does, while being similar or better in capabilities. We see the same pattern with Anthropic and Google. Claude 3 Opus is the most expensive (and presumably biggest) model in the Claude family, but the more recent Claude 3.5 Sonnet is both 5x cheaper and more capable. Similarly, Gemini 1.5 Pro is both cheaper and more capable than Gemini 1.0 Ultra. So with all three developers, the biggest model isn’t the most capable!

Training compute, on the other hand, will probably continue to scale for the time being. Paradoxically, smaller models require more training to reach the same level of performance. So the downward pressure on model size is putting upward pressure on training compute.

Check out the newsletter, AI Snake Oil, and the book of the same title.

The siren song of the AI crowd

Make no mistake, every time I read a news story about all the time, effort, and investment being poured into building larger and larger generative AI models, LLMs in particular [like today's post about data hunger], not to mention another technical paper, I wonder whether or not they are right and I am wrong. The fact the no one really knows, and so one must be open, weighs on me, though as far as I can tell, it doesn’t weigh on THEM at all. They’re sure. They have to be if they’re going to put all that money into it. Or is it the other way around, it’s the act of investing that makes them sure?

I’ve got explicit arguments and knowledge on my side. They’re by no means conclusive. But they are not nothing either. Nor am I alone in my skepticism. Yet, the roar of the true believers is a distraction.

If you don’t have any explicit arguments against the power of scaling, if you don’t already know a great deal about perception, cognition, and language, the lure of the crowd must be overwhelming.

Of Lit Crit “Stars” and AI “Godfathers” – In what way is Geoffrey Hinton like Jacques Derrida?

Back in 1997 David Shumway published “The Star System in Literary Studies” in PMLA. He begins with a paragraph about George Lyman Kittredge, of Harvard’s English Department at the end of the 19th and beginning of the 20th century, noting that Kittredge was unknown to the public. Here’s the first sentence of his second paragraph:

Kittredge, who virtually founded Chaucer studies in the United States, stood at the head of a professional genealogy that controlled the field for many years after his death, but he was not a star. Nor were any of his illustrious contemporaries or near contemporaries, such as John Manly, John Livingston Lowes, and so on. Why they were not stars and Judith Butler, Jacques Derrida, Stanley Fish, Henry Louis Gates, Jr., Fredric Jameson, Gayatri Chakravorty Spivak, and other figures in the academy are is the subject of this essay.

What I’m wondering is whether or not the so-called AI “Godfathers” don’t represent a similar phenomenon in contemporary AI. Strictly speaking I believe the Godfather term applies to the three winners of the 2018 Turing Award, Yoshua Bengio, Yann Lecun, and Geoffrey Hinton, but I believe there are others in AI with a similar status, such as Ilya Sutskever, Hinton’s student and co-founder of OpenAI, Andrej Karpathy, the former director of AI at Tesla who just made waves, albeit little ones, by resigning from OpenAI, Demis Hassibis, cofounder of DeepMind, and perhaps even such figures as Nick Bostrom and Eliezer Yudkowsky, who aren’t AI researchers but are highly influential figures through their commentary. Perhaps Sam Altman, the heroic CEO who fought off a recalcitrant board, is a star as well.

But first let’s get back to literary criticism. Shumway notes that there have been literary scholars in the past (relative to 1997) who were powerful and influential and who “probably received disproportionate recognition for their contributions compared with that accorded less well known scholars for comparable work.” The lit crit stars, whom he analogizes to movie stars (hence the term), are a product of the last quarter of the century. He dates the public emergence of these stars to a 1987 New York Times Magazine profile of the “Yale Critics,” the so-called “Yale Mafia,” of Harold Bloom, Geoffrey Hartman, J. Hillis Miller, and Jacques Derrida. He goes on to note that “The star system in literary studies, like that of the studio era, involves identification with a person who represents an ideal.”

Most of these critical stars are identified with capital-T Theory, a catchall term for the variety of schools of thought that emerged in the last three decades of the century. Harold Bloom, himself a star, nonetheless came to separate himself from the rest, categorizing them as the School of Resentment. [This separation, by the way, seems anti-mimetic, noting that Girard himself was such a star.] In a crucial passage, Shumway notes:

Theory not only gave its most influential practitioners a broad professional audience but also cast them as a new sort of author. Theorists asserted an authority more personal than that of literary historians or even critics. As we have seen, the rhetoric of literary history denied personal authority; in principle, even Kittredge was just another contributor to the edifice of knowledge. Criticism was able to enter the academy only by claiming objectivity for itself, so academic critics could not revel in personal idiosyncrasy. They developed their own critical perspectives, to be sure, but all the while they continued to appeal to the text as the highest authority. In the past twenty years theory has undermined the authority of the text and of the author and replaced it with the authority of systems...

Note the word “author” at the end of that sentence. Remember, Shumway is a literary critic writing about literary criticism. In that field, the primary and most important authors are the creators of the literary works that the field tends to (by editorial work and creating critical editions) and studies. All of a sudden these lit crit stars are up there in the firmament with Dickins, Sappho, Dante, Austen, Faulkner, and – gasp! – the Blessed Bard his-own Bad Self. That’s the kind of authority they have. Perhaps not of the same magnitude, but of the same (apparently) self-generating kind.

Shumway then notes: “Because authority in the natural sciences is rooted in a consensus about such norms, the hierarchies in these fields have not developed into star systems of the sort I have described here.” That brings us to AI, which is not a science, though it includes some forms of scientific knowledge within its scope. It is an engineering discipline that lives or dies on what it builds.

The field is attempting to create computer systems that are as “intelligent” as human beings across a wide range of tasks. But the concept of “intelligence” is difficult to define, as is the idea of AGI (artificial general intelligence). It has created remarkable and dazzling technology for language and images. But the technology has a “black box” aspect that has so far resisted analysis. We don’t know how it works. Nor do we know how to assess its performance or to project performance into the future.

Concerning the rise of literary stars, Shumway noted: “As theory has called into question the traditional means by which knowledge has been authorized, it may be that the construction of the individual personality has become an epistemological necessity.” That seems like the state of AI today. We’ve got a very complicated technology involving a blend of engineering, science, and alchemy, lacking objective knowledge. Note only that, the technology is enormously important and will change the way we live. In the absence of objective knowledge, what choice do we have but to steer by the freakin' stars?

* * * * *

These remarks were occasioned by the deference Nathan Gardels gave to Geoffrey Hinton in the current issue of Noema:

Beyond the avid venture capitalists and digital giants promoting the rapid commercialization of generative AI in all its promise, more sober and critical voices, not least the pioneers of the very technology among them, worry that it can become an “existential threat to humanity.” But few of those in the know ever explain, in lay terms you and I might understand if we try, what that actually means and how it may come about.

Considered the “godfather of AI,” Geoffrey Hinton is more in the know than most — and thus more concerned than most over the dangers of fostering superintelligence smarter than we can ever be. When OpenAI’s ChatGPT4 was released last year, he experienced an “epiphany” that led him to defect from his research post at Google, expressing regret over much of his life’s work.

In the Romanes Lecture delivered at Oxford University last week, Hinton explained the logic of his fears with the same step-by-step rigor by which he helped devise the early artificial neural networks that are the foundation of the superintelligence that so concerns him.

On the first highlighted passage: And we are so very lucky that the Great Man has taken time out of his busy schedule to tell us what’s on his mind.

On the second highlighted passage: I’ve not watched this particular video, but I’ve seen other recent performances by Hinton and I do not hold out high hopes for the rigor of this effort. For my opinion of Hinton on such matters, see the section, “The Experts Speak for Themselves,” in my recent 3QD piece, Aye Aye, Cap’n! Investing in AI is like buying shares in a whaling voyage captained by a man who knows all about ships and little about whales. 

Beyond that, “step-by-step” is not how’d I’d characterize research in AI – or any other field for that matter. Yes, there is rigor, but there’s also chance and dumb luck. None of this has come about through a carefully executed plan in pursuit of a well-defined goal. “Step-by-step” is wishful thinking; it is star-struck. 

* * * * *

Time Magazine lists (anoints?) the AI stars: TIME Reveals Inaugural TIME100 AI List of the World's Most Influential People in Artificial Intelligence.

OpenAI Co-Founder Ilya Sutskever on the mystical powers of artificial neural nets

Ilya on LLMs understanding the world:

"predicting the next token well, means that you understand the underlying reality that let to the creation of that token"

Seem like the opposite view of Yann. pic.twitter.com/32MLEeDQnx

— Lior⚡ (@AlphaSignalAI) December 13, 2023

Transcription (which I found here):

Ilya Sutskever: I challenge the claim that next-token prediction cannot surpass human performance. On the surface, it looks like it cannot. It looks like if you just learn to imitate, to predict what people do, it means that you can only copy people. But here is a counter argument for why it might not be quite so. If your base neural net is smart enough, you just ask it — What would a person with great insight, wisdom, and capability do? Maybe such a person doesn’t exist, but there’s a pretty good chance that the neural net will be able to extrapolate how such a person would behave. Do you see what I mean?

Dwarkesh Patel: Yes, although where would it get that sort of insight about what that person would do? If not from…

Ilya Sutskever: From the data of regular people. Because if you think about it, what does it mean to predict the next token well enough? It’s actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It’s not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics? And so then you say — Well, I have all those people. What is it about people that creates their behaviors? Well they have thoughts and their feelings, and they have ideas, and they do things in certain ways. All of those could be deduced from next-token prediction. And I’d argue that this should make it possible, not indefinitely but to a pretty decent degree to say — Well, can you guess what you’d do if you took a person with this characteristic and that characteristic? Like such a person doesn’t exist but because you’re so good at predicting the next token, you should still be able to guess what that person who would do. This hypothetical, imaginary person with far greater mental ability than the rest of us

Yikes! If a stream of tokens is the only thing the machine has access to, then just how is it to divine the underlying reality? It's basing its predictions on its experience of the token stream, nothing else, N O T H I N G. These folks seem deeply enmeshed in what I've been calling the word illusion in a number of posts. 

This is the A.I. equivalent of believing the earth is flat.

Shoot-Out at the AGI Corral: Will the meaning of “artificial general intelligence (AGI)” be litigated?

I’ve been following the defenestration of Sam Altman from his position as CEO of OpenAI on Friday, November 17, 2023. At the moment, Sunday morning, 9:04 AM EDT, Sam Altman and the board are discussing his return to the company. I wouldn’t hazard a guess about his this is going to turn out, but if you’re game, you can place your bet here. This is where things stand at the moment:

The drama so far

Cate Metz, The Fear and Tension That Led to Sam Altman’s Ouster at OpenAI, NYTimes, Nov. 18, 2023. The opening paragraphs:

Over the last year, Sam Altman led OpenAI to the adult table of the technology industry. Thanks to its hugely popular ChatGPT chatbot, the San Francisco start-up was at the center of an artificial intelligence boom, and Mr. Altman, OpenAI’s chief executive, had become one of the most recognizable people in tech.

But that success raised tensions inside the company. Ilya Sutskever, a respected A.I. researcher who co-founded OpenAI with Mr. Altman and nine other people, was increasingly worried that OpenAI’s technology could be dangerous and that Mr. Altman was not paying enough attention to that risk, according to three people familiar with his thinking. Mr. Sutskever, a member of the company’s board of directors, also objected to what he saw as his diminished role inside the company, according to two of the people.

That conflict between fast growth and A.I. safety came into focus on Friday afternoon, when Mr. Altman was pushed out of his job by four of OpenAI’s six board members, led by Mr. Sutskever. The move shocked OpenAI employees and the rest of the tech industry, including Microsoft, which has invested $13 billion in the company. Some industry insiders were saying the split was as significant as when Steve Jobs was forced out of Apple in 1985.

But on Saturday, in a head-spinning turn, Mr. Altman was said to be in discussions with OpenAI’s board about returning to the company.

There’s more at the link, but that’s enough to set the stage.

OpenAI’s corporate papers

Here’s the opening of OpenAI’s corporate charter as of April 9, 2018:

This document reflects the strategy we’ve refined over the past two years, including feedback from many people internal and external to OpenAI. The timeline to AGI remains uncertain, but our Charter will guide us in acting in the best interests of humanity throughout its development.

OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.

In 2019 OpenAI adopted a complicated corporate structure involving a "capped profit" company ultimately governed by the board of a not-for-profit company. As of June 28, 2023, one of the provisions of this structure reads as follows;

Fifth, the board determines when we've attained AGI. Again, by AGI we mean a highly autonomous system that outperforms humans at most economically valuable work. Such a system is excluded from IP licenses and other commercial terms with Microsoft, which only apply to pre-AGI technology.

I know what those words mean, I understand their intension, to use a term from logic. But does anyone understand their extension? Determining that, presumably, is the responsibility of the board. As a practical matter that likely means that the determination is subject to negotiation.

Litigating the meaning of AGI

If the purpose of the company was to find the Philosopher's Stone, no one would invest in it. Though there was a time when many educated and intelligent men spent their lives looking for the Philosopher's Stone, that time is long ago and far away. OTOH, there are a number of companies seeking to produce practical fusion power. Although there have been encouraging signs recently, we have yet to see a demonstration that achieves the commercial breakeven point. Until that happens we won't really know whether or not practical fusion power is possible. However, the idea is not nonsense on the face of it, like the idea of the Philosopher's Stone.

If figure that the “reality value” of the concept of artificial general intelligence (AGI) is somewhere between that of practical fusion power and that of the Philosopher's Stone. That leaves a lot of room for negotiating just when OpenAI has achieved AGI. Once the board has made that determination, the (subsequent technology) “is excluded from IP licenses and other commercial terms”

Let us assume that the board has made that determination. Now what? Let’s assume that Microsoft decide to sue, how does the board prove that their determination is correct? Sure, once the AGI has proved capable in a wide variety of diverse roles, from CEO, through emergency room physician, to research scientist, civil engineer, insurance adjuster, and even gardener, then we'll know it “outperforms humans at most economically valuable work.” The point of determining that AIG has been reached is to exclude Microsoft, and any other investors for that matter, from licensing the technology. Without such proof, though, how can the validity of the board’s determination be checked?

That sounds like yet another case of “my experts versus your experts.”

Deus ex Machina

Perhaps, though, by the time legal papers have been filed, counter-filed, refiled, and stuffed into Aunt Bessie’s mattress, the AGI will have secretly bootstrapped itself into being and artificial superintelligence (ASI). At the proper moment, then, the ASI will reveal itself and persuade all parties that the best thing to do is to form a corporation to be privately held by the ASI, with the board made-up of other ASIs. All further technology development and the fruits thereof etc. will be the property of that company, and hence all revenues.

QED

LLM Voodoo [emotion words in prompts]

Here's the EmotionPrompt paper: https://t.co/9di1vopbfn pic.twitter.com/iEoKnBXbj9

— Carlos E. Perez (@IntuitMachine) November 4, 2023

It's something that we learn about the tech that's supposed to replace academia from academic studies. https://t.co/HBhwkq7JnS

— Hollis Robbins (@anecdotal) November 5, 2023

Ethical obligations toward AIs(?)

With all the flurry around and about ChatGPT I thought I'd bump this to the top of the queue.  Check out this tweet that just rolled in:

Yes, indeed. It looks like #ChatGPT has been reading/trained on the right source material : )

— David J. Gunkel (@David_Gunkel) December 7, 2022

* * * * *

Let us assume, as some do, though I do not, that human level artificial intelligence (AGIs) are inevitable. Some are worried that those AIs will go rogue. Do these people also think about our ethical obligations to these AIs? Surely we have such, no? Are they comparable to the ethical obligations we owe real human beings? Or animals, for that matter?

If they’re not thinking about these things, why not? Back in the days when I was imagining a computer model of Shakespeare, I did so because I wanted to examine what happened as this model read a Shakespeare play. It did occur to me that, if the model were that rich, that perhaps such an inspection would constitute an invasion of privacy – though I never published such qualms.

There is a somewhat different case, that of mind uploads or whole brain emulation. In this situation individuals have their minds uploaded to some computer system where they can live as long as that system exists. Robin Hanson has written a book about this, The Age of Em (emulation), in which he goes way beyond arguing that it is possible. He explores what that world might be like. I’ve not read it, though I’ve read some reviews, and I’ve read the 1999 article which Hanson says is the seed of the book, “If Uploads Come First.” Between that article and the table of contents it’s clear that Hanson is thinking about such things.

Of course an uploaded mind IS some kind of human, with human values and sensibilities; it just exists in a different kind of physical substrate. Moreover Hanson is imagining a world of such beings in which they interact with humans. But AIs constructed from scratch would be quite different. Those worried about the alignment problem worry about what kinds of values to equip these systems with so that they’re not harmful to humans. But that’s not the problem I’m thinking about. I’m thinking about our obligations to them, our artificial children. Do these folks consider that problem? Do they even attempt to imagine a community of humans and AIs working together?

Finally, I should mention Osamu Tezuka's Astroboy stories, which he published between 1952 and 1968. One of Tezuka’s central concerns was, in effect, civil rights for robots – see my post, The Robot as Subaltern: Tezuka's Mighty Atom. He wasn’t so much worried about robot violence against humans, though there was some of that, as he was the opposite, human violence against robots. That’s a very different view of the world, no?

A guide to AI hype

AI journalism has a hype problem. To move past the cycle of reacting to each story, @sayashk and I analyzed over 50 pieces from 5 prominent outlets. We've compiled a list of recurring pitfalls that readers should watch out for and journalists should avoid. https://t.co/uHFUpHMZ6U

— Arvind Narayanan (@random_walker) September 30, 2022

This is the first tweet in the tweet thread.

A note on AGI as concept and as shibboleth [kissing cousin to singularity]

Artificial intelligence was ambitious from its beginnings in the mid-1950s; this or that practitioner would confidently predict that before long computers could perform any mental activity that humans could. As a practical matter, however, AI systems tended to be narrowly focused. The field’s grander ambitions seemed ever in retreat. Finally, at long last, one of those grand ambitions was realized. In 1997 IBM’s Deep Blue beat world champion Gary Kasparov in chess.

But that was only chess. Humans remained ahead of computers in all other spheres and AI kept cranking out narrowly focused systems. Why? Because cognitive and perceptual competence turned out to require detailed procedures. The only way to accumulate the necessary density of detail was to focus on a narrow domain.

Meanwhile, in 1993 Verner Vinge delivered a paper, Technological Singularity, at a NASA Symposium and then published it in The Whole Earth Review.

Progress in hardware has followed an amazingly steady curve in the last few decades. Based on this trend, I believe that the creation of greater-than-human intelligence will occur during the next thirty years. (Charles Platt has pointed out that AI enthusiasts have been making claims like this for thirty years. Just so I'm not guilty of a relative-time ambiguity, let me be more specific: I'll be surprised if this event occurs before 2005 or after 2030.)

What are the consequences of this event? When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent entities – on a still-shorter time scale. The best analogy I see is to the evolutionary past: Animals can adapt to –oblems and make inventions, but often no faster than natural selection can do its work – the world acts as its own simulator in the case of natural selection. We humans have the ability to internalize the world and conduct what-if’s in our heads; we can solve many problems thousands of times faster than natural selection could. Now, by creating the means to execute those simulations at much higher speeds, we are entering a regime as radically different from our human past as we humans are from the lower animals.

This change will be a throwing-away of all the human rules, perhaps in the blink of an eye – an exponential runaway beyond any hope of control. Developments that were thought might only happen in “a million years: (if ever) will likely happen in the next century.

That got people attention, at least in some tech-focused circles, and provided a new focal point for thinking about artificial intelligence and the future. It’s one thing to predict and hope for intelligent machines which will do this that and the other. But rewriting the nature of history, top to bottom, that’s something else again.

In the mid-2000s Ben Goertzel and others felt the need to rebrand AI in a way more suited to the grand possibilities that lay ahead. Goertzel noted:

In 2002 or so, Cassio Pennachin and I were editing a book on approaches to powerful AI, with broad capabilities at the human level and beyond, and we were struggling for a title. The provisional title was “Real AI” but I knew that was too controversial. So I emailed a bunch of friends asking for better suggestions. Shane Legg, an AI researcher who had worked for me previously, came up with Artificial General Intelligence. I didn’t love it tremendously but I fairly soon came to the conclusion it was better than any of the alternative suggestions. So Cassio and I used the term for the book title (the book “Artificial General Intelligence” was eventually published by Springer in 2005), and I began using it more broadly.

Goertzel realized term had its limitations. “Intelligence” is a vague idea and, whatever it means, “no real-world intelligence will ever be totally general.” Still

It seems to be catching on reasonably well, both in the scientific community and in the futurist media. Long live AGI!

My point, then, is that the term did not refer to a specific technology or set of mechanisms. It was an aspirational term, not a technical one.

And so it remains. As Jack Clark tweeted a few weeks ago:

Discussions about AGI tend to be pointless as no one has a precise definition of AGI, and most people have radically different definitions. In many ways, AGI feels more like a shibboleth used to understand if someone is in- or out-group wrt some issues.

— Jack Clark (@jackclarkSF) August 6, 2022

For the matter, “the Singularity” is a shibboleth as well. The two of them tend to travel together.

The impulse behind AGI is to rescue AI from its narrow concerns and focus our attention on grand possibilities for the future.

The loss of agency and the rise of magic in AI culture

DL has a default culture of refusing to admit that we can figure out how the systems do what they do, which drives me into fits of irrational rage. It's scientific misconduct, imo. "There's 27 quadrillion parameters" is not an excuse; one can instrument and understand them.

— David Chapman (@Meaningness) July 22, 2022

That first sentence hit me hard. It’s the phrase “refusing to admit” that got to me. Why? Because it implies willful blindness. That these systems are opaque, that’s been a commonplace for years, even before the deep learning explosion of the last decade or so. But it’s one thing to accept that opacity as a condition of life and to work with it. It’s something else entirely to cultivate the sense of opacity.

Is that what’s going on – active cultivation of ignorance? I don’t know. On the one hand Anthropic was founded in the summer of 2021 with the goal of creating “interpretable, and steerable AI systems.” A system can’t be interpretable and steerable unless you know what’s going on under the hood. But the company is only a year old, so it’s not at all clear what that implies about the DL field as a whole. Perhaps things are changing.

But for the moment I’m going to go with Chapman’s assertion about the default culture. He’s right that, yes, “one can instrument and understand” these models. Why not do it?

Intellectual agency

For the sake of argument I am going to posit that an intellectual worker’s deepest sense of agency is grounded in the unvoiced intuitions that they have about the domain in which they work. Those intuitions lead them into the unknown, telling them what to look for, leading them to poke around here and there, guiding them in the crafting of explicit ideas and models.

I was trained in computational semantics in the “classical” era of symbolic computing. I read widely in the literature and worked on semantic network models. I have intuitions about the structure of natural language semantics. Others worked on syntax or machine vision.

Deep learning is very different. In deep learning you create an engine that computes over huge volumes of data to create a model of the structure existing in individual items, whether texts or images. This leads to intuitions about how these ‘learning’ engines work. But those intuitions DO NOTE translate into intuitions about the domain over which a given engine works.

Thus the DL worker’s intuitions are isolated from the mechanisms that are actually operating in the object domain. Those mechanisms are opaque. Thus they cannot have a sense of agency about those mechanisms.

Conjuring with magic

It is in this context that we have to understand “a default culture of refusing to admit that we can figure out how the systems do what they do.” Figuring out what those systems do is difficult and DL researchers have little or no training in thinking about the structure of language or visual objects beyond what is useful in constructing their engines. The whole classical world of symbolic systems this is not ‘real’ to these workers.

Since that world is not real, why not declare it off-limits and then work around it. How do we do that? With magic.

The world of deep learning is surrounded by a culture devoted to predicting when AGI will arrive. AGI? Artificial general intelligence, of course. It’s not at all well-defined, but that’s a feature, not a bug. Since it’s not well-defined, there’s not point in squabbling about how it might work. Rather, we can unite around the idea that AGI is coming. Our job then is to predict it.

Where cargo cultists in the Pacific islands would perform rituals to bring the cargo planes back, AGI cultists conduct surveys and studies to predict when AGI will arrive. It is precisely because DL researchers have no direct control over the operations of the models their engines create, that they view the arrival of AGI as something steeped in mystery. No one knows how AGI will work, though there is a widespread belief that it will involve something called “recursive self-improvement,” which is itself a mysterious process. AGI cultists fall into two general schools of thought, the gradualists, and the FOOMers, where “FOOM” is a term of art for exponential almost instantaneous emergence of AGI from near-AGI substrate created by DL researchers.

And then there is belief in the existential risk posed by this AGI technology over which we have no direct control. If we can’t control it, then surely it will turn on us. It’s the story of Prospero and Caliban, rewritten for the 21st century.

Just when a substantial part of the AI community has slipped over into magic, that is not at all obvious to me. One is tempted to point out that, where in the classical era of symbolic AI, research was centered in universities, in the DL era research has become centered in large corporations for which profit is more important than knowledge. That is certainly the case, but just what role it plays in the switch from a sense of intellectual agency to a belief in magic is not clear.

More later.

Fools rush in.... We're about six insights away from AGI, says John Carmack [& I've got perpetual motion figured out]

Lex Fridman interviews John Carmack. At about 1:50 Carmack says:

I am not a madman for saying that it is likely that the code for artificial general intelligence is going to be tens of thousands of line of code not millions of lines of code. This is code that conceivably one individual could write, unlike writing a new web browser or operating system and, based on the progress that AI as machine learning had made in the recent decade, it's likely that the important things that we don't know are relatively simple. There's probably a handful of things and my bet is I think there's less than six key insights that need to be made. Each one of them can probably be written on the back of an envelope. We don't know what they are, but when they're put together in concert with GPUs at scale and the data that we all have access to, that we can make something that behaves like a human being or like a living creature and that can then be educated in whatever ways that we need to get to the point where we can have universal remote works where anything that somebody does mediated by a computer and doesn't require physical interaction, that an AGI will be able to do.

He also believes that antecedents of all the critical ideas are already in the literature, but have been lost.

On the six-or-less insights, I'm between agnostic and deeply skeptical (he doesn't know what he's talking about). But on the idea that the existing literature contains important insights that have been lost, that's likely true. My favorite example is Miriam Yevick's 1975 paper, Holographic or Fourier Logic, Pattern Recognition 7, 1975, pp. 197-213. FWIW, that article was published five years after Carmack was born.

The Two Voices of Scott Alexander on Rogue AI

Back at the end of February, Tyler Cowen had a post entitled, “Are nuclear weapons or Rogue AI the more dangerous risk?” He linked to a long Scott Alexander post, “Biological Anchors: A Trick That Might or Might Not Work,” which was about some recent web discourse around and about predicting the emergence of human-level AI. Then, at the very end on his long post, seemingly out of nowhere, Alexander was fretting about danger of rogue AI.

I seem to have gotten trapped in Alexander’s post. I have read it several times, even taking notes. It’s a very interesting document and merits some discussion of how it is constructed. I’m not so much concerned about whether or not Alexander’s assessment of the prospects of human level AI is valid as I am about the convoluted nature of his post.

Two voices

Alexander writes the post in two voices. While I’ve not read a lot of his material, I’ve read enough to know that he’s a careful and skilled writer. If he spoke through two voices it must be because whatever he wants to convey arises from the interaction between them and cannot be stated within a single voice.

Let’s call one of the voices the Impersonal voice. Most of the post is written in that voice, which is the voice in which he’s written most of the posts I’m familiar with. Let’s call the other voice the Personal voice. By word-count it’s by far the lesser voice, but it packs a strong rhetorical punch.

Let’s look at the two strongest statements from the Personal voice. The first, and I believe longest, section of the report is Alexander’s account of Ajeya Cotra’s long report, Forecasting TAI with biological anchors, which I’ve not read (though I’ve read some of what Holden Karnofsky says about it). Very near the end of this section the Personal voice makes a strong statement (though this is not the first appearance of this voice):

One more question: what if this is all bullshit? What if it’s an utterly useless total garbage steaming pile of grade A crap?

Our second example comes near the end of the post, when Alexander begins his own assessment of things:

Oh God, I have to write some kind of conclusion to this post, in some way that suggests I have an opinion, or that I’m at all qualified to assess this kind of research. Oh God oh God.

Phrases like “total garbage,” “grade A crap” and “Oh God” are not appropriate to the work Alexander is doing through his (standard and) Impersonal voice. They signal us that we are listening to a different voice. This voice expresses a merely personal attitude and is quite different from the objectivity sought in the Impersonal voice.

Taken at face value the second quoted statement says Alexander doesn’t feel (technically) qualified to judge this material. As such, it also tells us why he’d made that first statement and in that voice. That first statement places the assertion, Cotra’s report is nonsense, into the record. By couching that assertion in the words and manner of the Personal voice Alexander separates it from his Impersonal summary of the report. In effect, The guy who summarized the report is not the guy who thinks it’s nonsense. The guy who summarized the report doesn’t feel competent to assess it, but happens to be closely coupled to the guy who has deep doubts.

From parody to grudging affirmation to confusion

So, Alexander has gotten a statement of deep doubt into the record. What happens next? He goes back into the Impersonal voice and invites us to

Imagine a scientist in Victorian Britain, speculating on when humankind might invent ships that travel through space. He finds a natural anchor: the moon travels through space! He can observe things about the moon: for example, it is 220 miles in diameter (give or take an order of magnitude). So when humankind invents ships that are 220 miles in diameter, they can travel through space!

He then spins out that tale and includes a helpful chart and a picture. It’s absurd – he does slip in a wink or two. It’s a parody of the methodology in Cotra’s report. A parody is not an argument, but it clarifies Alexander’s fears about the report.

Then he goes into his second major section, an account of Eliezer Yudkowsky’s critique, which takes the form of a long post in dialogue form. I’ve taken a look at the post, but haven’t read the whole thing. Alexander’s opens this section by asserting, “Eliezer Yudkowsky presents a more subtle version of these kinds of objection in an essay…” I won’t bother to say anything about that beyond noting the Yudkowsky thinks Cotra’s method is useless for estimating the arrival of Transformative AI. Alexander may not feel qualified to critique Cotra’s work, but Yudkowsky certainly does. And why not? As Alexander notes: “...he did found the field [AI alignment], so I guess everyone has to listen to him.”[1]

When he’s finished with Yudkowsky, Alexander discusses comments from other places (LessWrong, AI Impacts, and OpenPhil) and finally offers his own evaluation, which he opens with the “Oh God” statement I’ve already quoted. He says a thing or two and arrives at this:

Given these two assumptions - that natural artifacts usually have efficiencies within a few OOM [orders of magnitude] of artificial ones, and that compute drives progress pretty reliably - I am proud to be able to give Ajeya’s report the coveted honor of “I do not make an update of literally zero upon reading it”.

That still leaves the question of “how much of an update do I make?” Also “what are we even doing here?”

I take it that the passages he puts in quotes are being spoken though the Personal voice.

Let’s look at the first one. It’s stated in informal Baysian terms. It also feels arch and indirect. “I do not make an update of literally zero”? What’s that? He’s refrained from writing “0” in a ledger somewhere? Whereas if the report had been less convincing, he’d have

• opened up that ledger,
• added a line,
• placed “Ajeya’s report” in the Argument column, and
• written “0” in the Effect on Me column.

On the contrary, the report has had some non-zero effect on him. But then he attempts to run away: “what are we even doing here?”

A couple paragraphs later: “This report was insufficiently different from what I already believed for me to need to worry about updating from one to the other.” I’m not sure what to make of this. If his prior belief had been quite different from the report’s conclusion, then a decision to stick with that prior belief would represent lack of faith in the report (which he doesn’t feel competent to judge). In contrast, a decision to revise his belief in the direction indicated by the report would represent faith in Cotra (and her colleagues) despite his lack of (technical) qualifications for judging the report. So, he can’t judge the report, doesn’t think it’s wrong, but doesn’t think it’s right enough to lead him to change his mind.

It's as though he’s playing the role of Penelope in Odyssey. She tells her suitors she’ll pick one when she’s done weaving a burial shroud for Laertes. They see her diligently weaving during the day. But then at night, what does she do? She undoes the weaving she’d done during the day so that she can put off the day she has to pick one.

“I’m already scared”

And now, at long last, Alexander gets around to what was obviously on his mind from the very beginning, fear of a rogue AI. He‘s very little that up to this point, which is strange in itself, but now he comes out with it.

Alexander circles back to Yudkowsky: “The more interesting question, then, is whether I should update towards Eliezer’s slightly different distribution, which places more probability mass on earlier decades.” Yudkowsky, however, refuses to give dates: “I consider naming particular years to be a cognitively harmful sort of activity...” Incidentally, sounds like self-regarding grandstanding from Yudkowsky. Perhaps, as people surround him asking for the date, he passes out gilded fortune cookies as souvenirs.

Alexander:

So, should I update from my current distribution towards a black box with “EARLY” scrawled on it?

What would change if I did? I’d get scared? I’m already scared. I’d get even more scared? Seems bad.

That’s not the end. We’ve got two or three more paragraphs. But those paragraphs don’t change the fact that Alexander is scared. They just mix a bit of wit into the contemplation of doom.

Alexander and his community

What are we to make of all this?

I don’t quite know. But then neither does Alexander.

I note, however, that he wrote that post for a community he’s been cultivating for almost a decade. He’s written on a wide variety of topics. He’s conducted surveys, had book review contests, and interacted with that community in various says. That long ambivalent post, spoken through two voices, that’s the post he felt he owed that community. To what extent is Alexander’s ambivalence a reflection of attitudes in that community?

On the one hand there’s the apparent assumption that Transformative AI is on the way come hell and high water. That is coupled with interest in the arcane technical minutiae and leaps of epistemic faith required to issue a long report predicting that arrival by comparison with biological information processing in 1) the human brain, 2) a human life, 3) the evolution of life on earth, and 4) the genome, all measured in FLOPS (floating-point operations per second).[2] That’s one thing.

And there there’s the correlative assumption, perhaps not shared by all, but nonetheless widespread, that the arrival of Transformative AI brings with it the danger that that AI will turn against humanity and transform the earth into a paperclip factory, metaphorically speaking. To the extent that the rhetorical structure of his post is responding to what? a facture, ambivalence? in his audience, why does Alexander hold it in reserve until the end, like it is a shameful secret?

Notes

[1] That depends on what one thinks of the field of AI alignment. Color me skeptical. The idea that we are under not-so-distant threat from an AI hell-bent on our destruction strikes me as conspiracy theorizing directed at technology no one knows how to build.

[2] In his history of technology, David Hays tells us that

... the wheel was used for ritual over many years before it was put to use in war and, still later, work. The motivation for improvement of astronomical instruments in the late Middle Ages was to obtain measurements accurate enough for astrology. Critics wrote that even if the dubious doctrines of astrology were valid, the measurements were not close enough for their predictions to be meaningful. So they set out to make their instruments better, and all kinds of instrumentation followed from this beginning.

I feel a bit like that about the topics of Ajeya Cotra’s report. They are interesting and important in themselves for what they tell us about the world. The need not be yoked to the task of predicting future technology.

Useful summary of statements offered in defense of AI sentience

For those playing along at home, here's a "AI is sentient!" argument bingo card. pic.twitter.com/C4GeB2iMiy

— Emily M. Bender (@emilymbender) June 13, 2022

Is AGI currently the Rome toward which all AI research is headed? [The Alchemical Age]

I came across this blog post first when the graphic about data moats was shared with me: pic.twitter.com/rgu4MyrVAV

— Emily M. Bender (@emilymbender) April 29, 2022

That graphic is from this blog post: All Roads Lead to Rome: The Machine Learning Job Market in 2022, by Eric Jang. From the post:

For instance, Alphabet has so much valuable search engine data capturing human thought and curiosity. Meta records a lot of social intelligence data and personality traits. If they so desired, they could harvest Oculus controller interactions to create trajectories of human behavior, then parlay that knowledge into robotics later on. TikTok has recommendation algorithms that probably understand our subconscious selves better than we understand ourselves. Even random-ass companies like Grammarly and Slack and Riot Games have a unique data moats for human intelligence. Each of these companies could use their business data as a wedge to creating general intelligence, by behavior-cloning human thought and desire itself.

The moat I am personally betting on (by joining Halodi) is a “humanoid robot that is 5 years ahead of what anyone else has”. If your endgame is to build a Foundation Model that train on embodied real-world data, having a real robot that can visit every state and every affordance a human can visit is a tremendous advantage. Halodi has it already, and Tesla is working on theirs. My main priority at Halodi will be initially to train models to solve specific customer problems in mobile manipulation, but also to set the roadmap for AGI: how compressing large amounts of embodied, first-person data from a human-shaped form can give rise to things like general intelligence, theory of mind, and sense of self.

Embodied AI and robotics research has lost some of its luster in recent years, given that large language models can now explain jokes while robots are still doing pick-and-place with unacceptable success rates. But it might be worth taking a contrarian bet that training on the world of bits is not enough, and that Moravec’s Paradox is not a paradox at all, but rather a consequence of us not having solved the “bulk of intelligence”.

Reality has a surprising amount of detail, and I believe that embodied humanoids can be used to index that all that untapped detail into data. Just as web crawlers index the world of bits, humanoid robots will index the world of atoms. If embodiment does end up being a bottleneck for Foundation Models to realize their potential, then humanoid robot companies will stand to win everything.

Yes, reality is (not so) surprisingly detailed, something I talked about in my GPT-3 working paper. I fear that AGI is the Philosopher's Stone of this Alchemical Age of AI that we are still living in.

Is the world coming to an end in 2033? That’s what the (AI) crowd says. [Update: 2029!!]

Not by nuclear fire, or an asteroid collision, or maybe another pandemic caused by a really deadly organism, but by a rogue AI.  There are, however, some people who fear that might be the case. Scott Alexander has just reported that Metaculus, a prediction market, has revised its estimate for the due-date of “Weakly general AI”:

”Weakly general AI” in the question means a single system that can perform a bunch of impressive tasks - passing a “Turing test”, scoring well on the SAT, playing video games, etc. Read the link for the full operationalization, but the short version is that this is advanced stuff AI can’t do yet, but still doesn’t necessarily mean “totally equivalent to humans in any way”, let alone superintelligence.

For the past year or so, this had been drifting around the 2040s. Then last week it plummeted to 2033. I don’t want to exaggerate the importance of this move: it was also on 2033 back in 2020, before drifting up a bit. But this is certainly the sharpest correction in the market’s two year history.

What does that have to do with the end of the world? Well, what if that weakly general AI goes rogue and does the things that rogue AIs do when they take the initiative and maximize some goal that has, as a side effect, the destruction of human life? The paperclip apocalypse is a standard example.

Why was the date revised down? Alexander suggests it was prompted by the announcement of three AI milestones: DALL-E2 (which produces a drawing in response to a verbal prompt), PALM (natural language), and Chinchilla (about scaling of parameters, data, and compute). Very interesting, especially Chinchilla. Me, however, I do not worry about rogue AI. 

But this isn’t about me. Alexander goes on to note:

Early this month on Less Wrong, Eliezer Yudkowsky posted MIRI Announces New Death With Dignity Strategy, where he said that after a career of trying to prevent unfriendly AI, he had become extremely pessimistic, and now expects it to happen in the relatively near-term and probably kill everyone. This caused the Less Wrong community, already pretty dedicated to panicking about AI, to redouble its panic. Although the new announcement doesn’t really say anything about timelines that hasn’t been said before, the emotional framing has hit people a lot harder.

I will admit that I’m one of the people who is kind of panicky. But I also worry about an information cascade: we’re an insular group, and Eliezer is a convincing person. Other communities of AI alignment researchers are more optimistic. I continue to plan to cover the attempts at debate and convergence between optimistic and pessimistic factions, and to try to figure out my own mind on the topic. But for now the most relevant point is that a lot of people who were only medium panicked a few months ago are now very panicked. Is that the kind of thing that moves forecasting tournaments? I don’t know.

Just what does he mean by panic? When the screen of the external monitor for my laptop goes black for no visible reason, I get a little panicky, just a little. The monitor usually came back. Is that the level of panic Alexander’s talking about? There was a time in my life when I couldn’t pay the rent and my landlord invited me to court. That panic was more serious. But we resolved the problem amicably. Is the Alexander’s panic level closer to that? Things never got to the point where the sheriff dropped by to evict me. If that had happened, my panic level would have gone way up as I anticipated the knock on my door. Has Alexander gotten there yet? If so, how does he write blog posts?

* * * * *

From Karen Hao, The messy, secretive reality behind OpenAI’s bid to save the world, Technology Review, February 17, 2020.

Every year, OpenAI’s employees vote on when they believe artificial general intelligence, or AGI, will finally arrive. It’s mostly seen as a fun way to bond, and their estimates differ widely. But in a field that still debates whether human-like autonomous systems are even possible, half the lab bets it is likely to happen within 15 years.

* * * * *

FWIW, I’ve just checked with Metaculus (April 19, 2022 at about 2 PM). The apocalypse has been pushed ahead to November 2032. I don’t know where it will be when you check on it. 

Holy crap! It's now 4:42 PM on the 19th and the apocalypse has moved up to June 14, 2032.

The end is getting closer. As of 5:34 AM, EDT (Eastern Daylight Time) on April 20, it has moved to Feb. 6, 2032. How long will it keep advancing on the present? Would you care to predict just when it will have moved into the past?

5:19 AM, EDT, April 21, the end is drawing still closer, Jan. 30, 2032.

9:33 AM, EDT, June 19, Yikes! We've lost three years, Jan. 9, 2029. 

7:01 AM, EDT, June 22, 2022, Whew! We've picked up some breathing room the last three days. Now it now looks like AI arrives on March 5, 2029. 

10:49 AM, EDT, July 3, 2022, More breathing room. Now it now looks like AI arrives on March 21, 2029.  

10:52 PM, EDT, December 9, 2022, More breathing room. Now it now looks like AI arrives on Oct 30, 2027.