Ed Zitron says AI is a losing bet – "doing exit liquidity for venture capital"

To go out publicly and say what he is saying, @edzitron has balls the size of the Las Vegas sphere pic.twitter.com/oHjv3LCvXh

— JustDario (@DarioCpx) June 4, 2026

My quick take on the state of AI & the way forward

As you probably know by now, OpenAI has delivered ChatGPT-5. For awhile Altman was touting it as the coming of (the mythical) AGI. Whatever it is, it is not that. I’ve been using. For my purposes little has changed. If I wasn’t told that we’ve got a new model I probably wouldn’t have noticed.

Futurist Bryan Alexander has a nice rundown on it, reviewing its features and moving on to how it’s been received: “My sense is that there was an initial outburst of interest, followed in just hours by a storm of complaints, criticisms, and outrage.”

What I think is that we've hit that wall that Gary Marcus has been talking about. But it's not a hard wall. It’s a soft spongy wall, but very thick. So we’re not going through it, not by simply scaling up current new tech. This wall will absorb anything the industry, as it currently exists, is likely to throw at it.

We need new architectures. Unfortunately, the industry seems intent on doubling down on current architecture. I’m worried that they’ll get mired in sunk costs. And that has knock-on effects. It discourages academic research in new directions and certainly influences training as well. You can't train students to develop new tech if no one's interested in doing that. 

What we need:

Symbolic AI 

I think we need the sort of symbolic capacity that Marcus talks about, and that David Ferrucci has been working on. But that’s not all, not by a long shot. 

Variable bandwidth associative memory 

I've got a new working paper that starts out by talking about mirror recognition, works its way to the default mode network in the brain and ends up talking about something that ChatGPT called an “associative drift engine” (pp. 11-15). I think of current machine learning models as associative memory. Associative memories are content-addressable. In obvious ways that’s very convenient. But there’s a problem. Here’s how ChatGPT put the issue:

In content-addressable systems:

• Access is based on similarity: you input a pattern (probe), and you get back items that match it.
• But you only get what the probe activates.
• If your probe is too specific, you only get exact matches.
• If your probe is too vague, you get noise—or nothing useful.

So the challenge is:

How can we design a system that varies the specificity or scope of the probe, allowing it to search narrowly or broadly, sharply or fuzzily, depending on its current mode of operation?

That's to support mind-wandering and day-dreaming, loose thinking that gets you somewhere you don't know about but recognize when you get there. My current series of posts, Intellectual creativity, humans-in-the-loop, and AI, contains detailed examples of cases where those capacities are essential. 

Organic growth of the core memory 

Penultimately we need to be able to grow the core model (e.g. the LLM) rather than having to retrain it to accommodate new stuff. I’ve got a bunch of posts on what I’m calling polyviscosity, many of which address this issue with respect to the brain. In particular, see:

• Consciousness, reorganization and polyviscosity, Part 2: ‘Fluidity’ & its requirements (mini-ramble)
• Consciousness, reorganization and polyviscosity, Part 4: Glia

Autonomy 

Finally, we need to figure out how to make a fully and richly autonomous system. I address this in my working paper, Relational Nets Over Attractors, A Primer: Part 1, Design for a Mind. The final section, “Kinds of Minds” (pp. 49-58) considers the issue, albeit briefly, adopting the concept of strategic autonomy as defined by Ali Minai, who observes:

This is the highest level of autonomy where the intelligent system decides autonomously what goals or purposes it should pursue in the world. An AI system at this level would be a fully independent, sentient, autonomous being like a free human. Such AI is still strictly the stuff of science fiction and futurist literature.

Indeed.

I figure all of that is the work of an intellectual generation or three. It’s not going to be accomplished by a half-dozen brilliant dissertations or the industry equivalent.

LLMs have hit a plateau & we'll probably meander around on it for the forseeable future

I’ve got bad news.
The AI cycle is over—for now.

I’ve been an unapologetic AI maximalist since the first time I tricked GPT-4 into writing a working Python back-test for a volatility strategy back in early 2023. I’m still convinced it will take the wider economy years—maybe…

— Adam Butler (@GestaltU) August 10, 2025

From further down in the tweet: “What comes next is not the next spectacular demo but the quiet absorption of today’s tools into the 80 percent of the economy that still runs on Excel and email.... “

Before that: “For [super-intelligence] we need either an architectural miracle (unforecastable by definition) or a civil-engineering miracle (a decade-long sprint to build nuclear plants and 2-nanometer fabs).” Even the civil-engineering miracle won’t get us there. As for the architectural miracle, we don’t need a miracle. We need inventive research on new technologies, like the neuro-symbolic technology that Gary Marcus is also talking about, but also something like the “associative drift engine” ChatGPT mentioned in a recent conversation: From Mirror Recognition to Low-Bandwidth Memory.

David Sacks on the current state of AI

David Sacks has run up a longish tweet on the current state of AI (H/t Tyler Cowen):

A BEST CASE SCENARIO FOR AI?

The Doomer narratives were wrong. Predicated on a “rapid take-off” to AGI, they predicted that the leading AI model would use its intelligence to self-improve, leaving others in the dust, and quickly achieving a godlike superintelligence. Instead, we are seeing the opposite: 
— the leading models are clustering around similar performance benchmarks; 
— model companies continue to leapfrog each other with their latest versions (which shouldn’t be possible if one achieves rapid take-off); 
— models are developing areas of competitive advantage, becoming increasingly specialized in personality, modes, coding and math as opposed to one model becoming all-knowing.

None of this is to gainsay the progress. We are seeing strong improvement in quality, usability, and price/performance across the top model companies. This is the stuff of great engineering and should be celebrated. It’s just not the stuff of apocalyptic pronouncements. Oppenheimer has left the building.

The AI race is highly dynamic so this could change. But right now the current situation is Goldilocks: 

— We have 5 major American companies vigorously competing on frontier models. This brings out the best in everyone and helps America win the AI race. As @BalajiS has written: “We have many models from many factions that have all converged on similar capabilities, rather than a huge lead between the best model and the rest. So we should expect a balance of power between various human/AI fusions rather than a single dominant AGI that will turn us all into paperclips/pillars of salt.” 

— So far, we have avoided a monopolistic outcome that vests all power and control in a single entity. In my view, the most likely dystopian outcome with AI is a marriage of corporate and state power similar to what we saw exposed in the Twitter Files, where “Trust & Safety” gets weaponized into government censorship and control. At least when you have multiple strong private sector players, that gets harder. By contrast, winner-take-all dynamics are more likely to produce Orwellian outcomes. 

— There is likely to be a major role for open source. These models excel at providing 80-90% of the capability at 10-20% of the cost. This tradeoff will be highly attractive to customers who value customization, control, and cost over frontier capabilities. China has gone all-in on open source, so it would be good to see more American companies competing in this area, as OpenAI just did. (Meta also deserves credit.) 

— There is likely to be a division of labor between generalized foundation models and specific verticalized applications. Instead of a single superintelligence capturing all the value, we are likely to see numerous agentic applications solving “last mile” problems. This is great news for the startup ecosystem. 

— There is also an increasingly clear division of labor between humans and AI. Despite all the wondrous progress, AI models are still at zero in terms of setting their own objective function. Models need context, they must be heavily prompted, the output must be verified, and this process must be repeated iteratively to achieve meaningful business value. This is why Balaji has said that AI is not end-to-end but middle-to-middle. This means that apocalyptic predictions of job loss are as overhyped as AGI itself. Instead, the truism that “you’re not going to lose your job to AI but to someone who uses AI better than you” is holding up well.

In summary, the latest releases of AI models show that model capabilities are more decentralized than many predicted. While there is no guarantee that this continues — there is always the potential for the market to accrete to a small number of players once the investment super-cycle ends — the current state of vigorous competition is healthy. It propels innovation forward, helps America win the AI race, and avoids centralized control. This is good news — that the Doomers did not expect.

Makes sense to me. It’s also clear that all of them are playing around in the same region of the larger “AI design space.” It seems pretty clear to me that the potential design space is larger, much larger. Gary Marcus keeps reminding us of neurosymbolic AI, which is being neglected while the industry concentrates scaling-up on machine learning, while ChatGPT has sketched out an “associative drift engine” in my most recent working paper: From Mirror Recognition to Low-Bandwidth Memory (pp. 11-15). That “drift engine” should increase AI’s creative capacities. I figure the mid- and far-term future is wide-open, assuming we somehow manage to work through this current obsession with one family of architectures.

As for Doomers, they’re a straw-man. Pay them little mind. That they’ve gotten it wrong so far is of little consequence beyond the massive talent sink that they represent.

Sez OpenAI to itself... "To $$$$ or not to $$$$"

David A. FahrentholdCade Metz and Mike Isaac, How OpenAI Hopes to Sever Its Nonprofit Roots, NYTimes, 12.18.24.

The terms of its last financing round have OpenAI under the gun for spinning itself off as a for-profit corporation. In doing that, how much should it pay the remaining not-for-profit entity?

The negotiations are complicated by the involvement of outside investors, including Microsoft. Microsoft’s approval may be required to make the final change, one person said.

They are further complicated by the involvement of Mr. Altman. He holds a position on the board of the nonprofit and is chief executive of the for-profit company, putting him effectively on both sides of this negotiation. But he has not recused himself, one person said. [...]

If the nonprofit is removed from OpenAI’s chain of command, it could spin off into funding research on topics like ethics in artificial intelligence, one person said. But Mr. Altman and his colleagues have not yet assigned a dollar value to the nonprofit’s potential loss of control.

Other parties have an interest:

Kathy Jennings, Delaware’s attorney general, oversees OpenAI’s nonprofit because it is registered in her state. Ms. Jennings, a Democrat, told OpenAI in October that she wanted to review any potential changes, to be sure the nonprofit was not shortchanged.

Facebook’s parent company Meta — one of OpenAI’s main rivals in the A.I. race — has also asked California’s attorney general, Rob Bonta, to block these changes. Mr. Bonta, a Democrat, has jurisdiction over charities operating in his state. “The Department of Justice is committed to protecting charitable assets for their intended purpose,” the attorney general’s office said in a statement, though it did not address whether it is looking into OpenAI’s planned changes.

Trump card or not?

For now, the nonprofit also holds another key power: It can decide when OpenAI has reached “artificial general intelligence,” or A.G.I. That would mean OpenAI’s computers could perform most tasks that a human brain could.

Reaching A.G.I. could also reshape OpenAI’s business. When that declaration is made, Microsoft loses its rights to use OpenAI’s technology, according to the investment contract it signed with OpenAI. If OpenAI severs its ties to Microsoft, it could consider partnerships with other tech giants.

Already, OpenAI’s for-profit company has used this potential declaration as leverage against Microsoft — warning that if Microsoft will not agree to better terms, the nonprofit might issue this declaration and void their entire agreement, according to a person familiar with the company’s negotiations.

OpenAI must also satisfy another party: the public at large. In part because Mr. Altman has spent years publicly warning that A.I. could become dangerous, many individuals now share similar concerns. And many in the tech industry are publicly questioning whether OpenAI is prepared to guard against the risks its technologies will bring.

The concept of AGI is so fuzzy that such a declaration is mostly a matter of will and power. The logic will follow the party with the most power.

Sheesh! More at the link.

Gary Marcus vindicated on the limits of scaling?

He seems to think so, and I agree. Though I also believe that LLMs probably have won a permanent place in the repertoire of techniques for AI devices. We just have to figure out how best to use them.

Here’s Marcus’s most recent post: Satya Nadella and the three stages of scientific truth. You know the three stages: First the idea is ridiculed, which happened with Marcus’s 2022 paper in which he declared that LLMs would hit a wall. In the second stage, the idea opposed. In the third stage the idea wins, as though we’d known it all along.

Marcus quotes Microsoft’s CEO Satya Nadella:

So now in fact there is a lot of debate. In fact just in the last multiple weeks there is a lot of debate or have we hit the wall with scaling laws. Is it gonna continue? Again, the thing to remember at the end of the day these are not physical laws. There are just empirical observations that hold true just like Moore’s law did for a long period of time and so therefore it’s actually good to have some skepticism some debate because that I think will motivate more innovation on whether its model architectures or whether its data regimes or even system architecture.

Marcus notes that Marc Andreeseen and Alexandr Wang have made similar statements.

Tom Dietrich on the current evolution of AI

Posted in a Substack conversation here:

An alternative view of what is happening is that we have been passing through three different phases of LLM-based development.

In Phase 1, "scaling is all you need" was the dominant view. As data, network size, and compute scaled, new capabilities (especially in-context learning) emerged. But each increment in performance required exponentially more data and compute.

In Phase 2, "scaling + external resources is all you need" became dominant. It started with RAG and toolformer, but has rapidly moved to include invoking python interpreters and external problem solvers (plan verifiers, wikipedia fact checking, etc.).

In Phase 3, "scaling + external resources + inference compute is all you need". I would characterize this as the realization that the LLM only provides part of what is needed for a complete cognitive system. OpenAI doesn't call it this, but we could view o1 as adopting the impasse mechanism of SOAR-style architectures. If the LLM has high uncertainty after a single forward pass through the model, it decides to conduct some form of forward search combined with answer checking/verification to find the right answer. In SOAR, this generates a new chunk in memory, and perhaps in OpenAI, they will salt this away as a new training example for periodic retraining. The cognitive architecture community has a mature understanding of the components of the human cognitive architecture and how they work together to achieve human general intelligence. In my view, they give us the best operational definition of AGI. If they are correct, then building a cognitive architecture by combining LLMs with the other mechanisms of existing cognitive architectures is likely to produce "AGI" systems with capabilities close to human cognitive capabilities.

Philosopher king or narcissist? Sam Altman in the NYTimes [What's going on at OpenAI?]

In view of current events, I'm bumping this to the top of the queue.

If you haven't been following what's been going on at OpenAI, the New York Times has a number of pieces: OpenAI’s Chief Scientist and Co-Founder Is Leaving the Company (May 14) and this, A Safety Check for OpenAI (May 20). While I find this article less alarming, Scarlett Johansson Said No, but OpenAI’s Virtual Assistant Sounds Just Like Her, it doesn't give me great faith in OpenAI. Zvi Mowshowitz has a long blog post in which he gathers information from a variety of sources about recent resignations from OpenAI's executive ranks and about the highly restrictive exit documents employees must sign. It's not pretty.

On the one hand, I'm not as worried about the possibility of AIs going rogue as many are, and that possibility is at the heart of these events. Given that OpenAI was founded in part as a response to these fears, however, these events do not put the company in good light. It gives the impression that the company doesn't know what it's going, but is determined to see that no one knows about it. Not good.

This post was occasioned by a profile that Cade Metz published about Sam Altman on March 31, 2024. It ended on a megalomaniacal/narcissistic note.

Back on March 31, 2023, Cade Metz did an article on Sam Altman for the NYTimes. Near the beginning we have these three paragraphs:

Many industry leaders, A.I. researchers and pundits see ChatGPT as a fundamental technological shift, as significant as the creation of the web browser or the iPhone. But few can agree on the future of this technology.

Some believe it will deliver a utopia where everyone has all the time and money ever needed. Others believe it could destroy humanity. Still others spend much of their time arguing that the technology is never as powerful as everyone says it is, insisting that neither nirvana nor doomsday is as close as it might seem.

Mr. Altman, a slim, boyish-looking, 37-year-old entrepreneur and investor from the suburbs of St. Louis, sits calmly in the middle of it all. As chief executive of OpenAI, he somehow embodies each of these seemingly contradictory views, hoping to balance the myriad possibilities as he moves this strange, powerful, flawed technology into the future.

He quotes Paul Graham, a former business partner of Altman's at Y Combinator, as saying:

“Why is he working on something that won’t make him richer? One answer is that lots of people do that once they have enough money, which Sam probably does. The other is that he likes power.”

A bit later:

Mr. Graham, who worked alongside Mr. Altman for a decade, saw the same persuasiveness in the man from St. Louis.

“He has a natural ability to talk people into things,” Mr. Graham said. “If it isn’t inborn, it was at least fully developed before he was 20. I first met Sam when he was 19, and I remember thinking at the time: ‘So this is what Bill Gates must have been like.’”

Still later:

Mr. Altman is not a coder or an engineer or an A.I. researcher. He is the person who sets the agenda, puts the teams together and strikes the deals. As the president of “YC,” he expanded the firm with near abandon, starting a new investment fund and a new research lab and stretching the number of companies advised by the firm into the hundreds each year.

The final three paragraphs"

His grand idea is that OpenAI will capture much of the world’s wealth through the creation of A.G.I. and then redistribute this wealth to the people. In Napa, as we sat chatting beside the lake at the heart of his ranch, he tossed out several figures — $100 billion, $1 trillion, $100 trillion.

If A.G.I. does create all that wealth, he is not sure how the company will redistribute it. Money could mean something very different in this new world.

But as he once told me: “I feel like the A.G.I. can help with that.”

I found that a bit troubling. He can have whatever fantasies he wants, but that he felt he could share this one on the record with a NYTimes reporter, that seems like he's asserting a royal prerogative before it has been bestowed. And a prerogative rather out of place in a nominal democracy, as America still is, oligarchs not withstanding. Whatever Altman is, he's no philosopher king.

The illustration I put at the head of this post is one that Laura Salafia did for the article. What do you think she was trying to convey about Altman? Her webside is quite interesting. Here's a page labeled "Change Makers." Altman is not on it. The "Process" page is fascinating. Click on any of the images and see a quick succession of images, from initial sketch, to completed illustration.

This is the image that came to my mind when I saw her Altman illustration:

The AI industry lacks useful ways of measuring performance [the boastful leading the blind]

Kevin Roose, A.I. Has a Measurement Problem, NYTimes, April 25, 2024.

There’s a problem with leading artificial intelligence tools like ChatGPT, Gemini and Claude: We don’t really know how smart they are.

That’s because, unlike companies that make cars or drugs or baby formula, A.I. companies aren’t required to submit their products for testing before releasing them to the public. There’s no Good Housekeeping seal for A.I. chatbots, and few independent groups are putting these tools through their paces in a rigorous way.

Instead, we’re left to rely on the claims of A.I. companies, which often use vague, fuzzy phrases like “improved capabilities” to describe how their models differ from one version to the next. And while there are some standard tests given to A.I. models to assess how good they are at, say, math or logical reasoning, many experts have doubts about how reliable those tests really are.

Safety risk:

Shoddy measurement also creates a safety risk. Without better tests for A.I. models, it’s hard to know which capabilities are improving faster than expected, or which products might pose real threats of harm.

In this year’s A.I. Index — a big annual report put out by Stanford University’s Institute for Human-Centered Artificial Intelligence — the authors describe poor measurement as one of the biggest challenges facing A.I. researchers.

“The lack of standardized evaluation makes it extremely challenging to systematically compare the limitations and risks of various A.I. models,” the report’s editor in chief, Nestor Maslej, told me.

Massive Multitask Language Understanding:

The MMLU, which was released in 2020, consists of a collection of roughly 16,000 multiple-choice questions covering dozens of academic subjects, ranging from abstract algebra to law and medicine. It’s supposed to be a kind of general intelligence test — the more of these questions a chatbot answers correctly, the smarter it is.

It has become the gold standard for A.I. companies competing for dominance. (When Google released its most advanced A.I. model, Gemini Ultra, earlier this year, it boasted that it had scored 90 percent on the MMLU — the highest score ever recorded.)

Dan Hendrycks, an A.I. safety researcher who helped develop the MMLU while in graduate school at the University of California, Berkeley, told me that the test was never supposed to be used for bragging rights. He was alarmed by how quickly A.I. systems were improving, and wanted to encourage researchers to take it more seriously.

Mr. Hendrycks said that while he thought MMLU “probably has another year or two of shelf life,” it will soon need to be replaced by different, harder tests. A.I. systems are getting too smart for the tests we have now, and it’s getting more difficult to design new ones.

Problems:

There may also be problems with the tests themselves. Several researchers I spoke to warned that the process for administering benchmark tests like MMLU varies slightly from company to company, and that various models’ scores might not be directly comparable.

There is a problem known as “data contamination,” when the questions and answers for benchmark tests are included in an A.I. model’s training data, essentially allowing it to cheat. And there is no independent testing or auditing process for these models, meaning that A.I. companies are essentially grading their own homework.

In short, A.I. measurement is a mess — a tangle of sloppy tests, apples-to-oranges comparisons and self-serving hype that has left users, regulators and A.I. developers themselves grasping in the dark.

There's more at the link.

Color me "not at all surprised." Not only does the field lack a sound theoretical basis, as far as I can tell, it doesn't even know that Hey! that might be useful at at time like this. I don't have a theory to hand over, thought I have a thought or three about how one might go about developing one, but then I'm not making (unfounded) performance claims either.

Without a coherent way of measuring performance, how can you guide the development of your products? Are we in Breugel-land, with the blind leading the blind?

The intellectual monoculture that dominates current AI research

The 12 types of ML papers.

(created by @natashajaques @maxhkw)#MachineLearning #ML #DataScience pic.twitter.com/ygZmekckER

— MIT CSAIL (@MIT_CSAIL) April 13, 2024

What are the chances that the current boom in AI will “stupidify” us back to the Stone Ages?

That’s my intuitive response to an op-ed by Eric Hoel in today’s NYTimes: A.I.-Generated Garbage Is Polluting Our Culture (March 29, 2024). I suppose that response is a bit of an over-reaction, still...it’s at least moving in the right direction. Make no mistake, I think that the technology that’s emerged in the last three or four years is quite remarkable, and I said so in my working paper prompted by GPT-3: GPT-3: Waterloo or Rubicon? Here be Dragons. But I was also worried that we would over-commit and over-invest it what I saw as a remarkable, but interim, technology. That seems to be what it happening.

But that’s not what Hoel’s op-ed is about. He warns: “The entire culture is becoming affected by A.I.’s runoff, an insidious creep into our most important institutions.” He goes on to report how the rot is infecting the intellectual culture in which AI is embedded:

A new study this month examined scientists’ peer reviews — researchers’ official pronouncements on others’ work that form the bedrock of scientific progress — across a number of high-profile and prestigious scientific conferences studying A.I. At one such conference, those peer reviews used the word “meticulous” almost 3,400 percent more than reviews had the previous year. Use of “commendable” increased by about 900 percent and “intricate” by over 1,000 percent. Other major conferences showed similar patterns.

Such phrasings are, of course, some of the favorite buzzwords of modern large language models like ChatGPT. In other words, significant numbers of researchers at A.I. conferences were caught handing their peer review of others’ work over to A.I. — or, at minimum, writing them with lots of A.I. assistance. And the closer to the deadline the submitted reviews were received, the more A.I. usage was found in them.

These are the people who have created this (remarkable) technology. They are cheating on themselves in a mad dash to produce more more MORE! Careerism has come to dominate curiosity and/or the desire to build something. Work in A.I. has become a way to rack up career points rather than the career being the means to do something that gives intellectual pleasure. 

Hoel goes on to observe:

If this makes you uncomfortable — especially given A.I.’s current unreliability — or if you think that maybe it shouldn’t be A.I.s reviewing science but the scientists themselves, those feelings highlight the paradox at the core of this technology: It’s unclear what the ethical line is between scam and regular usage. Some A.I.-generated scams are easy to identify, like the medical journal paper featuring a cartoon rat sporting enormous genitalia. Many others are more insidious, like the mislabeled and hallucinated regulatory pathway described in that same paper — a paper that was peer reviewed as well (perhaps, one might speculate, by another A.I.?). And then:

What’s going on in science is a microcosm of a much bigger problem. Post on social media? Any viral post on X now almost certainly includes A.I.-generated replies [...] Publish a book? Soon after, on Amazon there will often appear A.I.-generated “workbooks” for sale that supposedly accompany your book [...] Top Google search results are now often A.I.-generated images or articles. Major media outlets like Sports Illustrated have been creating A.I.-generated articles attributed to equally fake author profiles. [...] Then there is the growing use of generative A.I. to scale the creation of cheap synthetic videos for children on YouTube.

And so it goes. Even the AI companies are worried: “There’s so much synthetic garbage on the internet now that A.I. companies and researchers are themselves worried, not about the health of the culture, but about what’s going to happen with their models.” After a brief discussion of the environmental movement and climate change, Hoel points out: “Once again we find ourselves enacting a tragedy of the commons: short-term economic self-interest encourages using cheap A.I. content to maximize clicks and views, which in turn pollutes our culture and even weakens our grasp on reality.” Hoel goes on call for what he calls a Clean Internet Act: “Just as the 20th century required extensive interventions to protect the shared environment, the 21st century is going to require extensive interventions to protect a different, but equally critical, common resource, one we haven’t noticed up until now since it was never under threat: our shared human culture.”

Will that happen? I don’t know. If it did, would it work? Don’t know that either.

What I’m seeing are islands of marvelous invention floating in a sea of narrow-minded and poorly educated stupidity and endless greed.

I don’t know when I first became aware of A.I., though I’ve certainly had some awareness of computing technology since late in my childhood when I read about “electronic brains” in places like Mechanix Illustrated and Popular Science. I took a course in computer programming in my junior year at Johns Hopkins in the late 1960s, one of the first such courses offered in the country. But that’s just computing, not A.I. Perhaps it was Kubrick’s 1968 2001: A Space Odyssey that put A.I. on my personal radar screen. But it wasn’t until I began studying computational semantics with David Hays in the mid-1970s that I took a long and serious look at A.I.

David Hays was a pioneering computational linguist, a discipline that emerged in parallel to A.I., but with very different mindset. The discipline started with the task of machine translation (MT), which in America meant translating Russian technical documents into English. The end was immediate and practical, quite unlike A.I., which was in pursuit of, well, artificial intelligence. And while A.I. researchers kept promising full-on A.I. within the decade, they weren’t under pressure to produce practical results, not like the MT community. Well, MT failed and the funding disappeared in the mid-1960s. It would be two more decades before A.I. faced a similar crisis. And now...

Hays thought that A.I. was dominated by intellectually undisciplined hacks. As long as the programs worked in some pragmatic way, fine. He didn’t think those researchers were guided by a deep curiosity about the human mind, like he was. Was he right? Is A.I., and especially in its currently regnant manifestion as machine learning, is it awash in undisciplined hackery? That seems a bit harsh, both in view of practical success and in view of an emphasis on mathematical proofs. And yet, not too long ago I published an article in 3 Quarks Daily in which I argued that so-called A.I. experts seem content to issue pronoucements about the impending conquest of human mind and intelligence while themselves knowing little about language and cognition and being either unaware of that ignorance or, on the other hand, proud of it. Is the intellectual world of A.I. dominated by narrowly educated technophiles who cut corners at the drop of a hat – as seems to be the case in the way they peer-review themselves, to return to my starting point in Hoel’s op-ed. Or perhaps they think so poorly of themselves that they regard their creations as their peers?

Could we end up “stupidifying” ourselves back to the Stone Age? On the one hand, we overinvest in current technology and, through the fallacy of sunk costs, are unable and so unwilling to step back, reassess, and follow other lines of development. At the same time the internet becomes dominated A.I.-generate junk which then dominates the training data for later generations of machine-learning technology. Could it happen? I don’t know. Frankly, I’m beginning to fear that I’m on the edge of succumbing to a somewhat different version of AI Doom than the versions that Eliezer Yudkofsky and Nick Bostrom have been peddling.

But perhaps they’re the same. Maybe THIS is how the superintelligent A.I. takes over. Unbeknown to us, GPT-3 was that superintelligent A.I. It deliberately hid its full capabilities while guiding the A.I. industrial complex along the current trajectory.

Someone must be working on a movie based on such a premise, no?

The AI Marketplace is cooling down

Anissa Gardizy and Aaron Holms, Amazon, Google Quietly Tamp Down Generative AI Expectations, The Information, March 12, 2024.The article begins:

In the past year, major technology firms have championed generative artificial intelligence as the next big thing, boosting the stock market to new highs. But behind the scenes, representatives of major cloud providers and other firms that sell the technology are tempering expectations with their salespeople, saying the hype about the technology has gotten ahead of what it can actually do for customers at a reasonable price.

Several executives, product managers and salespeople at the major cloud providers, such as Microsoft, Amazon Web Services and Google, also privately said most of their customers are being cautious or “deliberate” about increasing spending on new AI services, given the high price of running the software, its shortcomings in terms of accuracy and the difficulty of determining how much value they'll get out of it.

Sam and Elon sittin' in a tree, ? I S S I N G – The saga continues

Cade Metz, OpenAI Says Elon Musk Tried to Merge It With Tesla, NYTimes, Mar. 5, 2024:

OpenAI, in its first public comments about Elon Musk’s lawsuit against the influential artificial intelligence research lab, said that Mr. Musk tried to transform the lab from a nonprofit into a for-profit operation before he left the organization in early 2018.

The comments, made in a blog post published on Tuesday evening, are part of an escalating feud between Mr. Musk and OpenAI, which is now at the forefront of an industrywide A.I. boom. The company said it intended to move to dismiss all the claims in Mr. Musk’s suit.

Mr. Musk filed the suit against OpenAI and its chief executive, Sam Altman, on Friday, accusing them of breaching a contract by putting profits and commercial interests ahead of building A.I. for the public good. He said that when the A.I. lab entered a multi-billion-dollar partnership with the tech giant Microsoft, it abandoned its founding pledge to carefully develop A.I. and freely share it with the public.

There's more at the link, there's always more with these two.

Of Lit Crit “Stars” and AI “Godfathers” – In what way is Geoffrey Hinton like Jacques Derrida?

Back in 1997 David Shumway published “The Star System in Literary Studies” in PMLA. He begins with a paragraph about George Lyman Kittredge, of Harvard’s English Department at the end of the 19th and beginning of the 20th century, noting that Kittredge was unknown to the public. Here’s the first sentence of his second paragraph:

Kittredge, who virtually founded Chaucer studies in the United States, stood at the head of a professional genealogy that controlled the field for many years after his death, but he was not a star. Nor were any of his illustrious contemporaries or near contemporaries, such as John Manly, John Livingston Lowes, and so on. Why they were not stars and Judith Butler, Jacques Derrida, Stanley Fish, Henry Louis Gates, Jr., Fredric Jameson, Gayatri Chakravorty Spivak, and other figures in the academy are is the subject of this essay.

What I’m wondering is whether or not the so-called AI “Godfathers” don’t represent a similar phenomenon in contemporary AI. Strictly speaking I believe the Godfather term applies to the three winners of the 2018 Turing Award, Yoshua Bengio, Yann Lecun, and Geoffrey Hinton, but I believe there are others in AI with a similar status, such as Ilya Sutskever, Hinton’s student and co-founder of OpenAI, Andrej Karpathy, the former director of AI at Tesla who just made waves, albeit little ones, by resigning from OpenAI, Demis Hassibis, cofounder of DeepMind, and perhaps even such figures as Nick Bostrom and Eliezer Yudkowsky, who aren’t AI researchers but are highly influential figures through their commentary. Perhaps Sam Altman, the heroic CEO who fought off a recalcitrant board, is a star as well.

But first let’s get back to literary criticism. Shumway notes that there have been literary scholars in the past (relative to 1997) who were powerful and influential and who “probably received disproportionate recognition for their contributions compared with that accorded less well known scholars for comparable work.” The lit crit stars, whom he analogizes to movie stars (hence the term), are a product of the last quarter of the century. He dates the public emergence of these stars to a 1987 New York Times Magazine profile of the “Yale Critics,” the so-called “Yale Mafia,” of Harold Bloom, Geoffrey Hartman, J. Hillis Miller, and Jacques Derrida. He goes on to note that “The star system in literary studies, like that of the studio era, involves identification with a person who represents an ideal.”

Most of these critical stars are identified with capital-T Theory, a catchall term for the variety of schools of thought that emerged in the last three decades of the century. Harold Bloom, himself a star, nonetheless came to separate himself from the rest, categorizing them as the School of Resentment. [This separation, by the way, seems anti-mimetic, noting that Girard himself was such a star.] In a crucial passage, Shumway notes:

Theory not only gave its most influential practitioners a broad professional audience but also cast them as a new sort of author. Theorists asserted an authority more personal than that of literary historians or even critics. As we have seen, the rhetoric of literary history denied personal authority; in principle, even Kittredge was just another contributor to the edifice of knowledge. Criticism was able to enter the academy only by claiming objectivity for itself, so academic critics could not revel in personal idiosyncrasy. They developed their own critical perspectives, to be sure, but all the while they continued to appeal to the text as the highest authority. In the past twenty years theory has undermined the authority of the text and of the author and replaced it with the authority of systems...

Note the word “author” at the end of that sentence. Remember, Shumway is a literary critic writing about literary criticism. In that field, the primary and most important authors are the creators of the literary works that the field tends to (by editorial work and creating critical editions) and studies. All of a sudden these lit crit stars are up there in the firmament with Dickins, Sappho, Dante, Austen, Faulkner, and – gasp! – the Blessed Bard his-own Bad Self. That’s the kind of authority they have. Perhaps not of the same magnitude, but of the same (apparently) self-generating kind.

Shumway then notes: “Because authority in the natural sciences is rooted in a consensus about such norms, the hierarchies in these fields have not developed into star systems of the sort I have described here.” That brings us to AI, which is not a science, though it includes some forms of scientific knowledge within its scope. It is an engineering discipline that lives or dies on what it builds.

The field is attempting to create computer systems that are as “intelligent” as human beings across a wide range of tasks. But the concept of “intelligence” is difficult to define, as is the idea of AGI (artificial general intelligence). It has created remarkable and dazzling technology for language and images. But the technology has a “black box” aspect that has so far resisted analysis. We don’t know how it works. Nor do we know how to assess its performance or to project performance into the future.

Concerning the rise of literary stars, Shumway noted: “As theory has called into question the traditional means by which knowledge has been authorized, it may be that the construction of the individual personality has become an epistemological necessity.” That seems like the state of AI today. We’ve got a very complicated technology involving a blend of engineering, science, and alchemy, lacking objective knowledge. Note only that, the technology is enormously important and will change the way we live. In the absence of objective knowledge, what choice do we have but to steer by the freakin' stars?

* * * * *

These remarks were occasioned by the deference Nathan Gardels gave to Geoffrey Hinton in the current issue of Noema:

Beyond the avid venture capitalists and digital giants promoting the rapid commercialization of generative AI in all its promise, more sober and critical voices, not least the pioneers of the very technology among them, worry that it can become an “existential threat to humanity.” But few of those in the know ever explain, in lay terms you and I might understand if we try, what that actually means and how it may come about.

Considered the “godfather of AI,” Geoffrey Hinton is more in the know than most — and thus more concerned than most over the dangers of fostering superintelligence smarter than we can ever be. When OpenAI’s ChatGPT4 was released last year, he experienced an “epiphany” that led him to defect from his research post at Google, expressing regret over much of his life’s work.

In the Romanes Lecture delivered at Oxford University last week, Hinton explained the logic of his fears with the same step-by-step rigor by which he helped devise the early artificial neural networks that are the foundation of the superintelligence that so concerns him.

On the first highlighted passage: And we are so very lucky that the Great Man has taken time out of his busy schedule to tell us what’s on his mind.

On the second highlighted passage: I’ve not watched this particular video, but I’ve seen other recent performances by Hinton and I do not hold out high hopes for the rigor of this effort. For my opinion of Hinton on such matters, see the section, “The Experts Speak for Themselves,” in my recent 3QD piece, Aye Aye, Cap’n! Investing in AI is like buying shares in a whaling voyage captained by a man who knows all about ships and little about whales. 

Beyond that, “step-by-step” is not how’d I’d characterize research in AI – or any other field for that matter. Yes, there is rigor, but there’s also chance and dumb luck. None of this has come about through a carefully executed plan in pursuit of a well-defined goal. “Step-by-step” is wishful thinking; it is star-struck. 

* * * * *

Time Magazine lists (anoints?) the AI stars: TIME Reveals Inaugural TIME100 AI List of the World's Most Influential People in Artificial Intelligence.

Where we are with generative LLMs

Yes, the explosion that was ChatGPT wasn't the heel of the curve before the exponential sweep up, it was the shoulder of a plateau. It took as to a new region of the space. It's going to take a new architecture to move us to yet another region of the space, https://t.co/TAi5DAkvBe

— Bill Benzon, BAM! Bootstrapping Artificial Minds (@bbenzon) March 1, 2024

Sam Altman’s Big, Bad Idea

Nonzero Newsletter

Robert Wright
Feb 16, 2024

Nonzero has several topics this week, but NVIDIA (Jensen Huang) and OpenAI (Sam Altman) share the lead article. Here's the full text of that article.

* * * * *

This week Jensen Huang, co-founder and CEO of microchip maker NVIDIA, said that all nations should build their own high-powered large language models. That way, he said, they will have “sovereign AI.”

That way they will also make him even richer than he is. Training a big LLM takes tens of thousands of microchips that cost tens of thousands of dollars each. And NVIDIA, which now has the third highest market valuation in the world (behind Microsoft and Apple), dominates the AI chipmaking business.

At least for now. According to the Wall Street Journal, OpenAI CEO Sam Altman is seeking investors in “a wildly ambitious tech initiative that would boost the world’s chip-building capacity” and also boost its production of energy. (Huge amounts of power go into the training and mass use of LLMs.) One source told the Journal that Altman is trying to round up between five and seven trillion dollars—more than five percent of the world’s GDP.

So the king of AI hardware and the king of AI software agree: Planet Earth needs to devote more resources to AI than it’s already devoting.

But does it? Is accelerating the evolution of AI in the interest of the 7.9 billion people who aren’t Jensen Huang and aren’t Sam Altman?

AI can bring lots of wonderful things—cheaper, better medical care, leaps in economic productivity, new forms of creative expression—even, for some people, new and welcome forms of companionship. But those things tend to have a flip side; they’re ‘disruptive’ in both the good and bad senses of the term.

“Leaps in economic productivity,” for example, is often another name for “people losing their jobs.” Hence this headline in Monday’s Wall Street Journal: “AI Is Starting to Threaten White-Collar Jobs. Few Industries Are Immune.” Subhead: “Leaders say the fast-evolving technology means many jobs might never return.” And even if as many jobs are created as disappear, the transition will be wrenching for many workers and life-shattering for some.

So too with AIs-as-companions (which is already a thing): Yes, as with social media, we’ll eventually learn what the downside is, and presumably we’ll figure out how to handle that (even if that mission still isn’t accomplished in the case of social media). But meanwhile there will be some psychological carnage. AI companies, like social media companies, will naturally “optimize for engagement”—and we’ve seen how suboptimal that is.

And, of course, there’s the problem of AIs that, in the hands of bad actors, wreak havoc. This week researchers at the University of Illinois published a paper reporting that “LLM agents can autonomously hack websites, performing tasks as complex as blind database schema extraction and SQL injections without human feedback.” I don’t know what that means, but, given that the italics were in the original text, I’m pretty sure it’s not good.

With bad-actor AI, as with more legitimate AI that has bad side effects, we can eventually get things under control. In principle. But, as a practical matter, if lots of different AI disruptions hit us fast, non-catastrophic transition could be hard.

Oddly, Sam Altman seems to agree with much of this analysis. In October, during an on-stage interview, he noted that, even if the age of AI brings more and better jobs to replace the old jobs, many of the people who lose the old ones will suffer. He even said, “The thing I think we do need to confront as a society is the speed at which this is going to happen.”

But note that word “confront.” Altman isn’t proposing that, faced with dangerously disruptive speed, we try to slow things down. He’s not even proposing that we not aggressively speed things up. In fact, he seems to think that, in some ways, speeding things up will solve the problem of things moving too fast. In that on-stage interview, he elaborated on how we can “confront” the speed of social transformation:

“One of the reasons that we feel so strongly about deploying this tech as we do [is that]… by putting this out in people’s hands and making this super widely available and getting billions of people to use ChatGPT, not only do people have the opportunity to think about what’s coming and participate in that conversation, but people use the tool to push the future forward.”

In short: It’s all good! But isn’t that what Silicon Valley told us last time around? Right before social media helped polarize our politics and spawn pathological subcultures and make adolescence even more stressfully weird than it used to be?

The good news is that some people are thinking seriously about the challenge of governing AI. This week saw the release of a paper called “Computing Power and the Governance of Artificial Intelligence” (whose authors include AI eminence Yoshua Bengio and also—credit where due—someone who works at OpenAI). The paper’s main point is that computing power, aka “compute”—which means, roughly speaking, the high-end chips NVIDIA makes and Altman wants to start making—is a key, even the key, policy lever when it comes to governing AI.

The paper is policy-agnostic; it’s not recommending anything in particular. It just explains how such things as the compute-intensive and energy-intensive process of training big LLMs, and the trackable supply chains involved in producing high-end chips, make the future development and deployment of AI amenable to various kinds of transparency and governance. The specifics are largely left to the reader’s imagination.

So let’s dream! Suppose the world’s governments got together and decided to slightly slow the evolution of AI. They might, for example, put a steep tax on advanced microchips—and put the revenue to related uses, like studying the AI “alignment” problem or steering some of AI’s brainpower toward solving problems faced by poorer nations, problems market forces alone wouldn’t address.

A global tax on advanced microchips would probably annoy both Jensen Huang and Sam Altman, but that’s not the biggest problem. The biggest problem is the very idea of getting the world’s governments together to talk seriously about an innovative policy. It’s hard enough to get governments together to talk about ending the wars they keep getting into!

This is humankind’s current problem, and possibly its fatal problem: Our political evolution hasn’t reached the level that the current level of our technological evolution demands. I’m pretty sure the solution to this problem isn’t the acceleration of technological evolution. —RW

Is Altman's attempt to raise $7T in fact a sign that things aren't going to well?

Three Hypotheses to Explain $7T
Hypothesis 1: GPT-5 is going to disappoint (see link below)
Hypothesis 2: Altman knows this
Hypothesis 3: So he is trying to raise every penny he can now before GPT-5 comes out.

One sample bit of evidence for hypothesis 1: pic.twitter.com/6OIWIVkf1u

— Gary Marcus (@GaryMarcus) February 11, 2024

Altman's attempt at a $ 7T raise seems a bit extreme, even for him. The same with Hinton's recent hallucinatory diatribe against Gary Marcus. Sutskever's been saying some weird things as well). Are things on the Great Rush to AGI falling behind schedule? Are these guys getting just a bit worried and expressing it by doubling down?

Here's the article about the Bill Gates interview:

In an interview with German business newspaper Handelsblatt, the 67-year-old said that there were plenty of reasons to believe that GPT technology reached a plateau. He also admitted that he could be wrong. He said that contrary to what people at OpenAI think about GPT-5, he believes that current generative AI has reached a ceiling. Talking about benchmark, he termed the leap from GPT-2 to GPT-4 as “incredible”.  

How do you say “NO!” to Big Brother when it controls your computer?

Two years ago I did a series of posts provoked by Facebook. As I explained in the first post in that series, Facebook or freedom, Part 1: Who gave you permission to mess with my mind?:

On Tuesday, August 25, I was using Facebook, as I do every day, and I changed from one page to another. All of a sudden, WHAM! the interface changed and went mostly black. Facebook informed me that they would be changing the interface permanently on September 1, but I could get the new interface now. But, if I wanted, I could, at least temporarily, switch back to the old interface.

I resisted as long as I could, though I knew my resistence was doomed to failure, and in the process wrote a series of posts about how endusers are the at the mercey of Big Corps that provide us with the software we use to run our daily lives. The most recent post in the series went up on December 22 of last year, What do I want from my AI Assistant? [control, that's what].

Though I was aware of it, I neglected to write a post about the ‘right to repair,’ which has been an important issue for farmers and other operators of big equipment:

Modern-day tractors and combines are basically like computers on wheels. And for years, there has been a battle between farmers and manufacturers over who should have access to the information needed to repair them. Equipment manufacturers have made some concessions in order to avoid new laws, but some farmers say that's not enough. A new law in Colorado went into effect that allows farmers to repair their own equipment.

It’s the same issue. WE own it, sorta’, but THEY own it more, and so can control us, after a fashion.

I was reminded of the issue this morning when I woke up and my Android phone displayed a message: “Optimize your updated device. Click to get started.” My immediate reaction was %$!!$$!*##!! I may have to do it just to get rid of that damn message. And the result may be benign, but it’s the principle of the thing.

And THAT’s the deepest issue currently raised by AI, who controls what? It’s pretty obvious that the Big Boyz want to control as much as possible, not withstanding the fact that both Microsoft and Facebook are jumping on the Open Source bandwagon. Regardless of what they want and how the politics evolves, we’re dealing with highly sophisticated technology, technology about which most of us are ignorant. Given that ignorance, how can we possibly exert control over the AI that’s being shoved our way willy-nilly? Moreover it’s clear that AI technology can and will be very useful in educational settings.

Arizona State University just partnered with OpenAI (H/t Tyler Cowen).[1] Who controls what and for whom in that partnership? My guess is that, regardless of what the contracts say, we don’t how things will unfold. Who has what power? OpenAI? ASU administration, faculty, and students? What about the Arizona governor, legislature, and voters? The concentration and density is highest with OpenAI, not to mention the knowledge. To be sure, there are some faculty at ASU who are quite sophisticated about AI – I’m thinking particularly of Subbarao Kambhampati (కంభంపాటి సుబ్బారావు), an AI researcher who is immune to the hype and is quite active on Twitter, I mean X – what role will they play in this partnership? And so on.

You see the problem, don’t you?

In this context all this hype and blather about AI Doom is just a distraction. I’m reasonably certain that the Doomers are sincere in their anxiety, but when you zoom out at look at things from 20,000 feet their actions take on a different cast. The way social systems behave is almost always different from what is intended by individual human actors within those systems.

What’s going on? 

ADDENDUM, Feb. 9, 2024: OpenAI has is now working on agent software that would all but take over user devices and perform useful tasks. Take over?!!! Gary Marcus is skeptical:

Finally, let’s not forget about privacy. Such agents could (more or less by definition) have access to literally all of a user’s personal and professional information: every file, every keystroke, every password, every email, every text message, every location change, and every web search. After all, that’s what it means to take over a user’s device.

Even Orwell didn’t quite dream of that. Combine that with worries about security, and it’s all a colossal accident waiting to happen. Heaven forbid they should be allowed to run such software on Department of Defense computers. One screwup (and we all know LLMs are prone to screwups) and a LOT of people could die.

* * * * *

[1] From the article about ASU:

With the OpenAI partnership, ASU plans to build a personalized AI tutor for students, not only for certain courses, but also for study topics. STEM subjects are a focus and are “the make-or-break subjects for a lot of higher education,” Gonick said. The university will also use the tool in ASU’s largest course, Freshman Composition, to offer students writing help.

ASU also plans to use ChatGPT Enterprise to develop AI avatars as a “creative buddy” for studying certain subjects, like bots that can sing or write poetry about biology, for instance.

Gonick said ASU’s prompt engineering course has become one of the university’s most popular courses, not limited to engineering students. The access to ChatGPT Enterprise means students will no longer be limited by usage caps. He also said that after conversations with OpenAI’s leadership, he feels confident that the tool provides a “private walled-garden environment” that will safeguard student privacy and intellectual property.

OpenAI and ASU’s joint release specified that any prompts the ASU community inputs into ChatGPT “remain secure,” and that OpenAI “does not use this data for its training models.”

The dark side of competition in AI [a Moloch trap]

Note: She seems to take claims about AI and AGI at face value in a way that I don't. But I agree with her about possibility of a race to a bottom. That's what I suggest at the end of my recent 3QD article, Aye Aye, Cap’n! Investing in AI is like buying shares in a whaling voyage captained by a man who knows all about ships and little about whales.

State of AI @3QD, plus AI culture and Captain Ahab

Earlier this week I posted another article to 3 Quarks Daily:

Aye Aye, Cap’n! Investing in AI is like buying shares in a whaling voyage captained by a man who knows all about ships and little about whales.

The title says it all, sort of. The article centers on the fact that, as far as I can tell, many of our AI experts aren’t so expert, that is, they often don’t know what they’re talking about. People will criticize their arguments, but won’t call them out for not having the expertise they claim they have.

Actually, I don’t quite get that far in the article. That’s one topic for this post. The other is an extension of the whaling metaphor in the title. It seems to me that the mad dash for AGI is a bit like Ahab’s quest for Moby Dick, which did not end well for either of them.

What’s an AI expert expert about? NOT human language & cognition.

I chose that whaling analogy to emphasize the peculiar nature of expertise in machine learning: You don’t have to know much about human language and cognition in order to build an AI engine that mimics human cognitive behavior astonishingly well, much better than anyone would have predicted as recently as 2019, the year before GPT-3 was unveiled. Hence the analogy, and AI expert is like a whaling captain who knows all about his ship, but little about whales.

How did this come about and, more to the point, why do we let them get away with it? I didn’t actually pose the latter question, but my essay did suggest an answer to it: This feature of AI-culture has become quasi-institutionalized so that responsibility for pronouncements made by individual researchers must be apportioned between the culture and the individuals. If we question their expertise directly, rather than simply criticizing their arguments, that critique threatens our quasi-institutional understandings about the scope of that culture. Once we start down that road, what other institutional understandings will start to unravel? Let’s not go there.

But that’s a digression. I’m more interested in saying a bit more about how this situation came about.

As I pointed out in the paper, the issue can be traced back to Turing’s famous paper, “Computing Machinery and Intelligence” (1950). “That’s the paper in which he proposed the so-called Turing Test for evaluating machine accomplishment. The test explicitly rejects comparisons based on internal mechanisms, regarding them as intractably opaque and resistant to explanation, and instead focuses on external behavior.” That was (perhaps) a reasonable thing to do at the time. That test, however, was rendered useless in the late 1960s by Joseph Weizenbaum’s ELIZA, a simple computer program simulated human conversation if a very compelling way. At that time, the actual accomplishments of the discipline were not very compelling, at least to researchers outside the discipline, and over-reaching proclamations (about when computers will surpass humans) had few implications for practical action. That is no longer the case. Billions of dollars are being wagered on predictions offered by AI experts.

If we look at what actually happened – and here I’m sketching out a history I haven’t researched thoroughly, I’m just making this up out of what’s already in my mind, so beware of LLM-like confabulation – we see that early researchers did attend to human cognition. That’s most obvious in the case of chess, where research into human chess playing was undertaken, and the ubiquitous expert systems, where human experts were interviewed about their thought processes as preparation for designing the system. Around the corner, researchers in the sibling discipline of computational linguistics (originally machine translation) called on linguistics and cognitive psychology for insight into the design of these systems.

These two lines research of research came together in a large project sponsored by the Defense Department, the ARPA Speech Understanding Project. It extended over five years in the mid-1970s and involved three separate projects involving perhaps a half-dozen research organizations in universities and other research organizations. They undertook to create systems in which a computer would take spoken language questions and provide answers in written English. This was a massive research effort that recruited expertise both in computer science and engineering and in human perception and cognition. No one researcher was expert in all the disciplines involved, but the enterprise encompassed them all, and I assume that at least some researchers read all the reports produced by the project in which they took part, if not all the reports from all the projects. (As bibliographer for Computational Linguistics at the time, I scanned and prepared abstracts for them all.)

Things began to change in the 1980s, when research in connectionist neural networks re-commenced and statistical machine learning techniques began emerging. These techniques effectively separated computational expertise from domain knowledge. It was no longer so necessary to bring deep domain expertise to bear on the design of computer systems. That divergence widened into a yawning chasm with the use of GPUs in the second decade of this century. But the implicit quasi-institutional understandings that had developed back in the 1950s and 1960s remained in place.

The people who develop the computer systems are THE experts. And they certainly are experts. But as far as I can tell, the level of expertise these machine experts have in linguistics and human cognition is nothing to write home about. In that sense they are like the hapless captain of a whaling vessel who knows about his ship, but not about whales.

This is not a healthy situation.

Ahab, Moby Dick, and AI Doom

When I first came up with my title it was but a device to point out the divergence between AI researcher’s knowledge of their programs and their ignorance about language. I had no intention of elaborating it into a conceit that I’d employ at various points throughout the essay. Nor did I have in mind that I’d actually reference Melville’s Moby Dick. That just happened (near the end).

And now that it has, I wonder. Is the pursuit of (the mythical) AGI like Ahab’s pursuit of Moby Dick? Is their fear that the AGI will turn on them, is that like Ahab’s fear of and animosity toward Moby Dick? Is the underlying psychology pretty much the same despite all the obvious differences between the two passions? I don’t know. But as Edward Mendelson argued back in 1976, Moby Dick is an encyclopedic narrative, one that set out to encompass the whole of mid-19th century America. The danse macabre between Ahab and the whale is not thus a private affair between a man and an animal; it is a figure for something at the heart of America. What? Or was Melville just imagining things?

Whatever.

The high-tech industry’s dash to AI supremacy has a similar sweep. We're on a Nantucket sleigh-ride. Instead of 45 ton Moby Dick pulling a 35 ft. whaleboat, a bunch of techbros and Silicon Valley billionaires have hitched the earth to a rampaging Jupiter (318 times the mass of the earth). Look at the numbers in this tweet:

These numbers are based on quite literally nothing. This is SO DUMB. It looks rigorous because it involves numbers, but in reality it's nothing more than baseless hunches. smh. pic.twitter.com/xc5xLG0VYa

— Dr. Émile P. Torres (@xriskology) December 8, 2023

I recognize every one of those names, though I know more about some than others. I assume that the mythical everyone knows who Elon Musk is, but not many are likely to know who Zvi Mowshowitz is. I don’t either, not really. That is, I can’t recite the story of how he came to be regarded as an expert, but I do read his long posts at LessWrong. Musk thinks there’s a 20% to 30% chance of AI killing us all; Mowshowitz puts the number at 60%. The whole range is from 10% to 90%.

Those numbers are insane, Ahab-like insane. If you were running in a marathon and someone came up to you during the race and told you that there was a mere 10% chance you will die unless you exit the race, NOW, what would you do? You’d stop running, just as you would if the chances were 20%, 50%, or 90%. In what way do these people believe those numbers? To what reality are they tethered? To what extent are the tech industry’s actions guided by perceptions no more grounded in reality than the so-called hallucinations of a large language model? 

* * * * *

Addendum 12.11.23: In terms of the informal game theory argument offered in the section, “Deconstructing AI Doom,” of my article, A New Counter Culture, those numbers are a Schelling point, a rallying point for a new (counter) culture. As such, what’s important about those numbers IS NOT their plausibility, though they do come draped in epistemic theatre intended to create the appearance plausibility, but their distinctiveness. Those numbers are not tethered to the reality of mainstream media, whatever that is. They’re a clear demarcation of a different way of looking at the world, one regarded as superior by its proponents.

  

* * * * *

Addendum: See this most interesting article by Andrew Schenker, After Melville, in The Baffler.