ChatGPT: Exploring the Digital Wilderness, Findings and Prospects

That is the title of my latest working paper. It summarizes and synthesizes much of the work I have done with ChatGPT to date and contains the abstracts and contents of all the working papers I have done on ChatGPT. It also includes the abstracts and contents of a number of papers establishing the intellectual background that informs that research. There is also a section that takes the form of an interaction I had with Claude 3.5 on methodological and theoretical issues. Finally, to produce the abstract I gave the body of the report to Claude 3.5 and asked it to produce two summaries. I then edited them into an abstract.

As always, URLs, abstract, TOC, and introduction are below.

• Academia.edu: https://www.academia.edu/127386640/ChatGPT_Exploring_the_Digital_Wilderness_Findings_and_Prospects
• SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5119597
• ResearchGate: https://www.researchgate.net/publication/388563205_ChatGPT_Exploring_the_Digital_Wilderness_Findings_and_Prospects

Abstract: The internal structure and capabilities of Large Language Models (LLMs) are examined through systematic investigation of ChatGPT's behavior, with particular focus on its handling of conceptual ontologies, analogical reasoning, and content-addressable memory. Through detailed analysis of ChatGPT's responses to carefully constructed prompts involving story transformation, analogical mapping, and cued recall, the paper demonstrates that LLMs appear to encode rich conceptual ontologies that govern text generation. ChatGPT can maintain ontological consistency when transforming narratives between different domains while preserving abstract story structure, successfully perform multi-step analogical reasoning, and exhibit behavior consistent with associative memory mechanisms similar to holographic storage.

Drawing on theories of reflective abstraction and conceptual development, the paper argues that LLMs inadvertently capture what wemight term the “metaphysical structure of our universe” – the organized system of concepts through which humans understand and reason about the world. LLMs like ChatGPT implement a form of relationality – the capacity to represent and manipulate complex networks of semantic relationships – while lacking genuine referential meaning grounded in sensorimotor experience. This architecture enables sophisticated pattern matching and analogical transfer but also explains certain systematic limitations, particularly around truth and confabulation.

The paper concludes by suggesting that making explicit the implicit ontological structure encoded in LLMs’ weights could provide valuable insights into both artificial and human intelligence, while advancing the integration of neural and symbolic approaches to AI. This analysis contributes to ongoing debates about the nature of meaning and understanding in artificial neural systems while offering a novel theoretical framework for conceptualizing how LLMs encode and manipulate knowledge.

Contents:

Introduction: Into the Digital Wilderness 5
Free-floating Attention, Systematic Exploration, and the Anthropomorphic Stance 8
ChatGPT: My Course of Investigation 12
Meaning, Truth and Confabulation, Latent Space 28
Prospects: Explicating the Ontology of Human Thought 42
A Dialogue with Claude 3.5 on Method and Conceptual Underpinnings 45
A Brief Narrative of My ChatGPT Work Based on My Working Papers 56
Working Papers about ChatGPT 62
Background Papers 74

Introduction: Into the Digital Wilderness

The world I entered when I started playing with ChatGPT is a wildnerness, strange and uncharted, uncharted by me, uncharted by anyone. By that I simply mean that it was something new, radically new. No one had been there before. Sure, a handful of people within the industry had been messing around in there, even a rather large handful considering how much work it took to make ChatGPT ready for the world at large. But its behavioral capabilities were, for the most part, unknown. In that sense it was a wildnerness.

But it was, and remains, a wilderness in another sense: the large language model (LLM) that underlies ChatGPT is a black box. We send a string of words into ChatGPT and it sends a string of words back out, but what the model does to derive the output from the input, that process remains deeply obscure. That is wilderness in a different sense. Wilderness in the first sense is about our experience of ChatGPT’s behavior. Wildnerness in this second sense is about the mechanisms that drive that behavior. It is a digital wilderness. This document reports on how I’ve structured my interaction with ChatGPT to give me clues about the mechanisms driving its behavior.

My methods are more “qualitative” or “naturalistic” than those standard in the literature, which many investigations employ standard batteries of benchmark tasks. While those are essential, there is much they don’t tell you. While I have done many things with ChatGPT – asked it to interpret texts, define abstract concepts, play games of 20 questions, among other things – perhaps my most characteristic task, and one I have spent more time on than others is simple: Tell me story. And ChatGPT did so, time and again. Consequently my methods are in some ways more like literary criticism, or, even better, like Lévi-Strauss’ analysis of myths, than conventional cognitive science. Consequently you will find many examples of ChatGPT’s dialog in my reports. You have to examine that dialog to see what ChatGPT is doing, what it is capable of doing.

Finally, I realize that the pace of development in this arena is such that ChatGPT is now old. The versions I used to conduct these investagations are no longer available on the web. However, as far as I can tell, none of the results I report depend on features idiosyncratic to those versions.

The rest of this introduction consists of short statements about what the various sections of this report contain.

DeepSeek works though distillation [Schmidhuber]

DeepSeek [1] uses elements of the 2015 reinforcement learning prompt engineer [2] and its 2018 refinement [3] which collapses the RL machine and world model of [2] into a single net through the neural net distillation procedure of 1991 [4]: a distilled chain of thought system.… pic.twitter.com/wpk2tFCoUx

— Jürgen Schmidhuber (@SchmidhuberAI) January 31, 2025

How long can this go on? A short note on American politics

The Trumpists’ blitzkrieg was not unexpected, though I think it’s been faster and more extensive than anyone had anticipated. What I’m wondering: How long can this go on?

I’m not thinking about weeks or months. I’m thinking about the next presidential election cycle. The Trumpists and the GOP are assuming that they’ll win in 2028 and install Vance or whomever in the White House. Their minions will be able to keep their jobs. They’ll be continuity of government.

But this is by no means guaranteed. The world’s too volatile, and much of that volatility is beyond (anyone’s) political control. The Democrats might well win. What then?

Will they blitz the Trumpist GOP out of Washington? That’s not the Democrat’s proceduralist style. But if they move away from that to a dirtier style, then it’ll be another blitz.

Will 2032 bring a blitz back in the other direction? What about 2036? How long can the body politic play the game of blitz and counter-blitz?

I don’t know. But I don’t think it can go on too long. It’s too chaotic. It’s one thing to blitz an administration out of Washington and then have the new regime settle down for a couple of decades. But a continuous blitz-and-counter-blitz is likely to collapse the Federal Government.

What would that look like? I don’t know. That’s why I’m writing this post.

Interesting times.

Friday Fotos: Reality has aspects many and various

Football • {sports commentary} • [Media Notes 154]

I confess, football doesn’t interest me very much, never has. But I’m an America and I live in America. So I can’t escape it.

When I was in junior high school and high school I played in the marching band. That required me to attend every football game so we could provide half-time entertainment. We were so good, however, that I suspect some people came to the games more to hear us play than to see the game itself.

That’s the only time in my life that I ever watched football regularly. Of course, we played a bit of touch football in gym class, but that was it. I attended one football game in college. I was in the band. When one half of the band finished a tune eight bars ahead of the other half, that’s when I decided to blow this pop stand.

When I was in graduate school at SUNY Buffalo a roommate bequeathed me a small B&W portable. I watched a number of football games on it. This was during the O. J. Simpson years and I’d watch the Bills games to see him run. He was sensational. Football I didn’t care about, human excellence, that’s another matter.

After that, sure, every once in a while I’d catch a game. At least I assume I did. As I said, I’m living in America. Then, for some reason, a couple of weeks ago I decided to catch a play-off game on Netflix. Why? Why not? So I watched the Baltimore Ravens vs. the Pittsburgh Steelers. I’m from Pennsylvania, so that inclines me toward the Steelers. (The name “Franco Harris” sticks in my mind, so I must have watched some games when he was playing). I sent to school in Baltimore, which would tip me toward the Ravens, thought it was the Colts in Baltimore when I was there (Johnny Unitas as QB). Fact is, I could have cared less who won. Didn’t even watch the fourth quarter.

A week later it was the Buffalo Bills vs. Kansas City Chiefs. I made it the whole way through on that one. But I would hardly say I watched the game. Us, I did watch it, in fits and starts. But I also cruised the web doing this and that.

Which brings up a question: Let’s say the total elapsed time from the beginning of a game to the end is two to two-and-a-half hours. Only an hour of that is game time, which is interrupted for various reasons for varying amounts of time. During those interruptions we’re either getting some kind of commentary on the game, or we’re getting commercials. Add up the total time devoted to the game and commentary on the game. Add up the total time devoted to commercials during the broadcast. What’s the ratio between the two? My guess is that game time would be the larger number, but I’d guess the ratio is closer to 3/2 than to 2/1 in favor of game time.

So, I guess we could say that the commercials exist so that we can watch the game. But it could easily go the other way. Of course, if you’re a football fan, and so heavily invested in the game, that that’s certainly your priority. But if you’re not a fan, then it could almost go the other way.

Which brings me to the real reason for this note: the commentary. That fascinates me. I’m interested in it as a perceptual and cognitive activity. As I understand it, we generally have two commenters, one commenting on the action (play-by-play) and the other commenting on this and that. I believe the second is doing color commentary.

The play-by-play commenter is expected to comment on what’s happening as it happens. That requires them to have had a great deal of experience watching football games, more experience than I’ve had. You have to be able to instantly recognize hundreds of different patterns of activity and associate them with appropriate verbal comments. This is not a time for careful deductive reasoning. It’s an associative process. The commentary must be so fluid as to be of a piece with the perceptual act.

I wonder how long it takes to develop this capacity to the level we see in professional commentators? 10,000 hours? I don’t know. Let’s do a quick calculation. Ten thousand hours works out to something less than 5000 games, somewhere between 4000 and 4500. Let’s say it’s 100 games a year, two games a week. That’s forty to forty-five years. That’s possible, but I think the pros get in the game well before that. So it’s not 10,000 hours. It’s less than half that.

And then there’s the color commentary. That doesn’t have to track the action moment by moment, so it’s not constrained in that way. But still, it’s not an occasion for deductive reasoning. The commentator has to have access to a large range of relevant information about the players and the game, past and present, and come up with relevant bits and pieces in a matter of seconds. So it’s still pretty much an associative process.

And THAT, those last three paragraphs, that’s why I’m writing this not. As for the rest, why note? It’s context.

Sabina Hossenfelder on DeepSeek and its implications

4:08: As we discussed previously, some experts including Yann LeCun and Gary Marcus, doubt that Large language models will ever get us to general intelligence, and I agree with them. The market will only react to this once it becomes clear that the existing LLMs can’t be recued with further updates. At that point a lot of money will evaporate like this. But at the moment, the stargate project is just pumping more money into an existing bubble. Building power plants, extending the grid, and improving data infrastructure generally seems like a good idea, and all these are part of the stargate project. 

But to me the Stargate Project is as crazy as if Americans had taken the first semi-conductors and spent $500 billion on factories to produce them, rather than letting markets do their thing and wait for technological developments to make microchips smaller and cheaper. Ie, to wait for them to make economic sense. It’s like the dot-com bubble, except instead of getting free T-shirts from Pets dot com, we get hallucinating chatbots and 17-hour debates about whether sentience can be monetized.

5:17: That said, the arrival of DeepSeek drives home an important message: you can save a lot of money if you let Americans do the heavy lifting and then build on that knowledge. And that goes well with the European approach, which is basically to wait and see what goes wrong in America. I’m not usually a fan of the European risk-aversity. It reminds me of how my younger brother was waiting for me to touch the electric fence. But in this case “wait and see” might indeed work out to our advantage. And if not, we’ll always be here to give Americans lectures about responsibility, sustainability, and how our regulation-heavy bureaucracy prevents us from having fun.

Three diverse gray-scale shots

Meeting of Minds: Richard F. Burton, Thor Heyerdahl, Marco Polo, and a Surprise Guest

Before I give you the conversation I cooked up with the help of Claude 3.5, let me explain roughly how things went. Once Claude learned that I wanted it to tell a story of some kind, it split the screen in two vertically.

1. I gave Claude a prompt. [Left panel]
2. Claude briefly told me what it would do. [Left panel]
3. I then produced some dialog. [Right panel]
4. Claude told me what it had done and gave me suggestions about how to proceed. [Left panel]
5. I told Claude how to proceed. [Left panel]

And the process repeated. This happened five times. On the fifth time I told it what to do next and requested that it end the dialog with this segment. When it was done I congratulated it and made some final remarks

I’m not going to attempt all that back and forth in the dialog below. I start out with my initial prompt, then I present the entire dialog without interruption. Note, however, that I have inserted subtitles at the point where Claude and I had a discussion about how to proceed. I conclude with Claude’s final remarks.

* * * * *

I want you to stage an imaginary dialog between three famous explorers: Richard Francis Burton, Thor Heyerdahl, and Marco Polo. Let’s imagine they’re all transported to the bar in the Explorer’s Club in Manhattan, where they have a few drinks and chat. There’s also a television in the room, but it’s not turned on when the conversation begins. Heyerdahl begins the conversation by introducing himself and explaining to the other two where they are and how they got there (time travel). Of course Marco Polo wouldn’t even have known that America exists. Burton would have known. Once introduced each tells of their most dangerous expedition.

1. The story begins

[The wood-paneled bar of the Explorer’s Club in Manhattan. Three men find themselves seated at a corner table, surrounded by mounted exotic animal heads and artifacts from around the world. A bartender quietly polishes glasses in the background.]

THOR HEYERDAHL: [adjusting his bow tie] Gentlemen, I realize this is rather startling, but allow me to explain. I’m Thor Heyerdahl, and through some inexplicable circumstance, we’ve all been transported to what’s known as the Explorer’s Club in Manhattan, New York. The year is - well, sometime in the 21st century. [gestures to the room] This is a gathering place for adventurers and explorers, founded in 1904.

MARCO POLO: [looking bewildered] Manhattan? New York? These words mean nothing to me. I am Marco Polo of Venice, and the last I remember, I was dictating my travels to Rustichello da Pisa in my prison cell in Genoa.

RICHARD BURTON: [sipping from a glass of whiskey] Fascinating. Sir Richard Francis Burton at your service. I’m quite familiar with New York, though I must say the city has changed considerably since my time. [to Polo] My good man, you’re sitting in a city on a continent that wouldn’t be known to Europeans until nearly two centuries after your death.

HEYERDAHL: [signaling to the bartender] Perhaps some drinks will help ease the shock. Now, since we’re all here, we might as well share stories. I’ll begin - my most perilous journey was arguably the Kon-Tiki expedition in 1947. Picture this: a balsa wood raft, 4,300 nautical miles across the Pacific Ocean, just to prove that ancient South Americans could have reached Polynesia.

BURTON: [leaning forward with interest] A worthy endeavor! Though I must say, my journey to Mecca in disguise as Sheikh Abdullah in 1853 carried perhaps even greater risk. One wrong move, one slip in my portrayal, and I would have been executed on the spot for being an infidel.

POLO: [taking a sip of wine] Executed? I understand such dangers well. During my journey to Cathay - what you would call China - we crossed the Gobi Desert. For days we heard what we thought were voices calling our names, evil spirits trying to lead us astray into the wastes. Many who wander off following these voices are never seen again.

BURTON: [skeptically] Come now, Polo. Surely you don’t expect us to believe in desert spirits?

HEYERDAHL: [diplomatically] Whether spirits or hallucinations from dehydration, the danger was real enough. [glances at strange black box on the wall] I say, that’s what we call a television. It can show images from around the world. Shall we turn it on? It might help demonstrate to Messer Polo just how much the world has changed.

POLO: [eyeing the television warily] More sorcery? In my travels I saw many wonders - paper money, burning black stones, even messages carried by pigeons across vast distances. But this...

BURTON: [chuckling] My dear Polo, you should have seen the skepticism I faced when I published your accounts in English. Many called you a liar or a braggart. Yet here we sit, centuries later, knowing you spoke truth about China’s wonders.

HEYERDAHL: [raising his glass] To vindication! My own theories about Pacific migration were mocked by the academic establishment. They said it was impossible to cross the Pacific on a primitive raft. Well, we showed them, didn’t we?

POLO: [curiously] You crossed an ocean larger than the entire realm of Kublai Khan... on a raft made of grass?

BURTON: Balsa wood, Polo. Though I must say, I prefer a well-appointed expedition. My search for the source of the Nile had 200 porters carrying everything from silver tea services to a folding boat.

HEYERDAHL: [laughing] Times have changed indeed. Though I suspect all of us share one thing - the desire to prove what others claimed impossible.

POLO: [nodding] True enough. When I returned to Venice after 24 years away, my own family didn’t recognize me. They thought me long dead. Even after I proved who I was, many refused to believe my tales of paper money and vast cities in Cathay.

BURTON: [raising his glass] To the unbelievers then - without whom we might never have felt so compelled to prove our points. Though I must say, Heyerdahl, your raft journey makes my disguised pilgrimages seem rather comfortable in comparison.

HEYERDAHL: [modestly] We all faced the challenges of our times. [gestures to television] Shall we see what modern explorers are up to?

2. Raiders of the Lost Ark

[The television suddenly comes to life, showing Raiders of the Lost Ark. The opening scene plays as Indiana Jones navigates the Peruvian temple.]

POLO: [startled by the moving images] By Saint Mark! What sorcery is this? The people are so small, yet they move as if alive!

BURTON: [fascinated] Remarkable. The quality of these moving photographs far exceeds anything Edison was working on in my time.

HEYERDAHL: [grinning] This is what we call a movie - “Raiders of the Lost Ark.” It’s about a fictional archaeologist named Indiana Jones. [gestures to the screen] He’s searching for ancient artifacts, much as we all did in our own ways.

BURTON: [watching Jones swing across a chasm] Rather acrobatic for an archaeologist, wouldn’t you say?

POLO: [wincing as Jones triggers a trap] In my travels through the temples of the East, I never encountered such elaborate mechanical devices. Though... [thoughtfully] there was that one Buddhist monastery in Tibet with rather ingenious defensive mechanisms...

HEYERDAHL: [as the giant boulder scene plays] Pure Hollywood invention, of course. Though I must say, during my excavations on Easter Island, we did have to be extremely careful about cave-ins.

BURTON: [snorting as Indy escapes] If my porters had abandoned me like his just did, I’d have had them flogged. Good help was essential in the exploration business.

POLO: [as Belloq appears] Ah, now this I understand - rival merchants seeking the same prize. Though in my day, we were more concerned with silk and spices than golden idols.

HEYERDAHL: Wait until you see what they’re really after - the Ark of the Covenant itself.

BURTON: [nearly choking on his whiskey] The Ark? From the Temple in Jerusalem? Preposterous! I’ve studied enough religious artifacts to know...

POLO: [interrupting as the Nazi villains appear on screen] Who are these men in matching black uniforms?

HEYERDAHL: [grimly] That’s a rather long and unpleasant story from my time period. Let’s just say they’re the villains.

BURTON: [watching the map room scene] Now this interests me - using ancient astronomical alignments to locate a buried site. During my expeditions to find the source of the Nile, we often relied on local astronomical knowledge. Though I never had a staff quite that elaborate...

POLO: [as Marion appears] At least they show the truth that women can be as fierce as men. I met many such women in Mongolia. Though they typically wore more practical clothing for combat.

HEYERDAHL: [chuckling] Just wait until you see what happens in the snake pit...

It's the light

About that funding freeze – It's stupid, careless, and cruel

Let's be clear, I do not like Donald Trump, and I certainly did not vote for him. Yet not too long ago I posted, The Vibes They Are a-Shiftin', in which I said:

Still, on the whole, the vibes are somehow good.

Does that mean that at long last the techtonic plates are beginning to shift? I sure hope so. And if they're shifting, it remains to be seen just who'll be guiding them into a new configuration. That is not at all written in stone.

I'm beginning to regret that. I'd forgotten that, whatever else he is, Trump is also cruel. Is that what his followers voted for?

Who the hell knows what the
next week will bring.

Robert Gordon over at Eating Policy has some cogent remarks under the title, Vast Carelessness:

I have spent years railing against long government memos with languorous timelines. How about a two-pager? Where’s the urgency?

Careful what you wish for. OMB’s two-pager on grants, with its 24-hour deadline, dropped like a bomb on a trillion-dollar grantmaking enterprise (and that’s just at HHS). It risks wreckage across the country.

The memo required agencies to pause all spending, with a footnote exempting “assistance received directly by individuals.” HHS suspended states from drawing down Medicaid funds and tribal facilities from drawing down Indian Health Service Funds. Also suspended were grants that provide funds for Head Start, child care, LIHEAP, and more. By midday yesterday, OMB issued a corrective note explaining that the footnote excluded Medicaid and Head Start, as well as other programs like SNAP and Pell grants. That was good, if incoherent, since Medicaid makes payments through intermediaries (states, which may in turn pay managed care organizations which pay providers) as much as any federal program. And the order leaves in the crosshairs, just at HHS, programs offering drug treatment and mental health services, funding foster care, serving the homeless, engaging in medical research, and on and on.

Smart lawyers have focused on the question whether the delay violates the Impoundment Control Act (sure seems to), but there’s another problem. The actions now unfolding may well violate the terms of any number of grant agreements. Grant agreements bind two parties—not only the grantee, but also the government. Here’s a snippet from the key text, the Uniform Grants Guidance:

Payments for allowable costs must not be withheld at any time during the period of performance unless required by Federal statute, regulations, or in one of the following instances: (i) The recipient or subrecipient has failed to comply with the terms and conditions of the Federal award; or (ii) The recipient or subrecipient is delinquent in a debt….

[...]

Many of us desperately want to simplify the government’s rules and procedures, but that does not mean we want no rules at all. When a state or grantee receives a grant from the federal government, the receipt of the grant does not turn that state or grantee into a vassal or a subject who can only appeal for the king’s mercy. [...]

What is remarkable is how easily the Administration could have pursued the same ideological goals in a more orderly way.

There's more at the link.

I talk Loony Tunes with John McWhorter @3QD

I’ve got a new piece at 3 Quarks Daily:

John McWhorter and I Talk About Cartoons

That went up yesterday. 

Today Brooks Riley followed up with this:

There's a lesson there: Cartoons play the court jester to the dominant view of reality.

I ended the article with a scene from Who Framed Roger Rabbit, one where Daffy Duck and Donald Duck battle one another at the piano. Such battles were common in cartoons. Here’s another one:

Here's a variation on the theme:

 

 

 Have fun!

Snoopy's response to DeepSeek

MAGA’s Big Tech Divide

That's an article in the NYTimes by Ezra Klein in which he interviews James Pogue. I asked Claude 3.5 to summarize it.

* * * * *

I’d like you to summarize the attached article. Assume an intelligent and well-educated reader. Thank you.

This article is a transcript of an Ezra Klein Show interview with James Pogue, a journalist who covers the "New Right" movement within MAGA politics. The conversation, published in January 2025, explores the intellectual currents and tensions within Trump's coalition following his return to the presidency.

The discussion centers on two competing factions within MAGA: a traditionalist wing represented by figures like JD Vance (Trump's vice president), and a tech-oriented wing aligned with Elon Musk and other Silicon Valley figures. The traditionalist wing emphasizes cultural restoration, masculinity, and skepticism of modern technology's effects on society. The tech wing, while sharing some of these cultural concerns, envisions using technological advancement to achieve MAGA's goals.

Pogue describes how these factions, despite their differences, are united in opposing what they call "the regime" - a perceived establishment comprising mainstream media, universities, NGOs, and traditionally powerful institutions. He notes how Trump, while not deeply engaged with either faction's ideology, has managed to balance both groups' interests, particularly evidenced by his relationship with both Vance and Musk.

The interview delves into complex ideological threads, including the movement's relationship with modernity, its vision of masculine virtue, and its understanding of American identity. Pogue suggests that while the movement's intellectual foundations are robust among younger staffers and theorists, it remains unclear how much these ideas will translate into actual governance, given the practical constraints of running the federal government and Trump's own tendency to prioritize personal loyalty over ideological purity.

A particularly notable element is the discussion of how the movement reconciles its critique of technology and modernity with its embrace of tech billionaires and social media. This tension, Pogue argues, reflects broader contradictions within the movement as it transitions from opposition to governance.

The conversation concludes by examining how these competing visions might shape American policy and society, with Pogue suggesting that the success of the administration may depend on whether these different factions can maintain their coalition while implementing their shared agenda.

Who are the most important thinkers in the various factions?

Based on the article, several key intellectuals and thinkers emerge across the different factions of the New Right movement:

For the traditionalist wing, Curtis Yarvin stands out as what Pogue calls "the dark lord intellectual godfather" of the movement. Yarvin is described as a reactionary who believes that people were better formed in earlier eras (specifically citing Pennsylvania farmers in 1800) and who has written against techno-optimism despite his own background in technology.

Patrick Deneen is another significant figure, whose book "Regime Change" is described as "Curtis Yarvin for normies." The article suggests his work provides an intellectual framework for the movement's desire to replace current elites and institutions.

For the tech-oriented wing, Marc Andreessen emerges as a key intellectual figure, particularly through his "Techno-Optimist Manifesto." Peter Thiel, while somewhat removed from current politics, is portrayed as bridging the traditionalist and tech wings through his earlier support of figures like JD Vance and Blake Masters.

The article also mentions some less prominent but influential figures like Micah Meadowcroft, who Pogue describes as "one of the smartest people of this world," and who articulates the movement's desire to rebuild American "yeomanry." There's also reference to Bronze Age Pervert, a figure who promotes ideas about masculinity and physical culture.

Interestingly, while JD Vance is described as a unifying figure who can speak to both wings of the movement, he's portrayed more as a political leader who synthesizes these ideas rather than an original thinker himself.

The article also notes that many of these ideas were initially developed by anonymous or pseudonymous figures on Twitter and in blogs, suggesting a broader intellectual ecosystem beyond the named figures.

This intellectual framework appears to draw from various traditions, including Catholic social teaching, classical republicanism, and tech entrepreneurship, creating what Pogue describes as a coherent, if sometimes contradictory, worldview.

Between Jersey City and Hoboken [a view from the old Van Leer chocolate factory]

The DeepSeek breakthrough – What’s it mean? [On the difference between engineering and science]

Frankly, at the moment I’m inclined to think it means that Silicon Valley just got handed its lunch. Strutting around about AGI this, $500 billion that... Voilà! We’re Masters of the Universe. They may or may not be “masters of their domain,” (likely not) in the Seinfeldian sense, but Masters of the Universe they are NOT.

Engineering automobiles, or rockets – that’s one thing. Engineering artificial minds. What’s going on is unprincipled hacking. Throw enough person-hours, compute, and money at it and, sure, you’ll do something....

Mutt: “Slow down, son, you’re ranting!”

Jeff: “OK, OK, I’ll slow down.”

https://t.co/HFuCmxf0Gr pic.twitter.com/KFBp2eyhdk

— Marc Andreessen 🇺🇸 (@pmarca) January 27, 2025

Full disclosure: My priors

I was trained in computational semantics back in the mid-1970s by David Hays, who had been a first-generation researcher in what was originally called machine translation (MT) but got rebranded as computational linguistics (CL) in the mid-1960s when it got defunded for failing to deliver practical benefit to the US Military. At the time I worked with him Hays had shifted his attention to semantics. In keeping with that focus he read a great deal about cognitive and perceptual psychology and neuroscience. He wanted his models to have some grounding in scientific fact.

He tended to think of AI researcher as unprincipled hackers. If the program worked, that’s all that mattered. They didn’t seem very interested in possible psychological reality.

And that’s what the current regime of work on deep learning looks like to me: hacking. To be sure, it’s brilliant, perhaps inspired hacking. If I didn’t believe that I wouldn’t have spent a great deal of my time over the last two years working with ChatGPT and now Claude 3.5, working to tease out clues about what’s going on under the hood.

That’s a problem, no one actually knows how these models work. Oh, there’s interesting research on that problem, some of it under the rubric of mechanistic interpretability. But that doesn’t seem to be a priority. Instead, the emphasis is on scaling up, more data, more compute, more parameters, more more more! (Slow down son!)

The impact of DeepSeek

Given that scaling up has worked in the past, and in the absence of any deep insight into how these things work, the scaling hypothesis, as it is sometimes called, had a certain superficial validity. The Chinese have just blown a big hole in the scaling hypothesis. Here’s Kevin Roose in The New York Times:

The first is the assumption that in order to build cutting-edge A.I. models, you need to spend huge amounts of money on powerful chips and data centers.

It’s hard to overstate how foundational this dogma has become. Companies like Microsoft, Meta and Google have already spent tens of billions of dollars building out the infrastructure they thought was needed to build and run next-generation A.I. models. They plan to spend tens of billions more — or, in the case of OpenAI, as much as $500 billion through a joint venture with Oracle and SoftBank that was announced last week.

DeepSeek appears to have spent a small fraction of that building R1. [...] But even if R1 cost 10 times more to train than DeepSeek claims, and even if you factor in other costs they may have excluded, like engineer salaries or the costs of doing basic research, it would still be orders of magnitude less than what American A.I. companies are spending to develop their most capable models. [...]

But DeepSeek’s breakthrough on cost challenges the “bigger is better” narrative that has driven the A.I. arms race in recent years by showing that relatively small models, when trained properly, can match or exceed the performance of much bigger models.

What that means is that the industry’s intuitive understanding of what the late Dan Dennett liked to call the design space for AI, that understanding is wrong. Yes, bigger does sometimes/often get you more performance. But if smaller can yield comparable performance, than something else is going on, something we can’t identify.

Were the Chinese just lucky? Or do they know something, something deep, that we don’t? In the absence of any further information, I’d guess that it’s both.

Can we figure out what they did and do it ourselves? Sure, no problem. DeepSeek is an open-source model and the researchers have released good documentation. We’ll replicate what they’ve done and perhaps improve on it, as they will also do. That’s not the issue.

The issue is understanding. We already knew that these models are black boxes. And we guessed/hoped that buiding a bigger box would make it better. We now know that that’s not necessarily true. What else don’t we know? How are we going to find out?

Four versions of "She's Not There"

The fourth version is the one we played in "The Saint Matthew Passion," a band I played in back in 1970-71 or so.* Were were a rock band with horns, trumpet, sax, and trombone. We modeled ourselves on Chicago Transit Authority (CTA, which later became just Chicago) and Blood, Sweat, and Tears. I have no idea where we got our version from, maybe Jon (our leader) just made it up. But it certainly wasn't modeled on one of the versions in this post. It's was The Zombies' song. I like the Santana version (this is the first time I've heard it). But I love the Vanilla Fudge version, which I knew back when their record first came out.

*NOTE: FWIW, we played it on our final gig, when I had a mystical experience. I doubt that that particular song had anything to do with it. It was but the vehicle. And the experience, the white-out and the opening-up, didn't happen during the song proper, but during an introduction that could have gone with just about any song. Whatever. It shook me. I'm still vibrating, ever so little, but still.

The old neighborhood in Jersey City

I was on the web before the New York Times was, me and Cuda Brown at Meanderings

Megan DeTrolio tells the story about how The New York Times got its domain name, and from one of its own reporters.

On Jan. 22, 1996, in an article tucked away on Page D7, The New York Times announced the public launch of its website.

“The New York Times begins publishing daily on the World Wide Web today, offering readers around the world immediate access to most of the daily newspaper’s contents,” stated the article, by Peter H. Lewis. “The electronic newspaper (address: http:/www.nytimes.com) is part of a strategy to extend the readership of The Times.”

Mr. Lewis had once owned that very URL.

In 1985, the Times editors A.M. Rosenthal and Arthur Gelb gathered a task force, which included Mr. Lewis, to work on a project called The New York Times in the Year 2000. Mr. Lewis this week shared the details of the project and his Times work in an email, from which much of this account is drawn.

Then an editor for the Science section and a personal computers columnist, Mr. Lewis recalled predicting that by the millennium, Times articles would be read on personal computer screens, in cyberspace.

“I recall Artie dismissing me with a wave,” Mr. Lewis wrote of Mr. Gelb.

Years later, the editor Bill Stockton, who Mr. Lewis said championed science and technology reporting, assigned Mr. Lewis to cover the “rise of the internet.”

At some point, “I asked permission to register a web domain for The Times, and was told no,” Mr. Lewis wrote in the email. “Several of us thought that was shortsighted.”

And so it goes. There's more at that link.

Back in 1994 I met Cuda Brown, not his real name BTW, in the African-American Forum on Compuserve. We hit it off. Cuda had been sending out an email to friends and family for several months. He called it "Meandings." He was thinking of putting it on the web. He asked me to join him. I forget just exactly when we went live, but here's Meanderings, archived in the WayBack Machine. And here I am, listed in Meanderings 1.06 for October 1994. That must have been when we live on the web. And here's Meanderings 2.05, where we partnered with Vibe Magazine to publish essays about O. J. Simpson. Those were the days!

By then we'd launched Gravity, with Meanderings as one of Gravity's offerings. Why "Gravity"? I think the idea was that we'd be a persistent Black Hole in Cyberspace. There you can see our tribute to Martin Luther King. I could go on and on about that. Maybe later.

Anyhow, we didn't last much longer than that. I forget why we folded. Probably because Cuda had to go back to working for a living. He'd taken time off work, as an investment banker in the utilities practice of Shearson Lehman (remember them?), to launch Meanderings from a server in his house. Whatever. By then Salon had come on line and I camped out in their Table Talk discussion forum, where I lingered 'till the bitter end in 2011. I forget what happened to Bill, Cuda's real name. But we kept in touch.

Those were the days.

Have the Chinese taken the lead in LLM sweepstakes? [with commentary by Dread Pirate Marcus]

Unbelievable results, feels like a dream—our R1 model is now #1 in the world (with style control)! 🌍🏆 Beyond words right now. 🤯 All I know is we keep pushing forward to make open-source AGI a reality for everyone. 🚀✨ #OpenSource #AI #AGI #DeepSeekR1 https://t.co/h0pT2Em14D

— Deli Chen (@victor207755822) January 24, 2025

Gary Marcus, The race for "AI Supremacy" is over — at least for now:

The race for "AI Supremacy" is over, at least for now, and the U.S. didn't win. Over the last few weeks, two companies in China released three impressive papers that annihilated any pretense that the US was decisively ahead. In late December, a company called DeepSeek, apparently initially built for quantitative trading rather than LLMs, produced a nearly state-of-the-art model that required only roughly 1/50th of the training costs of previous models, — instantly putting them in the big leagues with American companies like OpenAI, Google, and Anthropic, both in terms of performance and innovation. A couple weeks later, they followed up with a competitive (though not fully adequate) alternative to OpenAI's o1, called r1. Because it is more forthcoming in its internal process than o1, many researchers are already preferring it to OpenAI's o1 (which had been introduced to much fanfare in September 2024). And then ByteDance (parent company of TikTok) dropped a third bombshell, a new model that is even cheaper. Yesterday, a Hong Kong lab added yet a fourth advance, making a passable though less powerful version of r1 with even less training data.

None of this means however that China won the AI race or even took the lead. American companies will incorporate these new results and continue to produce new results of their own.

Instead, realistically, we are quickly converging on a tie — with some style points to China, for doing so much without hundreds of thousands of Nvidia H100s.

Others may catch up, too, because LLMs just got a lot cheaper, and consequently the requirement for vast arrays of special purpose hardware has somewhat diminished. There is almost no moat left whatsoever; new technical leads very short lived, measured in months or even weeks, not years.

Later:

China caught up so quickly for many reasons. One that deserves Congressional investigation was Meta's decision to open source their LLMs. [...]

Putting so many eggs in Altman’s basket, as the White House did last week and others have before, may also prove to be a mistake in hindsight. [...] Altman may be a master salesman, but Musk is correct that the US should not be so reliant on him and should not have given him such an eminent seal of approval based on so little.

In a brutal, viral tweet that captures much of my own thinking here, the reporter Ryan Grim wrote yesterday about how the US government (with the notable exception of Lina Khan) has repeatedly screwed up by placating big companies and doing too little to foster independent innovation.

Marcus concludes:

The race to AGI will be won not by the country with the most chips but by the one that best fosters true innovation. That could be the US, China, or perhaps some other country less locked into LLM orthodoxy, and more willing to bet big on new ideas.

There's more at the link.

Photos I've recently worked on

Ramble for 2025: music, 3-year plan, photos, LLMs, great critics

Lots to do.

Music

As I recently noted, I’ve recently subscribed to YouTube music AND I’m lovin’ it. It’s a pleasure to be able to type in the name of any piece of music – song, composer, performer, style, album, whatever – and have it come up. Mostly, though not always. I continue listening to the rock and roll of my late teens and early 20s – Good Lord! am I really that Boomer!?

In the past couple of days a couple of tunes have haunted me: most of all “I Love You More than You’ll Ever Know” from Blood, Sweat, and Tears’ first album. Just why I find it so compelling. Perhaps the fact that I played it in the “Saint Matthew Passion” has something to do with it. No, not Bach, not beyond the name. That was the name of a band I played in back in ’69-’70-perhaps-’71. That’s when I got serious about improvising.

And then there’s “Once” from a Stan Getz album, Voices. The album was based on a gimmick. Instead of a string ensemble for back-up – itself something of a gimmick in a jazz context – the album had a voice choir. Shockey but effective! “Once” is based on Ravel’s “Pavane pour une infante défunte,” which is hauntingly lovely.

Three-year plan

It’s got a three-month horizon, a one-year horizon, and a three-year horizon. Everything will be adjusted as we go along.

Three months: Get an agent. Even better, get and agent and get a book contract. Tentative title for the book: Welcome to the Fourth Arena. It’s a general interest book on how AI can potentially remake the world. I’ve got a set of chapter topics/titles that I like and I’ve been reviewing Kisangani 2150 for material to conclude the book with. I pretty much know what I’ve got to do to get the proposal together. I need two sample chapters as well. At the moment I’m figuring on the opening chapter, derived from the article I’ve linked above, and a concluding chapter based on the Kisangani material

One year: Complete the book and deliver it to the publish. Begin the work of monetizing my photographs. To that end I’ve been putting together a Flickr album that samples the photos I’ve done over the last two decades. That’s been fun. I’ve now got 488 photos in that album, and I’ve not yet been through the whole body of work. I like what I’ve got so far. But how do I whittle it down?

Three years: I want to be in a new apartment, with all my stuff, and have a nice, but not necessarily new, car. This depends on drumming up some new income. We’ll see.

Photos

Ah yes, as I’ve said, that’s going along nicely. And I’m thinking about what I’m doing as I go along, the process. It seems to me that what I’m trying to do is compare all the photographs together and see what rises to the top. Easier said than done.

So I’ve got this album where I’m collecting photos. I keep going back through the album, editing it a bit. I’ll drop a photo or three from the album. And add some new ones. I’m also producing new renderings of old photos. They’re looking good, better than the old renderings.

I’m beginning to get a glimpse, a glimmer, of my style, whatever that it. These new renderings seem to be helping. That is, an important element of my style is how I rendering each image, my choice of colors, tonalities, balances. That’s certainly as important as my choice of subject matter. That’s what pulls everything together across the various subjects.

LLM-land

I’m still being dogged by the need to finish my report on ChatGPT. I did manage to get the section on ontology completely drafted. That was a major piece of work. I’ve got two more big chunks to do before I wrap it all up. There’s the section, “Prospects: In Search of the Star Trek Computer,” which will end the report, and the introduction, which will have one-sentence introductions for the rest of the report.

And I’m deep into Claude 3.5. I’ve been enjoying have it describe photos. Some of those have turned out to be quite interesting, especially when they get into technical issues, which lead automatically to aesthetics. I also want to develop the meeting-of-minds game.

Greatest literary critics

I know, I know. I’ve got to finish this off. All I need to do is a final chapter on Harold Bloom, which I more or less know how to do, and I’m ready to wrap it up. I have no idea when I’ll get around to this. After all, I’ve got a book proposal to write.

Soon.

Imagine, a future where flying cars are real and Rosie the Robot helps the missus clean house

In the court of the Orange King

Ross Douthat, The Wars Within Trump’s Court, NYTimes, Jan. 25, 2025

A scene from the first week of the second Trump administration: After the president held a White House event announcing a shared venture, with up to $500 billion of funding, among OpenAI, Oracle and SoftBank to build a vast new data center for the artificial intelligence future, Elon Musk sniped on X that the money for the venture wasn’t really there.

Asked if his billionaire ally’s snarking bothered him, the president shrugged it off: “No, it doesn’t. He hates one of the people in the deal.” This was reference to Musk’s conflicts with Sam Altman, the quietly polarizing head of OpenAI. And, President Trump added, “I have certain hatreds of people too.”

It was an illuminating moment, not just an amusing line. Every new administration has factions that end up hating one another despite being on the same official team. But the second Trump White House is starting out with a remarkable degree of open conflict between different individuals, constituencies and worldviews. [...]

For the near term, at least until the Democratic Party gets up off the mat, this means the most important conflicts in American politics are happening within the court of Trump. I’ve already written about one obvious place of potential strife — the broad tension between MAGA populism and Silicon Valley libertarianism. But here are a few more internal wars to watch.

Thus:

• Protectionists vs. Wall Street
• Middle East hawks against realists and doves
• Robert F. Kennedy Jr. vs. the future

Concerning the last:

For instance, after the same OpenAI announcement that inspired Musk’s snarky undermining, Kennedy’s former running mate, Nicole Shanahan, warned Megyn Kelly that the use of A.I. to design new personalized mRNA vaccines, a scenario touted by Oracle’s Larry Ellison at the announcement, “could lead to an extinction event.”

That’s a stark formulation of the potential stakes in a conflict between courtiers. And the odd thing is that there are people on the other side, people working on A.I. right now, who share a version of Shanahan’s crankish-sounding take. They don’t think mRNA tech will kill everyone. But they do suspect, or fear, or hope, that A.I. is ushering in a post-human paradigm, fast.

Which means that what would be, in one sense, the best possible economic news for the Trump administration — a leaping-ahead of A.I. progress — could also make his court the site of existential arguments, a culture war to end all culture wars, that leaves every other issue in its shade.

Photoshop’s Generative AI Rocks!

BEFORE:

AFTER:

For my money, the difference between the first image and the second is (just about) as remarkable as any of the verbal wizardry I’ve seen from Claude or ChatGPT. The visual difference between the two images is easy to see. It’s the sort of thing, where, if you didn’t actually have to figure out how to transform the first into the second, you’d think it could be done in the blink of an eye, with the flick of the wrist.

And now you can, almost. With generative AI it really is almost that easy. Almost, but not quite. We’re not yet to the point where one can say, “computer, clean that up for me,” and it’ll know what you want and be able to do it. While I’m tempted to say we’re not far from it now, I don’t really know. The phrase “clean that up” is doing a lot of work in that command. I’m not sure we’re to the point where a so-called agent powered by a ginormous so-called Foundation Model can do that. That agent might have to be personally trained by the use to know what the scope of that phrase is.

The thing is, I’m not sure we need such an agent. What I actually had to do was not difficult and did not take much time, say two, three, five minutes, and this was my first time through. I had to:

1. pick a tool,
2. set a parameter on the tool, which took an adjustment to get it right,
3. manually trace over the area I wanted altered, which took a little skill, but nothing beyond the reach of anyone capable of using Photoshop at all, and finally,
4. direct Photoshop to execute the operation.

That last took a half-minute to a minute on my machine, which is maybe three years old. It would go much quicker with a new machine. Now, if I had fifty of those to do, in that case, yes, it would be nice to be able to get the job done with a simple command,

Now, if I were a painter with the requisite draftsmanship, I could paint a picture like either of those photos. Neither of them would present a technical problem to such a craftsman. But that craftsman is unlikely to want to execute an image like the first. Why junk up the scene with that intrusive table and what looks like a piece of a bike rack in the background.

Things are a bit different for the photographer. Sure, taking a shot like the second would as easy as the first shot, if those blasted things weren’t in the way. But they were, and that’s a big problem. I would think the problem would have been almost impossible to handle with traditional analog photography. You would have to manually paint over the intrusions. That’s not at all practical. With digital photography things are different. You can easily go in change any pixel you want to. You could, in theory, manually edit the first image so that it comes out like the second. But it would be hellish and time-consuming. There are probably people who can and have done that sort of thing. I hope they get paid and arm and a leg for doing it. But the need for that kind of skill is now all-but-over.

Now, notice that green smudge at the lower right. If I were shooting this for a magazine, I’d probably have to get rid of it. That would be difficult to do and I’m not sure how well the AI would be able to paint the girl’s feet. Perhaps I’ll give it a try some time.

But I’m in no hurry. “Why not?” you ask. “Because I like it there.” “Why, pray tell,” you ask. Because it gives a sense of distance, of space. Whatever it is, probably some kind of plant, it’s between the photographer and the kids. The photographer, that’s me, likes such things. He’ll even deliberately introduce such “defects” into his photos. They’re part of his aesthetic.

In this case, however, I doubt that there was any deliberation. I saw the kids move out of the corner of my eye. So I turned and took the shot. The green blob intruded, which is fine. But so did that table, not so good.

Now, back to the underlying AI tech. As I said up top, the difference between those two images is as remarkable to me as the verbal skills of ChatGPT or Claude. But, and this is very important, you need to understand that, to a first approximation, LLM tech treats language the same way this visual tech treats visual information. LLMs treat language as though it consisted of strings of colored beads. You and I know that those colored beads are in fact letters that spell out words and the spaces and punctuation between words. The AI tech doesn’t “know” that. You and I know that those strings are words, symbols; the AI tech doesn’t know that. As far as it is concerned, you might as well feed it (images of) strings of colored beads.

Remember step 3 above, where I trace over the part of the image I want removed? That’s a prompt. Or rather, that plus the source image constitute a prompt to the AI. It then produces a new image with the changes specified in the prompt. Considered at the appropriate level of abstraction, it’s the same as prompting ChatGPT or Claude to take a sad story as input and return a happy one.

[As Sean Connery said in that movie, “Here endeth the lesson.”]