Eye on the High Line

Dolly Parton, RIP

How LLMs work [in pictures]

We will use everyday language where possible and mark technical terms in purple. Key is that modern LLMs have a component of making answers (in blue, called "inference") and a component of improving answers (red, called "training"). pic.twitter.com/iUmGaNMKUz

— Kording Lab 🦖 (@KordingLab) August 25, 2026

The real work is done by a core machine, the neural network which is used to generate one word at a time based on its inputs. The probability of each word depends on prior words. And words are chosen one by one according to those probabilities. pic.twitter.com/7zWO87TThs

— Kording Lab 🦖 (@KordingLab) August 25, 2026

The result of this are what is sometimes called large-scale "stochastic parrots". Very impressive text but often somewhat superficial. But these are the old LLMs (think 2022). Innovation was super fast since.

— Kording Lab 🦖 (@KordingLab) August 25, 2026

A second idea (B) for making answers, tool use. Call programs like databases, internet search, calculators etc. pic.twitter.com/zyPU00vmV3

— Kording Lab 🦖 (@KordingLab) August 25, 2026

And all of this, is better with better “training“ goals. Wherever truth is defined, push towards this truth (D). And where it is not defined, use LLMs to check answers ("self evaluation", (E)) pic.twitter.com/vLOy2UCsYb

— Kording Lab 🦖 (@KordingLab) August 25, 2026

A great deal of the strengths and weaknesses of today's LLMs can be understood from these component ideas.

— Kording Lab 🦖 (@KordingLab) August 25, 2026

On the waterfront [uptown Hoboken]

We need to consider debt forgiveness in the context of our $40 trillion deficit

Paul Vigna, An Ancient Sumerian Solution to Our $40 Trillion Deficit, Aug. 22, 2026.

The U.S. federal debt has hit $40 trillion. Add the debt owed by U.S. states, corporations and consumers, and the figure rises to about $77 trillion in debt, set against an annual gross domestic product of about $32 trillion. The interest on all of it is compounding constantly. It’s not just the United States, either. Globally, there is $350 trillion in debt, roughly treble global G.D.P. It’s like snowpack on a mountainside. It may look stable right now, but it’s creating the conditions for an avalanche. We are past the point where we can deal with our current debts in normal ways. The options open to us are extremely unlikely or highly destructive: Grow our way out of it, raise taxes, inflate the debt away or wait for the economic fallout.

Ancient societies had another method to deal with debt. It was called an amargi — a blanket declaration of public debt cancellation. All public debts written off. Disappeared. It sounds laughable, I know. But, really, that’s just because the idea has been buried so deeply in history that you’ve probably never heard of it. In the ancient world, it presented a pragmatic solution to an intractable problem. And now, faced with impossible-to-repay debts that are weighing down our economy, is the time to look at the amargi and the lessons it offers about how to think about finance.

Vigna then runs through a quick inventory debt forgiveness in the ancient world.

The reason the practice often worked in the first place was because the ancient world understood something about our monetary system we have mostly forgotten: Money is an invented social construct. It isn’t real, not in the way a tree or a stone is real. The system of money and credit is a thing humans made up. It’s a record-keeping device for distributing resources. And since money is a human creation, we can alter it when needed.

It’s hard to predict what will happen if we don’t address our debt, but it’s not hard to predict that the outcome will be bad. Governments can borrow to cover up other problems for only so long. Sometimes what comes next is a hyperinflationary spiral and economic collapse, as in Zimbabwe or Weimar Germany. Sometimes it’s a national default that rocks financial markets, such as in Egypt and Anatolia under the Ottoman Empire in the 1870s. And sometimes it’s just a steadily slipping quality of life as debt saps people’s ability to build their own wealth.

Money represents resources, or at least access to resources. When so much of it is going to debt repayment, it means money that could be spent on goods or services is diverted. The federal government now spends over $1 trillion a year on debt interest — money that could otherwise be spent on roads, schools or health care. [...] We are unlikely to see a modern amargi, but as a set of principles for reconsidering our relationship with money, its applicability is rich. [...] We need to start seeing money and debt the way people saw it back then: As a system to distribute resources that, like any system, can be periodically reset.

Red flowers on the street corner

Nestled among the beans

Why did Robert Wright (have) to drink the Silicon Valley Kool Aid? [GT-6]

I thought that, when I’d posted notice of my 3 Quarks Daily review of Robert Wright’s The God Test that that was that. But then I read his interview with Liron Shapira, AI Dystopia Is Just 8 Years Away — Robert Wright, Bestselling Author of “The God Test.” Whoops! There’s that phrase again.

The phrase I’m talking about is “reverse engineer.” I believe that the term has been kicking around in psychology for several decades, but I associate it with Steven Pinker’s 1997 book, How the Mind Works (pp. 21-22:

Reverse-engineering is what the boffins at Sony do when a new product is announced by Panasonic, or vice versa. They buy one, bring it back to the lab, take a screwdriver to it, and try to figure out what all the parts are for and how they combine to make the device work. We all engage in reverse-engineering when we face an interesting new gadget. In rummaging through an antique store, we may find a contraption that is inscrutable until we figure out what it was designed to do. When we realize that it is an olive-pitter, we suddenly understand that the metal ring is designed to hold the olive, and the lever lowers an X-shaped blade through one end, pushing the pit out through the other end. The shapes and arrangements of the springs, hinges, blades, levers, and rings all make sense in a satisfying rush of insight.

With that in mind, let’s look at some remarks Wright made to Liron:

Even so, what this approach to training does is, I think, replicate specific cognitive functionality in these machines that exist in the human mind. I don’t mean it does things exactly the way the human mind does. These models have independently invented things that natural selection invented. Edge detection’s a very clear case.

Wright doesn’t use the phrase there, but that’s what he has in mind. Here Wright uses the phrase:

... but even with the current paradigm, you give it kinds of data, visual, auditory, written, whatever, and it basically reverse engineers parts of the human mind that do the transmutation of one form of data into another.

There’s the key phrase. Here’s one last passage. The phrase isn’t here, but the idea certainly is:

You just make the machine good at predicting the next sequence of letters. When you first show it the stuff, it’s pure gibberish, but the machine itself finds a way of mapping the meaning. We do give it the basic... We do say you gotta use vectors to represent the words. It didn’t invent that. But we didn’t say, “By the way, you should choose numbers to fill in the blanks in the vectors that capture this thing we call meaning.” No, it in effect discovered that meaning is a property of words. I would put it that way.

NO.

It’s one thing to talk about psychologists “reverse engineering” bits of human behavior. I have no problem with that. But that’s not what Wright is doing here. He’s talking about machine learning, about transformers, reverse engineering human language and cognition. That’s not at all a useful way of conceptualizing what’s going on. It’s anthropomorphizing. It’s also the kind of mystification that Silicon Valley has been using to hype AI.

Once again I refer you to Berk Idem’s review, where he notes:

My problem is the road he takes in the book. Wright keeps telling the story of AI as if the machines discovered things on their own, that they found the meanings of words, that they grew something like an eye, that they started to evolve, when in fact people set almost all of the machinery up on purpose, with a pretty clear idea of what they were doing and why. I never expected myself to be on the “intelligent design” side of a debate, yet here I am, for instance, arguing that LLMs did not miraculously discover meaning, they were designed to do that. Wright gives too much credit to what models discover during training and too little to the architecture, objective, and research program that produced those discoveries.

Idem is correct. You should consult his review for more details, but I’ll say a couple of words about edge detection in vision and the meaning of words.

Back in 1959 Hubel and Wiesel published their seminal work on the visual cortex of the cat, work that led to their 1981 Nobel Prize in Physiology or Medicine. Virtually all work in machine vision has been informed by that study in one way or another. Machine vision systems are engineered to be sensitive to edges.

The same is true for language. It simply is not true that transformers “in effect discovered that meaning is a property of words.” There is a long tradition within linguistics and computational linguistics of thinking about the meanings of words as a function of the contexts in which words are used, something I review in a recent working paper, The Origins of LLMs – A long tectonic subduction event finally producing a visible volcanic eruption in November 2022 (the subtitle was suggested by ChatGPT). The idea dates back to the 1950s while its computational exploitation dates back to work that Gerard Salton began in the late 1960s on information retrieval. He’s the one who came up with the idea of using vectors to represent linguistic meaning. The transformer is thus a recent elaboration of an idea that’s been around for decades, an idea that human researchers came up for conceptualizing meaning.

* * * * *

There’s much more in Wright’s interview with Liron, much of it interesting. And some of it is bothersome, but I’ve said enough on that score. The God Test is an interesting book. But be careful of what Wright attributes to the machine. You’re better off having no explanation than accepting one that’s misleading at best.

Elsbeth and the tech bro’s panic room [Media Notes 190]

I’ve been watching episodes of Elsbeth off and on. It’s a spin-off of The Good Wife and The Good Fight. Elsbeth Tascioni is an eccentric attorney assigned to the New York Police Department on a consent decree. She functions as a sleuth in each episode. I’m interested in episode 36, “Bunker Down” in season 3 (airing on Nov. 13, 2025). Here’s how Wikipedia characterizes the episode:

Paranoid fintech [financial technology] billionaire Craig Hollis traps his crisis manager Anders Whitman in Craig's state-of-the art panic room, erratically believing that Anders will report his "indiscretions" to his company's board of directors.

Hollis is presented as particularly eccentric and willful in his ways. I couldn’t help but think of this as a deliberately nasty satire of an archetypal billionaire. Perhaps it was Hollis's glee in all the gadgetry incorporated into the room.

High tech billionaires are not particularly rare in movies and TV. But this one somehow struck me as a particularly nasty depiction. Not twirling mustaches nasty – he was clear shaven – but nasty. I’m wondering if we’re in for a rash of such characters.

How Much Would an AI Crash Destroy?

Youtube page:

Two chip stocks recently drove seventeen percent of the entire global stock market's return in a single month — which tells you just how concentrated the AI trade has become, and how exposed the average investor now is without realising it. In this video we look at how much wealth an AI crash could actually destroy, with estimates from Dean Baker, former IMF chief economist Gita Gopinath, and Oliver Wyman running into the tens of trillions of dollars. We cover why the usual places to hide — small caps, value funds, international stocks — are now packed with AI stocks, what the Bank for International Settlements found when it compared today's buildout to the great railway and dot-com bubbles, and why a technology being real has never been enough to protect the people who overpaid for it. This isn't a crash prediction. It's a look at the downside risk, the illusion of diversification, and why boring, unexciting investing tends to win in the end.

Three tablescapes

Cyberpunk City: Hong Kong & Ghost in the Shell

A third transition in science and the evolving cosmos

Kauffman SA, Roli A. (2023) A third transition in science? Interface Focus 13: 20220063. https://doi.org/10.1098/rsfs.2022.0063

Abstract: Since Newton, classical and quantum physics depend upon the‘Newtonian paradigm’. The relevant variables of the system are identified. For example, we identify the position and momentum of classical particles. Laws of motion in differential form connecting the variables are formulated. An example is Newton’s three laws of motion. The boundary conditions creating the phase space of all possible values of the variables are defined. Then, given any initial condition, the differential equations of motion are integrated to yield an entailed trajectory in the prestated phase space. It is fundamental to the Newtonian paradigm that the set of possibilities that constitute the phase space is always definable and fixed ahead of time. This fails for the diachronic evolution of ever-new adaptations in any biosphere. Living cells achieve constraint closure and construct themselves. Thus, living cells, evolving via heritable variation and natural selection, adaptively construct new-in-the-universe possibilities. We can neither define nor deduce the evolving phase space: we can use no mathematics based on set theory to do so. We cannot write or solve differential equations for the diachronic evolution of ever-new adaptations in a biosphere. Evolving biospheres are outside the Newtonian paradigm. There can be no theory of everything that entails all that comes to exist. We face a third major transition in science beyond the Pythagorean dream that‘all is number’ echoed by Newtonian physics. However, we begin to understand the emergent creativity of an evolving biosphere: emergence is not engineering.

* * * * *

Eli Stark-Elster, Earth’s Organisms Developed Via Evolution. Some Theorists Wonder: What if the Entire Cosmos Did, Too? Smithsonian Magazine, August 18, 2026.

Historically, scientists have viewed the cosmos as akin to a rock—complex, perhaps even beautiful, but formed through arbitrary events. Evolutionary cosmologists instead argue that universes, like living organisms, grow and reproduce, spinning off new universes with small variations from their progenitors. In doing so, these cosmic offspring are refined across generations into forms that maximize the number of universes yet to be born. Since the Big Bang, our universe, like any developing child, has been unfurling into an optimal shape.

Under this view, the universe is not a rock. It is an egg.

This analogy comes from Julian Gough, an Irish poet, novelist and musician best known for writing the short story that plays at the end of the video game “Minecraft.” He has recently become a public advocate for and scholar of cosmological evolution. Gough writes a Substack blog, The Egg and the Rock, where he publishes essays, personal updates and predictions about new data from the James Webb Space Telescope.

Over a decade ago, Gough began wrestling with the question of why the complexity of our universe has increased over time. “It goes from … a ball of hot gas to building out structures like stars and galaxies,” he says. From there come planets and, on at least one of them, biological life. “That’s a very strange thing for hot gas to do.”

Maybe, he wondered, the universe evolved into its current form. “It seemed to me an evolutionary explanation was the natural one,” he says. “In every sphere, when we discover a self-ordering, self-complexifying system, it turns out it’s had a previous evolutionary history. And why would that not apply to the universe itself?” [...]

Smolin published his ideas in a 1992 paper titled “Did the Universe Evolve?” as well as a 1997 popular-science book called The Life of the Cosmos. Gough found both and read them excitedly. And yet, it seemed as though virtually nothing had come of Smolin’s ideas since. “Obviously,” says Gough, “[I thought] it must be wrong, because it had been 25 years and nobody seemed to be putting it forward as one of the possible mainstream explanations. Then I dug into the literature and realized—oh, my God—it hasn’t been falsified. It just hasn’t been engaged with properly at all.”

Smolin’s theory had not completely vanished. Some futurist philosophers had extended the idea to explain the emergence of complex life from hot gas and stars. Humans exploit increasingly powerful energy sources, a progression they argued could culminate in artificial black holes. If those black holes spawned new universes, then a cosmos capable of producing technologically advanced life might produce more offspring. The progression from hot gas to stars, life and technology could, they proposed, be a product of cosmic natural selection. [...]

On July 8, 2022, four days before Webb released its first data, Gough published his predictions on Substack. Already aware that his amateur background might make him sound unreliable, he feared being publicly wrong. “I was really terrified,” he says. “This could be fantastically humiliating.”

Fortunately for Gough, the results were the opposite. Webb found galaxies forming earlier, faster and in a more orderly manner than standard physics models had anticipated. In November 2022, NASA reported evidence that some galaxies had begun assembling only about 100 million years after the Big Bang, while follow-up observations of a galaxy just 470 million years after the event found an unusually massive black hole—nearly as massive as all the stars surrounding it.

In a 2025 paper published in the Astrophysical Journal, scientists described these discoveries as a “conundrum [that is] part of the larger challenge to understand the stunning prevalence of massive structures and galaxies in the first few 100 million years after the Big Bang.”

There's more at the link. [H/t Tyler Cowen]

Cf. my article, Welcome to the Fourth Arena – The World is Gifted, 3 Quarks Daily, June 20, 2022.

Friday Fotos: The Mystery that is the Bergen Arches

Algorithmic Grammar of Flexible Cognition

🧠🤖New preprint
Algorithmic Grammar of Flexible Cognition:
A Walk through Latent Operationshttps://t.co/pgBHEw0TtB
Flexible cognition moves adaptively btwn cognitive modes. We formalize it as navigation, a walk over latent operations in neural space as much as the environment. pic.twitter.com/7HesQlvV5c

— Ida Momennejad (@criticalneuro) August 21, 2026

Abstract of the article linked above:

Flexible behavior requires moving adaptively between cognitive modes, between memory and generalization, or cached inference and step-by-step reasoning. Reinforcement learning (RL) offers a language to formalize adaptive behavior in terms of learning and meta-learning over states, actions, policies, and rewards, and neuroscience identifies representational geometries of memory and abstraction. However, the unit of analysis remains representations and a few operations or their tradeoffs. This misses the rich compositional operations commonly asso- ciated with prefrontal-hippocampal interactions. What is missing here includes, first, operations as units of analysis; second, higher order operations that can act on entire cognitive maps and representations, transferring structures, reshaping or merging them; and third, a cognitive and algorithmic grammar for the selection, ordering, and composition of operations. Here we examine the intersection of RL, computational neuroscience, and AI interpretability to identify the tools to address this gap. The latent spaces of transformers offer high-dimensional neural spaces as a testbed for composition of functions. Analyzing the order, branching, recurrence, and composition in sequences of algorithmic operations can help cognitive science identify grammar over algorithmic operations. In the other direction cognitive sciences help shift AI evaluation from benchmarks to adaptive paradigms, and AI architecture from input-output and next-token objectives to setting algorithmic operations and grammar as objectives, e.g., next-primitive-prediction.