The Origins of LLMs – A long tectonic subduction event finally producing a visible volcanic eruption in November 2022

A new working paper. Title above, link, abstract, contents, and introduction below.

Academic.edu: https://www.academia.edu/167895960/The_Origins_of_LLMs_A_long_tectonic_subduction_event_finally_producing_a_visible_volcanic_eruption_in_November_2022

Abstract: Large language models are the result of a rich research tradition stretching back to the 1950s. This tradition involves a network of researcher doing work in the following: classical MT and computational linguistics → symbolic/statistical/network alternatives → associative and distributed memory models → statistical MT and vector-space methods → neural MT → Transformers → LLMs.

Contents

Introduction: The tangled web of ideas resulting in LLMs 1
Google Translate, a capsule history 2
Vector semantics 5
Firth and distributional semantics 7
Contra Cowen 9
Phase shift 11
Kuhnian paradigm shift 13
Associative memory 15
Sydney Lamb and associative memory for PCs 17
Principles and Development of Natural Intelligence 18

Introduction: The tangled web of ideas resulting in LLMs

As part of my ongoing investigation of Tyler Cowen’s recent monograph, The Marginal Revolution: Rise and Decline, and the Pending AI Revolution (2026), I’ve been thinking about large language models (LLMs), which he discusses in Chapter 4, “Why Marginalism Will Dwindle, and What Will Replace It?” For the most part Cowen presents his readers with the Silicon Valley view: Just as Athena emerged fully-formed from the head of Zeus, so large language models emerged fully-formed from Silicon Valley laboratories in November of 2022. And that IS how things appeared to the public at large. After seeing AIs in science fiction movies, and reading about them for years, all of a sudden, here they are, out of nothing, on the web in the form of ChatGPT.

For all practical purposes, Cowen is a member of the public. He may have been following developments for years. Given his interest in chess I’m sure he followed that story at least since IBM’s Deep Blue beat Kasparov in 1997. And AI plays an important role in his 2013 book, Average is Over. Beyond that, he has contacts in Silicon Valley going back I don’t know how long. But this is not his intellectual field, which is centered on economics. It’s one thing to read about it, to talk with researchers and entrepreneurs, it’s something else to conduct research and publish.

I am in a different position. While my Ph.D. is in English Literature, my dissertation – “Cognitive Science and Literary Theory” – is as much about knowledge representation and computational linguistics as it is about literature. I was trained in that are by the late David G. Hays, who was a first-generation researcher in computational linguistics with the RAND Corporation in the 1950s and 1960s. For the last three years I’ve been conducting research in the behavior of LLMs and have been collaborating with Ramesh Viswanathan, and expert in machine vision and cognitive science at Goethe University Frankfurt.

THAT, broadly speaking, is my field. While I wouldn’t expect Cowen’s views to be the same as mine, I would have been happier if he had at least acknowledged that there the future of LLMs and their adequacy if a matter of controversy among the experts. He should have at least mentioned Gary Marcus, Yann LeCun, Fei-Fei Li, and Melanie Mitchell. He might even have mentioned that Ilya Sutskever, a student of Geoffrey Hinton who was on the OpenAI team that developed the GPT series, that Sutskever has abandoned the idea that pure scaling is the royal road to artificial general intelligence (AGI, whatever that is). Cowen has done none of this. To read him you’d think that the basic scientific and engineering issues have been settled and it’s full speed ahead – “To infinity and beyond,” to quote Buzz Lightyear.

I don’t know quite what I’m going to say about this in the piece I’m writing for my series on the marginalism monograph. I don’t want to recount the full history, for which this working paper can serve as an outline. At the very least I will point out that matters are by no means settled and that there is a statistical tradition with roots in 1950s linguistics and 1960s document retrieval that can serve as a tertium quid between GOFAI (good old fashioned AR) and computational linguistics on the one hand and neural-network based learning on the other.

A Kuhnian paradigm shift?

On page 14 I have a section entitled, “A Kuhnian paradigm shift, or only an invitation to one?” That’s not the original title, which was simply, “Kuhnian paradigm shift.” Why the change?

Simple. There certainly was a dramatic change in the wake of ChatGPT. But I think that change was mostly institutional, in the deployment of resources, the development of institutions, the proliferation of roles in institutions, and of training. It’s not at all clear to me that there was a Gestalt change in anyone’s conceptions about the nature of intelligence in machines, or in humans for that matter. For it is Gestalt switch, a reconfiguration of understanding, that is the hallmark of a paradigm change in Kuhn’s conception of an intellectual revolution. It is not at all clear to me that there was a widespread change comparable to going from a mentality where one sees the Morning Star and Evening Star as two different entities to a mentality where one sees them as two manifestations of a single entity, the planet Venus.

Perhaps something like that has happened here and there, but I suspect that, for the most part, everyone from the most senior researchers through the general public sees the world as composed of the same kinds of entities as processes as they saw before encountering ChatGPT, or GPT-3. Those who think we’re well on the road to AGI (artificial general intelligence) still think of AGI the way they did in, say, 2015 or 2021, as the case may be. The same is true for those who doubt that we’re on that road. All that is changed is people’s awareness of the behavior displayed by the devices we have created. Their sense of what those devices are, what they deeply and essentially are, that hasn’t changed.

Though it may be under tension. The fact is, whatever any of us believes, we don’t really know why kind of behaviors these devices will be exhibiting next year, two years after that, or in ten years. There are a lot of open questions hanging in the air. I suspect that once those questions are resolved, that is, if and when they are, then we will see genuine changes in mentality.

I regard the matter as open to discovery and investigation.

Beyond all that, well, you can read through the rest of this document, which records a dialog I had with ChatGPT (May 29, 2026) beyond the asterisks. I begin by asking ChatGPT to review the history of Google Translate. Why? Because language is the through line. The computational study of language began in the 1950s with the problem of machine translation, translating a text from one natural language to another. The technique used by LLMs for capturing single-word semantics has its origins in statistical methods for document retrieval that originated in the 1960s and 1970s. Google Translate switched to neural-net technology. A year later the transformer was invented in a Google lab. The transformer, as you know, is the engine used to create current large language models. Google Translate is a natural starting point.

Do LLMs require statistical foundations?

Weijie Su, Do Large Language Models (Really) Need Statistical Foundations?, arXiv:2505.19145v2 [stat.ME], June 2, 2025.

Abstract: Large language models (LLMs) represent a new paradigm for processing unstructured data, with applications across an unprecedented range of domains. In this paper, we address, through two arguments, whether the development and application of LLMs would genuinely benefit from foundational contributions from the statistics discipline. First, we argue affirmatively, beginning with the observation that LLMs are inherently statistical models due to their profound data dependency and stochastic generation processes, where statistical insights are naturally essential for handling variability and uncertainty. Second, we argue that the persistent black-box nature of LLMs -- stemming from their immense scale, architectural complexity, and development practices often prioritizing empirical performance over theoretical interpretability -- renders closed-form or purely mechanistic analyses generally intractable, thereby necessitating statistical approaches due to their flexibility and often demonstrated effectiveness. To substantiate these arguments, the paper outlines several research areas -- including alignment, watermarking, uncertainty quantification, evaluation, and data mixture optimization -- where statistical methodologies are critically needed and are already beginning to make valuable contributions. We conclude with a discussion suggesting that statistical research concerning LLMs will likely form a diverse “mosaic” of specialized topics rather than deriving from a single unifying theory, and highlighting the importance of timely engagement by our statistics community in LLM research.

H/t Jessica Hullman:

Something a bit cringey that becomes clearer when you see the various statistical challenges laid out like this is that sometimes they arise not just because LLMs are too complex for us to understand, but also because they are proprietary objects. E.g., once a large LLM has been trained, it’s been found that it can be more efficient to distill its knowledge into a smaller model than to train a smaller model from scratch. This motivates developers of big models to figure out ways to make their outputs resistant to distillation by competitors. It’s all just statistics I suppose, but I’d much prefer to work on problems like uncertainty quantification or watermarking outputs than how to resist sharing knowledge! Similarly, secrecy around training data curation can make it harder to theorize about dependencies between data mixtures and model capabilities.

In reading this alongside other recent takes on the state of stats in ML, it’s interesting to me that despite a growing consensus that we need to develop interpretable models to make sense of LLMs, there still seems to be a contingent of ML researchers who dismiss further integration of classical stats. For example, Su cites evaluation of LLMs as a place where we need statistically grounded methods to avoid an evaluation crisis with similarities to the replication crisis in social science, where researchers game the evaluations they present (there are various reasons to worry about this, some of which we summarized here a few years ago). But others refer to attempts to incentivize more thorough reporting of uncertainty in ML evaluation as “a weird obsession with statistics.” What’s up with that, I wonder?