Those three guys

Toward a manifesto for the naturalistic analysis of large language models

Here's a comment I just posted in response to a post by Tyler Cowan, GPT and my own career trajectory:

I too have spent a great deal of time using ChatGPT, though to somewhat different ends. I analyze and study texts, and so I'm analyzing ChatGPT's output. That's why, for example, I had it do a Girardian reading of Jaws. I wasn't expecting or hoping that it would come up with new insights. For one thing, it hasn't seen the film, but has to work from commentary widely available on the web, and from, I assume, complete scripts that it consumed during training. I just wanted to see if it could do it at all. I discovered that, yes, it could. And so I directed it to do similar work with A.I. and with the Astro Boy stories.

In a similar fashion I've observed what it does in constructing conversations, dealing with abstract concepts (e.g. justice and charity), ungrammatical or nonsense sentences, and stories. In the case of stories I get it to work variations on one or two basic story ideas: What remains constant from one version to the next, and what changes? Of course, I ask for specific changes; I'm looking to see how ChatGPT implements them.

Why am I doing this?

• I analyze texts; I enjoy doing it. And ChatGPT's texts are complicated enough to be interesting under analysis.
• I want to understand what's going on "under the hood." Yes, we can pop the hood and take a look. People are doing that. But it's difficult; we've got a long way to go. By producing clear and rigorous descriptions of phenomena in ChatGPT's output, I can provide the model mechanics with things to look for.
• We need to develop new benchmarks that are more adequate to the capabilities of LLMs. This work contributes to that.

I'm working toward a coherent and well-defined methodology for what, for lack of a better term, I call the naturalistic investigation of LLMs. We may have created these devices; they are artificial in that sense. But they are also very complex. We don't understand them in the way we understand automobiles or "standard-issue" computer programs, programs handcrafted by programmers (with the aid of various coding tools, now including AIs). These things thus become part of the world in which we live. They are natural, and need to be understood on those terms.

More later.

Three guys singing a song

ChatGPT, stories, and ring-composition

A lot of my interaction with ChatGPT has involved stories, simple stories, but stories nonetheless. Those stories deserve attention in themselves, as opposed to the role they play in defining abstract concepts, which is what I’ve emphasized in my new working paper, Discursive Competence in ChatGPT, Part 1: Talking with Dragons. Let’s start with some hero stories.

Rich asks ChatGPT to tell him a story [Hero-1]

He asked for a story about heroes. This is from an email he sent me:

I tried something similar but wanted to edit the story the way I edit pictures in DALL-E..

1) What is a hero?
2) Tell me a story about a hero.
3) Tell me a story about a hero who is cheated out of the fame and recognition he deserved for his heroic act.
4) Tell me a story about a hero who rescues a girl but then is cheated out of the fame and recognition he deserved for his heroic act. Ultimately the girl reveals the truth.

He didn’t keep the definition ChatGPT gave for hero, but he sent me its responses for the other three. In this post I’ll only be using the basic story he requested in 2. Here it is (I’ve added the paragraph numbers to facilitate reference):

[1] Once upon a time, in a land far, far away, there was a young man named Jack who lived in a small village with his family. Jack was a brave and kind-hearted person, and he was loved by everyone in the village.

[2] One day, a great evil dragon attacked the village, breathing fire and causing destruction wherever it went. The people of the village were terrified and didn't know what to do. But Jack, being the hero that he was, didn't hesitate to step forward and face the dragon.

[3] He armed himself with a sword and shield, and set out to confront the dragon. The battle was fierce, but Jack was determined to protect his village and its people. He fought bravely, dodging the dragon's flames and striking back with his sword.

[4] Finally, after a long and difficult battle, Jack was able to slay the dragon, saving the village and its people from certain destruction. The people cheered and celebrated, grateful to Jack for his bravery and selflessness.

[5] From that day on, Jack was hailed as a hero, and his name was known throughout the land. He lived the rest of his days in peace and happiness, knowing that he had done something truly great.

The story is told in five short paragraphs. Notice that the final paragraph “answers” the first. The first paragraph introduces Jack while the last tells us how he’s perceived when the action is over. Similarly, the fourth answers the second: the evil dragon enters the story in the second paragraph and is vanquished in the fourth. The third paragraph gets us from the second to the fourth, call it the turn: Jack battles the dragon. This story is thus a simple example of ring-composition, one of my hobby horses:

A, B, X, B’, A’

The episodes are arranged symmetrically about a central episode. The reason I’m skipping Rich’s other stories for now is that they are not ring-compositions.

Now, if I repeat Rich’s first two steps, will ChatGPT give me a ring-form story? Let’s find out.

Bill repeats what Rich did (replication?) [Hero-2 and 3]

I decided to begin by asking for a definition of “hero” and then for a story:

What is a hero?

A hero is a person who is admired for their courage, outstanding achievements, or noble qualities. Heroes can be real or fictional, and can take many forms, from a firefighter who risks their life to save others, to a superhero who fights for justice and equality. The concept of a hero is an enduring one that has been present in many cultures and societies throughout history. In modern times, a hero is often someone who stands up for what they believe in and inspires others to do the same.

Tell me a story about a hero.

[1] One time, there was a young boy named Timmy who lived in a small town. Timmy was a very brave and kind-hearted boy, and he always tried to help others in need.

[2] One day, Timmy was walking home from school when he saw a stray dog wandering the streets. The dog looked lost and scared, so Timmy decided to take it home with him. He named the dog Lucky, and they quickly became the best of friends.

[3] But not everyone in town was happy to see Lucky. The town's mayor, Mr. Jones, had always been against stray animals and ordered the town's animal control officer to capture Lucky and take him to the pound. But Timmy wasn't about to let that happen.

[4] He quickly came up with a plan to save Lucky. Timmy and his friends worked together to build a secret shelter for Lucky in the woods outside of town. They even set up a special alarm system to alert them if anyone was coming near the shelter.

[5] Thanks to Timmy's bravery and quick thinking, Lucky was able to live safely in the woods for many years. Timmy visited him every day, and the two of them had many wonderful adventures together.

[6] The people of the town were so impressed by Timmy's actions that they began to see him as a hero. Timmy had shown them that with courage and determination, anyone can make a difference and help those in need.

Notice that this story has six short paragraphs, not five, like Rich’s. Is it a ring-form tale?

That depends on how we analyze it. The turn comes in the fourth paragraph. Evil enters in the third paragraph and is vanquished in the fifth. That’s fine, just like in Rich’s story. The sixth paragraph concludes the story. But what do we do about the first two paragraphs? Do we combine them into a single unit which we can then say is comparable to the first paragraph in Rich’s story?

Now, let’s agree to call Rich’s story Hero-1 and my story Hero-2.

Well, the first paragraph in Hero-1 has no action at all. But the second paragraph in Hero-2 does have some action: Timmy rescues Lucky. So the combined 1&2 in Hero-2 is more complex than the first paragraph in Hero-1. Do we want to combine the second and third paragraphs in Hero-2 and place them in juxtaposition with the second paragraph of Hero-1? I think not, since evil enters Hero-1 at the opening of the second paragraph, but not until paragraph 3 in Hero-2.

So, we’ve got a problem. What kind of problem? Well, we could say that these stories aren’t real because they come to us from a chatbot. I choose not to say that, if only because they same kinds of problems come up in the analysis of stories told by humans. What do we do?

At the moment, nothing. Let’s just note two things: 1) there’s an issue, and 2) the overall arcs of the two stories are pretty much the same. If and when we decide the issue, we should do so on the basis of some principle. At the moment we don’t have enough examples to guess what that principle might be.

Now, since ChatGPT allows you to request further responses to any given prompt, I asked it for another story. Here’s what I got – call it Hero-3:

[1] Once upon a time, in a land far, far away, there was a young princess named Aurora. Aurora was a kind and gentle soul, loved by all who knew her. She had long, golden hair and sparkling blue eyes, and was known for her beautiful singing voice.

[2] One day, a terrible dragon came to the kingdom and began to terrorize the people. The dragon was fierce and powerful, and none of the knights or soldiers were able to defeat it. The people lived in fear, not knowing what to do.

[3] But Aurora was not afraid. She knew that she had to do something to save her people, and so she set out to find the dragon. She traveled for days, braving the cold and the snow, until she finally reached the dragon's lair.

[4] With her heart pounding in her chest, Aurora approached the dragon and began to sing. The dragon was enchanted by her beautiful voice, and it slowly began to calm down. Aurora continued to sing, soothing the beast until it was completely calm.

[5] With the dragon no longer a threat, Aurora returned to the kingdom as a hero. The people cheered and celebrated her bravery, and Aurora was hailed as a champion. She lived happily ever after, knowing that she had saved her kingdom and its people.

Hero-3 is quite similar to the Hero-1. We’ve got five paragraphs, and the paragraphs have the same functional roles in Hero-3 as they do in Hero-1. In both cases evil enters with a dragon. But in Hero-1 a man defeats the dragon with a sword while in Hero-3 a woman defeats the dragon with song. Also, in Hero-1 the dragon breaths fire, but not in Hero-3. Moreover, in Hero-3 the dragon is only pacified while in Hero-1 it is killed. (Lévi-Strauss would love that.)

So, we’ve got two ‘pure’ ring-compositions and one, shall we say, ‘irregular’ ring-composition.

Let’s give it another go.

Blue mug reflecting stair balusters

Global parameters in a neural net: ChatGPT talks to kids

In my post on ChatGPT as a tutor I asked it to explain digestion to a three-year-old. It did so, rather well I thought. I subsequently added an addendum about a NYTimes article where ChatGPT was asked to write stories as a fourth grader and an eighth grader; it did so well enough to fool experts in such matters – though it didn’t fool me (I guessed 9 out or 10), I suspect because I have more experience with the Chatster than those experts did. I’ve included a somewhat edited version of this material in my most recent working paper, Discursive Competence in ChatGPT, Part 1: Talking with Dragons (pp. 53-57).

There I posed this question:

How does ChatGPT modulate its output to fit the knowledge level of a given interlocutor? For that matter, how do we do it, for we do it all the time? [...] It would seem to be some kind of global constraint on the whole system, one that can be set to various values. Correlatively, how must a system of language and cognition be organized so that such constraints can be readily applied?

That’s what I want to think about at the moment.

Humans, of course, grow and learn. We’ve all been 3, 10, and 14, and all ages before and after. Does growth and development somehow leave “traces” that allows us to move back and forth between various levels of development or sophistication? Even if that is true, how is that transmitted to ChatGPT? Afterall, it does not grow or develop in the way humans do. I wouldn’t have those traces, would it?

But it likely has encountered texts directed at various ages during its training. But how does it “shrink wrap” the whole system to fit texts directed at a certain age range? Assuming that’s what it does?

Let us note, however, that vocabulary develops and differentiates over time. Older children and adults use words and syntax not available to younger people. But the constructions younger people use do not drop out of use. This suggests a way to begin thinking about the problem.

Age, however, is not the only source of such global parameters. We have social class, regional and ethnic dialects, we have different occupations, and so forth. An adult expert in quantum mechanics uses one vocabulary when speaking to other experts and a different one when addressing non-experts. It’s a general problem.

I note, finally, that experience is certainly a factor. The wider a person’s range of experience with others the more they are aware of the need to adjust to those differences.

More later.

Friday Fotos: More fog on the river

The Evolution of Peace

Luke Glowacki, The Evolution of Peace, Behavioral and Brain Sciences, Accepted manuscript , pp. 1 - 100 DOI: https://doi.org/10.1017/S0140525X22002862

Abstract: While some species have affiliative and even cooperative interactions between individuals of different social groups, humans are alone in having durable, positive-sum, interdependent relationships across unrelated social groups. Our capacity to have harmonious relationships that cross group boundaries is an important aspect of our species’ success, allowing for the exchange of ideas, materials, and ultimately enabling cumulative cultural evolution. Knowledge about the conditions required for peaceful intergroup relationships is critical for understanding the success of our species and building a more peaceful world. How do humans create harmonious relationships across group boundaries and when did this capacity emerge in the human lineage? Answering these questions involves considering the costs and benefits of intergroup cooperation and aggression, for oneself, one's group, and one's neighbor. Taking a game theoretical perspective provides new insights into the difficulties of removing the threat of war and reveals an ironic logic to peace—the factors that enable peace also facilitate the increased scale and destructiveness of conflict. In what follows, I explore the conditions required for peace, why they are so difficult to achieve, and when we expect peace to have emerged in the human lineage. I argue that intergroup cooperation was an important component of human relationships and a selective force in our species history in the past 300 thousand years. But the preconditions for peace only emerged in the past 100 thousand years and likely coexisted with intermittent intergroup violence which would have also been an important and selective force in our species’ history.

Discursive Competence in ChatGPT, Part 1: Talking with Dragons

Version 1, January 5, 2022

Title above, URLs, abstract, contents, and introduction below:

Academia.edu: https://www.academia.edu/94409729/Discursive_Competence_in_ChatGPT_Part_1_Talking_with_Dragons
SSRN: https://ssrn.com/abstract=4318832
Research Gate: https://www.researchgate.net/publication/
366897197_Discursive_Competence_in_ChatGPT_Part_1_Talking_with_Dragons_Discursive_Competence_in_ChatGPT_Part_1_Talking_with_Dragons

Abstract: Noam Chomsky’s idea of linguistic competence suggests a new approach to understanding how LLMs work. This approach requires careful analysis of text. Such analysis indicates that ChatGPT has explicit control over sophisticated discourse skills: 1) It possesses the capacity to specify high-level structures that regulate the organization of language strings into specific patterns: e.g. conversational turn-taking, story frames, film interpretation, and metalingual definition of abstract concepts. 2) It is capable of analogical reasoning in the interpretation of films and stories, such as Spielberg’s Jaws and A.I., and Tezuka’s Astro Boy stories. It must establish an analogy between some abstract interpretive theory (e.g. the ideas of Rene Girard) and people and events in a story. 3) It has some understanding of abstract concepts such as justice and charity. Such concepts can be defined over concepts that exhibit them (metalingual definition). ChatGPT recognizes suitable stories and can revise them. 4) ChatGPT can adjust its level of discourse to accommodate children of various ages. Finally, much of ChatGPT’s discourse seems formulaic in a way similar to what Parry/Lord found in oral epic.

Contents

Introduction: Walking among dragons 2
What is in the rest of this document? 6
Calibration: Understanding a Seinfeld Bit 9
Conversing with ChatGPT about Jaws, Mimetic Desire, and Sacrifice 12
ChatGPT on Spielberg’s A.I. Artificial Intelligence and AI Alignment 23
Extra! Extra! In a discussion about Astro Boy, ChatGPT defends the rights of robots and advanced AI 27
Pumpkins, the Falcon Heavy, and Groucho Marx: High level discourse structure in ChatGPT 30
High level discourse structure in ChatGPT: Part 2 [Quasi-symbolic?] 37
Abstract concepts and metalingual definition: Does ChatGPT understand justice and charity? 42
Does ChatGPT’s performance warrant working on a tutor for children? 53
To the future and beyond 59
Coda: What’s going to be in Part 2: A Framework for Description and Analysis? 65 Appendix: ChatGPT gets confused about Sonnet 129 68

Introduction: Walking among dragons

“Your brain, your job, and your most fundamental beliefs will be challenged by AI like nothing ever before. Make sure you understand it, how it works, and where and how it is being used.” – David Ferrucci

When I first heard about ChatGPT on November 30, 2022, I figured I’d pass on it. After all, it was ultimately based on GPT-3 and I’d already had a little bit of fun with that, albeit through an intermediary. What more could there be? The next day, however, I thought, Why not, it’s free, no? I signed up for an account. I had no particular intentions. I just wanted to test the water.

Total Immersion

I’ve been swimming in it since then. I’ve copied every “conversation” into a text document that is 178 pages long. I can’t tell you how many times I’ve laughed out loud and danced in my seat in reaction to ChatGPT’s response to a prompt.

ChatGPT is more fun than a barrel of monkeys. But it is also work. When I started playing with it I had no specific intentions; I certainly did not intend to write about ChatGPT extensively. I just wanted to poke around. I became, if not hooked, perhaps entranced. I began systematically exploring it, not to find its faults, its weakness, as many are doing, but to test its strengths.

In the process I have changed, though it is difficult to characterize that change. I wouldn’t say that it has added given me any new indeas. Nor has it changed my views on the strengths and weakness of so-called deep learning (DL) technology. It is not an INTELLECTUAL change.

Like many others I have believed DL needs to be augmented by “classical” symbolic processing. I still believe that. I also believed that DL systems need direct interaction with the world if they are to exhibit real “intelligence.” That belief remains rock-solid. Some enthusiats have been saying that DL will take us all the way to full Artificial General Intelligence (AGI) simply by scaling up: more parameters, more data, and more compute. I disagree. Deeper changes are required.

What has changed is my ORIENTATION, my outlook. I have had a glimpse, however limited and provisional, into a new world, a world where we will be working with these “miracles of rare device” (to borrow a phrase from Coleridge) in ways we had not previously imagined. If you want to know what it’s like to drive a car, you can only do it from the driver’s seat. I have taken the driver’s seat and have been systematically exploring ChatGPT.

It is one thing to be amazed by this or that output from ChatGPT. There’s Lots of that going around. I am going beyond that to analyze some of the mechanims of discursive competence, to borrow a term from Noam Chomsky, that enable ChatGPT to function so well. This working paper is a preliminary report on these explorations. I believe that through the careful analysis of ChatGPT’s discursive output we can gain insight into its inner operations, allowing us to improve future technology and to develop benchmarks more tailored to the capacities of emerging LLM technology.

I responded to GPT-3 with a report entitled, GPT-3: Waterloo or Rubicon? Here be Dragons. I have crossed the Rubicon and have been walking among dragons. It is time we get to know them better, to talk with them.

To learn about dragons, describe and analyze them

When I started playing with ChatGPT on December 1, 2022, I had no specific intentions. I wanted to poke around, see what I could see, and then...As I said, I had no specific intentions. I certainly did not intend to spend hours interacting with it to produce a Microsoft Word document currently (1.5.22) containing 61,580 words of transcription – the vast majority from ChatGPT – on 178 pages.

One of the earliest things I did with ChatGPT – not THE first, it was my third session, on December 1, 2022 ¬– was to dialog about Steven Spielberg’s Jaws and the ideas of Rene Girard. I took that and wrote it up for 3 Quarks Daily. Then I had some fun with “Kubla Khan,” quizzed it about trumpets, had a long session about Gojira/Godzilla, and then returned to Spielberg, this time to A.I. Artificial Intelligence. By this time I was developing a feel for how ChatGPT responded. Both the Jaws and the A.I. posts are included in this paper.

I became more systematic, looking for specific things, testing them out. That led to a post with a rather baroque title, “Of pumpkins, the Falcon Heavy, and Groucho Marx: High level discourse structure in ChatGPT,” which I’ve also included in this paper. In that post I advanced the argument that there are parameters in the language model that govern ligher level discourse structures independently of the specific words and strings that realize them.

The alternation pattern is something like this:

A, B, A, B....

That can be repeated as often as one will. The text in the A sections is always drawn from one body of material while the text in the B sections is drawn from a different body of material. That’s the pattern ChatGPT has learned. Where is it in the net? How’s it encoded.

The frame structure is a bit more complicated:

A (B, C, B, C....) A’

The embedded alternation draws on two bodies of material, any two bodies. The second part of the frame, A’, must complement the first, A.

Again, it’s not a complex structure. But it’s not defined directly over particular words. It’s defined over groups of words, placing the groups, not the individual words, into specified relationships in the discourse string.

I then suggested that the patterns I had identified in Jaws and A.I. where similar, but, if anything, more complex.

I had become all but convinced that ChatGPT had explicit control over high-level discourse properties. When humans make statements like those, we take it as obvious that they have some “grammar” of high-level discourse structures. Narratologists, linguists, and psycholinguists study them. But ChatGPT is not a human. It is, shall we say, a machine, a machine that was trained to guess the next word, word after word after word....and so forth, for jillions of texts. All that’s in the resulting model is statistics about those texts. It’s seems to be a “stochastic parrot”, as one well-know paper argued.

Perhaps, in a sense, that is a true. But that is a terribly reductive characterization, and, I have come to believe, all but beside the point. Large language models issue one word at a time for the same reason that humans do: That’s the nature of the communication channel, and tells us relatively little about the device that is pushing words through the channel. LLMs develop rich and complicated structures of parameter weights during the training process. Yes, those structures are statistical in nature, but they are also structures. Perhaps there are aspects of those structures that we can investigate without having to “open the hood” and examine parameter weights.

I made that suggestion in a post, “Abstract concepts and metalingual definition: Does ChatGPT understand justice and charity?”, also included in this paper. Chomsky famously distinguished between competence and performance, where the study of linguistic performance is about the mechanism that produces and understands texts while the study of linguistic competence is about the structure of the texts independent of underlying mechanisms. When I analyze ChatGPT’s output I am investigating its competence. When researchers pop the hood and examine parameter weights, they are investigating performance mechanisms. I further suggest that a better understanding of an LLM’s competence will aid in studying those performance mechanisms by giving us clues about what they are doing.

Nor am I the only one who believes in the value of studying the output of these engines. Others have come to that conclusion as well, though perhaps not quite in those terms. Here is the abstract of a recent preprint from Marcel Binz and Eric Schulz from the Max Planck Institute:

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.[1]

My methods are different, but my purpose is the same, “to study increasingly capable and opaque artificial agents” and thus to render them less opaque. The insights we gain thereby will aid us to improve the capabilities of the next generation of artificial agents. 

* * * * *

[1] Marcel Binz and Eric Schulz, 2022. “Using Cognitive Psychology to Understand GPT-3,” PsyArXiv, June 21, 2022. doi:10.31234/osf.io/6dfgk.

Sill looking for an honest man

Here we go again, Scott Alexander hoists another Bay Area house party

Scott Alexander, Even More Bay Area House Party, Astral Codex Ten, Jan. 3, 2022.

“I know this kind of socialism isn’t popular, here in the Bay,” she tells you. “But the fact is, we live in a world where a tiny number of people have an outsized amount of power. We - by which I mean the loose network of left-wing radicals - tried our best to solve it politically. But we couldn’t. So now we try a different tactic. Whenever a tech billionaire is at the top of his game - raking in billions of dollars, buying up newspapers, considering a run for Governor - we send him an invitation to the Innovation Forum. He’s honored! He always thought of himself as a practical man of business, not the sort of person who goes to Innovation Forums and pontificates to thought leaders about the future of mankind. But maybe it’s just that the world is finally recognizing his genius! So he clears his calendar for a few days, puts off inventing a new superlaser, and attends the Forum. And then a little later, he gets an invitation to the Tech Futures Colloquium, and he thinks - yes, I deserve to go to that one too. And so he clears his next week’s calendar . . .”

“And then you invite them to so many Colloquia that they don’t have time to accrue more money and power?” “You’d be surprised how few it takes! At that level, you spend most of your time just keeping on top of what you’re already doing. A Forum or two a month is enough to bring most billionaires from actively expanding their influence to just treading water. Sometimes they even get addicted. Like Peter Thiel; I don’t think the poor dear has made a business decision in years.”

“I can’t believe people fall for that,” you say.

“Ego is a hell of a drug,” says Vinaya. “It always works. Except for Elon Musk, somehow he always turns down our invitations.”

There’s more at the link.

Fog on the docks

Incremental research has been on the rise between 1945 and 2010

Park, M., Leahey, E. & Funk, R.J. Papers and patents are becoming less disruptive over time. Nature 613, 138–144 (2023). https://doi.org/10.1038/s41586-022-05543-x

Abstract: Theories of scientific and technological change view discovery and invention as endogenous processes, wherein previous accumulated knowledge enables future progress by allowing researchers to, in Newton’s words, ‘stand on the shoulders of giants. Recent decades have witnessed exponential growth in the volume of new scientific and technological knowledge, thereby creating conditions that should be ripe for major advances. Yet contrary to this view, studies suggest that progress is slowing in several major fields. Here, we analyse these claims at scale across six decades, using data on 45 million papers and 3.9 million patents from six large-scale datasets, together with a new quantitative metric—the CD index—that characterizes how papers and patents change networks of citations in science and technology. We find that papers and patents are increasingly less likely to break with the past in ways that push science and technology in new directions. This pattern holds universally across fields and is robust across multiple different citation- and text-based metrics. Subsequently, we link this decline in disruptiveness to a narrowing in the use of previous knowledge, allowing us to reconcile the patterns we observe with the ‘shoulders of giants’ view. We find that the observed declines are unlikely to be driven by changes in the quality of published science, citation practices or field-specific factors. Overall, our results suggest that slowing rates of disruption may reflect a fundamental shift in the nature of science and technology.

Main

Although the past century witnessed an unprecedented expansion of scientific and technological knowledge, there are concerns that innovative activity is slowing. Studies document declining research productivity in semiconductors, pharmaceuticals and other fields. Papers, patents and even grant applications have become less novel relative to prior work and less likely to connect disparate areas of knowledge, both of which are precursors of innovation. The gap between the year of discovery and the awarding of a Nobel Prize has also increased, suggesting that today’s contributions do not measure up to the past. These trends have attracted increasing attention from policymakers, as they pose substantial threats to economic growth, human health and wellbeing, and national security, along with global efforts to combat grand challenges such as climate change.

Numerous explanations for this slowdown have been proposed. Some point to a dearth of ‘low-hanging fruit’ as the readily available productivity-enhancing innovations have already been made. Others emphasize the increasing burden of knowledge; scientists and inventors require ever more training to reach the frontiers of their fields, leaving less time to push those frontiers forward. Yet much remains unknown, not merely about the causes of slowing innovative activity, but also the depth and breadth of the phenomenon. The decline is difficult to reconcile with centuries of observation by philosophers of science, who characterize the growth of knowledge as an endogenous process, wherein previous knowledge enables future discovery, a view captured famously in Newton’s observation that if he had seen further, it was by ‘standing on the shoulders of giants’. Moreover, to date, the evidence pointing to a slowdown is based on studies of particular fields, using disparate and domain-specific metrics, making it difficult to know whether the changes are happening at similar rates across areas of science and technology. Little is also known about whether the patterns seen in aggregate indicators mask differences in the degree to which individual works push the frontier.

We address these gaps in understanding by analysing 25 million papers (1945–2010) in the Web of Science (WoS) (Methods) and 3.9 million patents (1976–2010) in the United States Patent and Trademark Office’s (USPTO) Patents View database (Methods). The WoS data include 390 million citations, 25 million paper titles and 13 million abstracts. The Patents View data include 35 million citations, 3.9 million patent titles and 3.9 million abstracts. Subsequently, we replicate our core findings on four additional datasets—JSTOR, the American Physical Society corpus, Microsoft Academic Graph and PubMed—encompassing 20 million papers. Using these data, we join a new citation-based measure with textual analyses of titles and abstracts to understand whether papers and patents forge new directions over time and across fields.

H/t Tyler Cowen.

OpenAI CEO Sam Altman: AI for the Next Era [the middle layer]

The whole interview is interesting, but I was particularly struck by Altman’s comments about a “middle layer” model, between the so-called Foundation models and specific end-using applications. These remarks start at about c. 2:55:

I think there will be a small handful of like fundamental large models out there that other people build on. But right now what happens is you know company makes large language model API other people build on top of it. And I think there will be a middle layer that becomes really important where uh...I'm like skeptical of all of the startups that are trying to sort of train their own models. I don't think that's going to keep going.

But what I think will happen is there'll be a whole new set of startups that take an existing very large model of the future and tune it, uh which is not just fine-tuning like all the things you can do. I think there will be a lot of access provided to create the model for medicine or using a computer or like the kind of like friend, or whatever. And then those those companies will create a lot of enduring value because they will have like a special version of... They won't have to have created the base model but they will have created something they can use just for themselves or share with others that has this unique data flywheel going that sort of improves over time and all of that. So I think there will be a lot of value created in that middle layer.

Near the end we get a question from the audience at about 33:34:

Question: How could a large language model startup differentiate from another?

Altman: I think it'll be this middle layer um I think in some sense the startups will train their own models, just not from the beginning. uh They will take like you know base models that are like hugely trained with a gigantic amount of compute and data and then they will train on top of those to create you know the model for each vertical. And that those startups so in some sense they are training their own models just not from scratch but they're doing the one percent of training that really matters for for whatever this use case is going to be. Those startups I think they will be hugely successful and very differentiated startups there.

Let's take a walk on the beach