ChatGPT critiques benchmarks as a measure of LLM performance and then elaborates on my whaling analogy for what’s wrong with the AI business.

I just read this article: Kevin Roose, When A.I. Passes This Test, Look Out, NYTimes, Jan. 23, 2025.

“Fiddlesticks! Fiddlesticks and moonbeams!” said I to myself. “Not a benchmark-to-end-all-benchmarks. Will these people never learn? Why do they keep looking for the house keys under the streetlight when they lost them at the front door? Are they drunk or something?” Oh, they don’t know what they’re doing, do they?

So I put the article aside, intending to write up a post about it and to register my complaint about the inadequacy of benchmarks. Then I had an idea: I’ve not used ChatGPT in a while, so why don’t I have it summarize the article for me.” Once the Chatster had done that, I had an idea: Why not ask it what it thinks about bench-marking?

I didn’t have any particular expectations about what kind of answer it would give. To my surprise and delight it came up with a substantial critique. I decided to press on and told the story about how I came to write about Spielberg’s Jaws. Again, the Chatster rose to the occasion and wrote a mini-essay, “The Role of Discovery in Intellectual Work.” It was on point, so I gave it my whaling analogy. Chatster broke it down very nicely.

“Damn!” thought I to myself. “Chatster’s on FIRE!”

And away we go. 

NOTE: See my post, Competence and Performance in Bench-marking LLMs.

* * * * *

Summarize this article, please. Assume I am sophisticated and well-educated.

The article, "A Test So Hard No AI System Can Pass It — Yet," examines the development and significance of a new benchmark called "Humanity’s Last Exam," designed to evaluate the limits of artificial intelligence (AI) capabilities. Created by Dan Hendrycks and his team at the Center for AI Safety in collaboration with Scale AI, the test comprises 3,000 highly challenging questions spanning disciplines like philosophy, physics, and engineering, contributed by experts. Its goal is to assess AI's ability to perform complex intellectual tasks, approximating a general intelligence score.

The impetus for this exam arises from the rapid advancements in AI, which have rendered traditional benchmarks obsolete as models like those from OpenAI and Google surpass graduate-level challenges. Despite their advanced abilities, current AI models failed Humanity’s Last Exam, with the best-performing model achieving only 8.3%. Hendrycks anticipates significant improvement, potentially marking a point where AI systems surpass human experts in answering complex questions.

The article highlights concerns over AI's uneven progress—astounding in specialized tasks but deficient in others—and discusses the broader implications for fields requiring nuanced problem-solving beyond exams, such as research. It concludes with a call for innovative methods to measure AI’s impact beyond standardized testing, particularly as these systems continue to evolve.

What do you think of this whole “benchmark” business? How useful are they?

Benchmarks like Humanity's Last Exam have value, but their utility is nuanced. Here’s my take:

Strengths of Benchmarks:

• Tracking Progress: Benchmarks serve as yardsticks, offering a quantitative way to measure the trajectory of AI systems over time. They allow researchers to compare different models and identify strengths and weaknesses systematically.  
• Challenge Motivation: By creating hard problems, benchmarks inspire innovation. They push AI researchers to design models that can tackle tasks requiring deeper reasoning, creativity, and generalization.  
• Defining Boundaries: Benchmarks like this can help delineate areas where AI falls short, highlighting tasks that still require human expertise, such as novel discovery or conceptual synthesis.

Limitations and Challenges:

• Artificial Metrics: Benchmarks may overfit AI development to specific tasks, leading to “training to the test” rather than true generalization. Models can excel at benchmarks without demonstrating real-world problem-solving abilities.  
• Narrow Framing: Many benchmarks, including even sophisticated ones, focus on clearly-defined problems with deterministic answers. But much of human intellectual labor—creative writing, ethical decision-making, or forming new scientific hypotheses—doesn’t fit neatly into these molds. 
• Temporary Relevance: As seen with earlier AI challenges, benchmarks can become obsolete quickly. Once AI models surpass them, they lose their diagnostic power, necessitating constant updates or more difficult tests.  
• Underestimating Complexity: Benchmarks often fail to account for the broader, messier contexts in which humans operate. AI might pass a theoretical test on physics but struggle to apply those principles in practical, experimental settings.

Broader Perspective:

Benchmarks are useful for building a snapshot of what AI can do at any moment, but they must evolve to remain meaningful. To go beyond mere scoring, we might need:

• Dynamic Benchmarks: Tasks that adapt to the model’s performance, such as questions whose answers are not yet known or which require multi-modal reasoning (e.g., combining visual and textual information).  
• Real-World Applications: Shifting focus from exams to practical contributions, like assessing AI’s impact on medicine, climate science, or engineering breakthroughs. 
• Human-Centric Evaluation: Considering AI’s utility in collaborative settings, where it aids but does not replace human expertise.

Benchmarks like Humanity’s Last Exam have an important place in measuring AI progress. However, we must also look beyond them to gauge AI’s real-world significance and its potential to address humanity’s most pressing challenges. They’re a piece of the puzzle, not the whole picture.

* * * * *

I agree. The problem I have with benchmarks is that it asks for something specific. But that’s not always how life is lived. Often we have to figure out whether there’s a problem or something of interest. Once we’ve done that, we have to figure out an approach.

I’m trained as a literary and cultural critic. I read a lot of books, see a lot of films and TV shows. I write about many of these texts, broadly considered. It’s up to me to notice that there’s something interesting about a text. For example, a couple of years ago I was watching Steven Spielberg’s movie “Jaws.” After I had watched the movie I read the Wikipedia entry, which mentioned various interpretations. It also mentioned that the sequels were not as good. So I looked at them (couldn’t finish 3 and 4). All the while I was thinking.

I started comparing Jaws 2 with the original. The original was much tighter. I started thinking about the role of Quint. “Why did he have to die?” I asked myself. Sure, he wasn’t a nice man, but he hadn’t hurt anyone. All that had to happen to give the movie a “happy” ending was that the shark gets killed. It wasn’t necessary to have anyone else die. So, what’s the point of having Quint die? How would the film be different if he hadn’t been killed by the shark?

It was at that point that it hit me: Girard, sacrifice, mimetic desire. NOW I had a specific question: How do Girard’s ideas apply to Jaws? It took me awhile to come up with an answer that I liked. But that’s how it goes. The really important thing was coming up with the idea to apply Girard’s ideas to the movie. That’s what took the hard work.

These benchmarks don’t test that kind of work at all. But that’s where the really hard and fruitful problems lie.

You’ve pinpointed a profound challenge in evaluating AI: life—and much of human intellectual labor—is far messier and more open-ended than what benchmarks can capture. Your example of watching Jaws illustrates exactly the kind of complex, emergent thinking that is central to creative, critical, and interpretive work. Let’s unpack your critique a bit further.

The horror! The horror! ChatGPT gets lost while summarizing Heart of Darkness

I have already reported that ChatGPT failed in an attempt to summarize Sir Gawain and the Green Knight. Heart of Darkness is a very different text, but similar in length (roughly 40K words). I note that while we do not think of summarizing a text as something requiring creativity, it is difficult to do well.

Note that I have blogged extensively about Heart of Darkness and have gathered many of those posts into a working paper: Heart of Darkness: Qualitative and Quantitative Analysis on Several Scales, https://www.academia.edu/8132174/Heart_of_Darkness_Qualitative_and_Quantitative_Analysis_on_Several_Scales_Version_3.

Could you summarize Heart of Darkness.

Heart of Darkness is a novella by Joseph Conrad that tells the story of a man named Charles Marlow, who is sent on a journey up the Congo River to find and bring back a rogue trader named Kurtz. Along the way, Marlow encounters the brutal realities of colonial exploitation and the darkness of the human soul. The novella is considered a classic of literature and is widely studied for its themes of imperialism, madness, and the search for meaning.

That’s a bit short. The final sentence tells us almost nothing about what happens. Let’s dig in for more information:

How does the story begin?

The story begins with the narrator, Charles Marlow, telling his story to a group of men on a boat anchored on the Thames River in London. He begins by describing how he got the job as a riverboat captain in the Congo Free State, a colony controlled by a private company. He sets off on a journey up the Congo River to meet Kurtz, an enigmatic figure who has become a god-like figure to the local people and has a reputation as a brilliant and gifted man. However, as Marlow travels deeper into the jungle, he begins to uncover the brutal realities of colonial exploitation and the darkness that lies within the human soul. He also hears stories about Kurtz and his descent into madness. The story starts with Marlow's journey and his reflections on the journey and the people he meets.

How does he get his job?

Marlow gets his job as a riverboat captain in the Congo Free State through an acquaintance of his aunt. The company that controlled the colony was in need of a captain for one of their steamboats, and Marlow's aunt's acquaintance, who was a Director of the company, recommends him for the job. Marlow is not particularly qualified for the position, but he is eager to take on the challenge and sees it as an opportunity to satisfy his curiosity about the Congo and the mysterious Kurtz. He is also drawn to the idea of traveling to the "blank spaces" on the map, the unknown and unexplored areas of the world.

What happens on his trip upriver?

On his trip up the Congo River, Marlow encounters a number of obstacles and challenges. He first has to deal with the treacherous and unpredictable nature of the river itself, which is full of rapids and obstacles. He also has to deal with the heat and the dense jungle, which make the journey physically demanding.

As he travels deeper into the jungle, Marlow also encounters the brutal realities of colonialism. He sees firsthand the exploitation and oppression of the native people by the European colonizers, and the greed and corruption that drive the colonial enterprise. He also witnesses the dehumanizing effects of this system on both the colonizers and the colonized.

Marlow also meets a number of people on his journey, including a group of European traders and adventurers, who are all in search of ivory and other resources. These men are portrayed as being morally bankrupt and unfeeling, driven only by their greed.

As Marlow gets closer to his destination, he begins to hear more and more about Kurtz and his descent into madness. Marlow becomes increasingly obsessed with meeting Kurtz and understanding the truth about him. In the end, Marlow finally meets Kurtz and is shocked by what he finds. Kurtz, who was once a brilliant and idealistic man, has become a tyrant, using the native people for his own gain and imposing his will through fear and violence. The meeting with Kurtz is the climax of the story, which also marks the end of Marlow's journey.

That last sentence is dead wrong. In the rest of this interaction ChatGPT keeps making errors and I keep guiding it back on track. Finally, I just give up.

Really? I thought his journey ended when he returned from the Congo and talked with someone? Who did he talk with?

You are correct that Marlow's journey to the Congo ends when he returns and his story is being told to a group of men on a boat anchored on the Thames River in London. The men listening to Marlow's story are not identified, but they are described as being "ordinary types" and Marlow seems to be talking to them because he wants to share his experiences and what he has learned from his journey. The story is framed as Marlow telling his story to these men, and it is through his storytelling that the reader is able to understand and interpret the events of the novel.

Didn't he talk with those men long after he'd returned?

Yes, you are correct. Marlow tells his story to the group of men on the Thames River long after he has returned from his journey to the Congo. It is not specified in the book how much time has passed between Marlow's return from Congo and the moment he is telling the story, but it's implied that it's some time after. The story is presented as a recollection of Marlow's experiences and thoughts, rather than a contemporaneous account of them.

Did he ever talk with Marlow's fiance?

It is not specified in the story whether Marlow ever speaks with his fiancée about his journey to the Congo or his experiences there. The fiancée is not a character in the story, and there is no mention of her being informed or involved in any way with Marlow's journey. The focus of the story is on Marlow's inner journey and his reflections on the events he witnesses, rather than on his personal relationships.

Didn't Marlow talk with Kurtz's fiancée to tell her about Kurtz's death?

You are correct, towards the end of the novella, Marlow does meet Kurtz's Intended, his fiancée. Marlow feels a sense of obligation to inform her of the truth about Kurtz, who has been described as a god-like figure by the Company and the people in the colony. When he meets her, he does not give her a straight answer about Kurtz's condition when he died, instead he presents him as a man of great talents and virtues, he even lies to her saying that Kurtz's last words were her name. Marlow does this because he believes that the truth about Kurtz's descent into madness and tyranny would be too difficult for her to bear and it would be better for her to remember him as the man she loved.

ChatGPT botches an attempt to summarize Sir Gawain and the Green Knight

To the extent that this performance is characteristic of how ChatGPT summarizes long texts, those summaries cannot be trusted. Yes, this is only one case, and, yes, it’s a medieval romance, hardly typical of contemporary long texts. For all I know, it may do fine with, for example, trial transcripts, or long briefings, but...I’m going to need some pretty strong reassurances – evidence that it is well able to summarize this kind of text – before I trust it with anything important.

Note: Summarizing text is an important and difficult skill. I’ve seen many botched plot summaries in Wikipedia.

With that said, here’s my session with ChatGPT on Sir Gawain:

Are you familiar with Sir Gawain and the Green Knight?

Yes, I am familiar with Sir Gawain and the Green Knight. It is a 14th-century Middle English romance poem that tells the story of Sir Gawain, a knight of King Arthur's court, and his encounter with a mysterious green-skinned knight who challenges him to a game. The poem is written in a Middle English dialect known as West Midlands and is one of the best-known Arthurian stories. It is notable for its use of alliteration, its moral themes, and its depiction of the chivalrous code of conduct that was followed by knights in medieval times. Do you have any specific questions about the poem?

The poem is divided into four main sections. Would you summarize the third section.

Certainly! The third section of Sir Gawain and the Green Knight is called "The Return of the Green Knight." It begins with Sir Gawain setting out on horseback to keep his promise to the Green Knight and meet him at the Green Chapel one year later. Along the way, Gawain encounters a series of temptations and tests of his courage and honor. He is offered food and shelter by a lord and lady, but he refuses to accept any gifts from them, as he has vowed to accept nothing from anyone until he has fulfilled his promise to the Green Knight.

Eventually, Gawain reaches the Green Chapel and finds the Green Knight waiting for him. The Green Knight taunts Gawain, reminding him of his promise to receive a blow from the Green Knight's axe in return for the blow that Gawain had struck a year earlier. Gawain prepares himself for the blow, but at the last moment, the Green Knight reveals that he was actually testing Gawain's courage and honor. The Green Knight is actually Sir Bertilak, the lord who had offered Gawain food and shelter along the way, and the lady who had tried to seduce Gawain was his wife.

The Green Knight reveals that Gawain had passed all of the tests, and he reveals the gift that Gawain had received from the lady: a girdle that had the power to protect the wearer from injury. The Green Knight tells Gawain that he had been proud to host such a noble and chivalrous knight in his home and that he had learned much from Gawain's example. Gawain, in turn, is relieved and grateful to the Green Knight for revealing the truth and sparing his life. The third section of the poem ends with Gawain returning to King Arthur's court, where he is praised for his bravery and honor.

No, that sounds mostly like the fourth section. The third section takes place at the castle of that lord and lady you mention in your first paragraph. The Lord in names Sir Bercilak and Gawain stays with him for three days. What happens on those three days?

I apologize for the misunderstanding. You are correct that the third section of Sir Gawain and the Green Knight takes place at the castle of Lord Bertilak, also known as the Green Knight. On the first day, Gawain and Lord Bertilak go hunting, and Bertilak gives Gawain the game that they catch as a gift. Gawain reluctantly accepts the gift, but he makes a point of not accepting any more gifts from Bertilak until he has fulfilled his promise to the Green Knight.

On the second day, Bertilak's wife tries to seduce Gawain, offering him gifts and promising to be his lover. Gawain resists her advances and refuses to accept any of the gifts she offers him.

On the third day, Bertilak and Gawain go hunting again, and Bertilak gives Gawain the game that they catch as a gift. Gawain again reluctantly accepts the gift, but he makes a point of not accepting any more gifts from Bertilak until he has fulfilled his promise to the Green Knight.

That evening, Gawain sets out for the Green Chapel, where he will keep his promise to the Green Knight and receive a blow from his axe. The third section of the poem ends with Gawain leaving Bertilak's castle and setting out on the final leg of his journey to the Green Chapel.

This is confused. On the first day Bertilak goes deer hunting while Gawain stays at the castle. What happens there and what happens when Bertilak returns?

I apologize for the confusion. You are correct that on the first day of Gawain's visit to Lord Bertilak's castle, Bertilak goes hunting while Gawain stays at the castle. While Bertilak is out hunting, Gawain is visited by Bertilak's wife, who tries to seduce him and offers him gifts in an attempt to persuade him to be her lover. Gawain resists her advances and refuses to accept any of the gifts she offers him.

When Bertilak returns from hunting, he gives Gawain the game that he has caught as a gift. Gawain reluctantly accepts the gift, but he makes a point of not accepting any more gifts from Bertilak until he has fulfilled his promise to the Green Knight. The first day ends with Gawain and Bertilak eating a meal together and retiring for the night.

OK. But doesn’t the lady kiss Gawain and Gawain in turn kisses Bertilak in exchange for the deer Bertilak killed?

Yes, you are correct. In the story of Sir Gawain and the Green Knight, the lady of the castle does indeed try to seduce Gawain and offers him gifts in an attempt to persuade him to be her lover. When Gawain resists her advances, she kisses him and gives him a gift of a gold ring. Gawain, in turn, kisses Bertilak in exchange for the deer that Bertilak has given him as a gift.

However, it is important to note that Gawain does not succumb to the lady's advances and remains faithful to his vows of chivalry and to his host, Lord Bertilak. Gawain's acceptance of the gifts and kisses is a test of his honesty and honor, and he ultimately proves himself to be a noble and chivalrous knight.

Where did that gold ring come from?

System crashes while ChatGPT is defending its mention of a gold ring. It’s overloaded just now, Sunday morning, Jan. 8, 2023. There is no gold ring in the text. I logged out at this point.