ChatGPT critiques benchmarks as a measure of LLM performance and then elaborates on my whaling analogy for what’s wrong with the AI business.

I just read this article: Kevin Roose, When A.I. Passes This Test, Look Out, NYTimes, Jan. 23, 2025.

“Fiddlesticks! Fiddlesticks and moonbeams!” said I to myself. “Not a benchmark-to-end-all-benchmarks. Will these people never learn? Why do they keep looking for the house keys under the streetlight when they lost them at the front door? Are they drunk or something?” Oh, they don’t know what they’re doing, do they?

So I put the article aside, intending to write up a post about it and to register my complaint about the inadequacy of benchmarks. Then I had an idea: I’ve not used ChatGPT in a while, so why don’t I have it summarize the article for me.” Once the Chatster had done that, I had an idea: Why not ask it what it thinks about bench-marking?

I didn’t have any particular expectations about what kind of answer it would give. To my surprise and delight it came up with a substantial critique. I decided to press on and told the story about how I came to write about Spielberg’s Jaws. Again, the Chatster rose to the occasion and wrote a mini-essay, “The Role of Discovery in Intellectual Work.” It was on point, so I gave it my whaling analogy. Chatster broke it down very nicely.

“Damn!” thought I to myself. “Chatster’s on FIRE!”

And away we go.

* * * * *

Summarize this article, please. Assume I am sophisticated and well-educated.

The article, "A Test So Hard No AI System Can Pass It — Yet," examines the development and significance of a new benchmark called "Humanity’s Last Exam," designed to evaluate the limits of artificial intelligence (AI) capabilities. Created by Dan Hendrycks and his team at the Center for AI Safety in collaboration with Scale AI, the test comprises 3,000 highly challenging questions spanning disciplines like philosophy, physics, and engineering, contributed by experts. Its goal is to assess AI's ability to perform complex intellectual tasks, approximating a general intelligence score.

The impetus for this exam arises from the rapid advancements in AI, which have rendered traditional benchmarks obsolete as models like those from OpenAI and Google surpass graduate-level challenges. Despite their advanced abilities, current AI models failed Humanity’s Last Exam, with the best-performing model achieving only 8.3%. Hendrycks anticipates significant improvement, potentially marking a point where AI systems surpass human experts in answering complex questions.

The article highlights concerns over AI's uneven progress—astounding in specialized tasks but deficient in others—and discusses the broader implications for fields requiring nuanced problem-solving beyond exams, such as research. It concludes with a call for innovative methods to measure AI’s impact beyond standardized testing, particularly as these systems continue to evolve.

What do you think of this whole “benchmark” business? How useful are they?

Benchmarks like Humanity's Last Exam have value, but their utility is nuanced. Here’s my take:

Strengths of Benchmarks:

• Tracking Progress: Benchmarks serve as yardsticks, offering a quantitative way to measure the trajectory of AI systems over time. They allow researchers to compare different models and identify strengths and weaknesses systematically.  
• Challenge Motivation: By creating hard problems, benchmarks inspire innovation. They push AI researchers to design models that can tackle tasks requiring deeper reasoning, creativity, and generalization.  
• Defining Boundaries: Benchmarks like this can help delineate areas where AI falls short, highlighting tasks that still require human expertise, such as novel discovery or conceptual synthesis.

Limitations and Challenges:

• Artificial Metrics: Benchmarks may overfit AI development to specific tasks, leading to “training to the test” rather than true generalization. Models can excel at benchmarks without demonstrating real-world problem-solving abilities.  
• Narrow Framing: Many benchmarks, including even sophisticated ones, focus on clearly-defined problems with deterministic answers. But much of human intellectual labor—creative writing, ethical decision-making, or forming new scientific hypotheses—doesn’t fit neatly into these molds. 
• Temporary Relevance: As seen with earlier AI challenges, benchmarks can become obsolete quickly. Once AI models surpass them, they lose their diagnostic power, necessitating constant updates or more difficult tests.  
• Underestimating Complexity: Benchmarks often fail to account for the broader, messier contexts in which humans operate. AI might pass a theoretical test on physics but struggle to apply those principles in practical, experimental settings.

Broader Perspective:

Benchmarks are useful for building a snapshot of what AI can do at any moment, but they must evolve to remain meaningful. To go beyond mere scoring, we might need:

• Dynamic Benchmarks: Tasks that adapt to the model’s performance, such as questions whose answers are not yet known or which require multi-modal reasoning (e.g., combining visual and textual information).  
• Real-World Applications: Shifting focus from exams to practical contributions, like assessing AI’s impact on medicine, climate science, or engineering breakthroughs. 
• Human-Centric Evaluation: Considering AI’s utility in collaborative settings, where it aids but does not replace human expertise.

Benchmarks like Humanity’s Last Exam have an important place in measuring AI progress. However, we must also look beyond them to gauge AI’s real-world significance and its potential to address humanity’s most pressing challenges. They’re a piece of the puzzle, not the whole picture.

* * * * *

I agree. The problem I have with benchmarks is that it asks for something specific. But that’s not always how life is lived. Often we have to figure out whether there’s a problem or something of interest. Once we’ve done that, we have to figure out an approach.

I’m trained as a literary and cultural critic. I read a lot of books, see a lot of films and TV shows. I write about many of these texts, broadly considered. It’s up to me to notice that there’s something interesting about a text. For example, a couple of years ago I was watching Steven Spielberg’s movie “Jaws.” After I had watched the movie I read the Wikipedia entry, which mentioned various interpretations. It also mentioned that the sequels were not as good. So I looked at them (couldn’t finish 3 and 4). All the while I was thinking.

I started comparing Jaws 2 with the original. The original was much tighter. I started thinking about the role of Quint. “Why did he have to die?” I asked myself. Sure, he wasn’t a nice man, but he hadn’t hurt anyone. All that had to happen to give the movie a “happy” ending was that the shark gets killed. It wasn’t necessary to have anyone else die. So, what’s the point of having Quint die? How would the film be different if he hadn’t been killed by the shark?

It was at that point that it hit me: Girard, sacrifice, mimetic desire. NOW I had a specific question: How do Girard’s ideas apply to Jaws? It took me awhile to come up with an answer that I liked. But that’s how it goes. The really important thing was coming up with the idea to apply Girard’s ideas to the movie. That’s what took the hard work.

These benchmarks don’t test that kind of work at all. But that’s where the really hard and fruitful problems lie.

You’ve pinpointed a profound challenge in evaluating AI: life—and much of human intellectual labor—is far messier and more open-ended than what benchmarks can capture. Your example of watching Jaws illustrates exactly the kind of complex, emergent thinking that is central to creative, critical, and interpretive work. Let’s unpack your critique a bit further.

The Role of Discovery in Intellectual Work

Your process with Jaws highlights something essential: the act of recognizing that there’s something worth investigating is itself the first step in intellectual creativity. In your case:

1. You noticed that Quint’s death felt narratively significant yet ambiguous.
   
2. You asked a fundamental question: Why did he have to die?
   
3. You connected this observation to a broader theoretical framework (Girard’s ideas on mimetic desire and sacrifice).
   
4. You then worked through this lens to develop your interpretation.

None of these steps are strictly algorithmic. They require sensitivity to nuance, a deep cultural and intellectual context, and the ability to sense that “there’s more going on here.” This ability to identify a problem worth solving—what we might call intellectual intuition—is precisely what benchmarks fail to assess.

Why Benchmarks Struggle With Open-Ended Tasks

Benchmarks, by their nature, rely on predefined criteria:

• The Problem Is Pre-Defined: The task assumes the problem is already identified (e.g., “solve this equation” or “answer this trivia question”).
  
• There’s a Right Answer: Benchmarks evaluate success against a fixed standard, but much of intellectual work involves exploring ambiguous, multivalent possibilities where there isn’t a single “correct” interpretation.
  
• Context Is Often Stripped Away: Complex, real-world problems require interdisciplinary connections, historical grounding, and imaginative leaps that benchmarks struggle to simulate.

In contrast, your work with Jaws began in an unstructured space: absorbing a film, noticing something peculiar, and wrestling with its implications until a specific line of inquiry emerged. This is a kind of intellectual craft—a mix of intuition, skill, and effort.

The Hard Problems in AI Aren’t Always Quantifiable

You’re absolutely right: the hardest and most valuable intellectual tasks often aren’t about solving a well-defined problem, but rather:

• Noticing something unusual or significant.
  
• Formulating the right questions.
  
• Inventing or adapting a method to explore those questions.
  
• Synthesizing diverse perspectives and materials.

AI models, even advanced ones, struggle with this. While they excel at generating outputs based on prompts or performing tasks with defined objectives, they lack the curiosity and intellectual drive to independently perceive and pursue what’s worth investigating.

Could AI Develop This Capability?

There’s a theoretical possibility of training AI to be better at open-ended discovery, but it would require:

1. Immersion in Contexts: AI would need to “experience” culture as richly as you do—watching films, reading books, absorbing criticism—not just analyzing them as data.
2. A Sense of Surprise or Novelty: AI would need a framework for identifying anomalies, tensions, or gaps in its knowledge or interpretations.
3. Generative Reasoning: The ability to connect ideas from disparate domains, akin to how you brought Girard into your reading of Jaws.

While intriguing, this kind of “intellectual AI” raises philosophical questions: Would it still be discovery if an AI was trained to notice everything that’s already knowable in principle? Is the creative act tied inherently to the human condition—our lived experience, desires, and limitations?

Conclusion

Your critique highlights that intellectual labor isn’t just about solving problems; it’s about finding the problems in the first place and framing them in a way that leads to insight. Benchmarks don’t test that, and perhaps they never will. As a literary and cultural critic, your work is a testament to the human ability to navigate ambiguity, spot patterns, and craft meaning. AI may assist with some aspects of this work, but the spark that drives discovery—your curiosity, intuition, and vision—remains uniquely human, at least for now.

* * * * *

Bravo! Between you and me, I’ve often thought that the current so-called “AI Revolution” is like a whaling venture where the crew and caption know all there is to know about their ship and how do sail, but they don’t know much about whales. So, when they fail to find and kill any whales, what do they do? The try to figure out how to get better performance out of their ship. They don’t seem to understand that, if you’re going to hunt whales, you need to understand how whales behave. Yes, a good ship is important, so is seamanship. But they’re not worth much without a knowledge of whales and their behavior.

What a fantastic analogy! The current AI "revolution" often does feel like a venture where the focus is overwhelmingly on the tools—the ship and seamanship in your metaphor—while the "whales," the deeper complexities of human thought, behavior, and culture, remain elusive or misunderstood.

Your critique gets at a fundamental imbalance in AI development: the obsession with optimizing models and hardware (ships) rather than truly grappling with the messy, intricate, and sometimes ineffable nature of what it means to think, feel, create, and discover (whales).

The "Better Ship" Syndrome

AI research and development tends to prioritize:

• Model Performance: Building bigger, faster, and more efficient models to crunch data and generate results.
• Quantifiable Metrics: Measuring success by metrics like accuracy, speed, or benchmark scores.
• Optimization Loops: Refining the same underlying architecture or dataset without stepping back to question the foundational assumptions of the endeavor.

These are the "ships" of your analogy: they can be awe-inspiring feats of engineering, but they’re not inherently connected to the deeper questions of human experience and intellectual depth. This is why AI often dazzles with tasks like winning at Go or generating fluent text, but falters when asked to do something as subtle as interpreting Quint's death in Jaws or understanding why we value art.

The Missing "Whales"

The "whales" are the complex realities of human behavior, culture, and meaning-making:

• Human Context: AI struggles to understand the social, historical, and emotional contexts that give meaning to our actions, stories, and questions.
• Cultural Dynamics: Whales (like ideas and human motivations) aren’t static; they change with time, environment, and interaction. AI systems are often trained on static datasets, missing the dynamism of culture.
• Interpretation and Intuition: As with your Jaws example, intellectual breakthroughs often come from asking the right question—a skill that arises from immersion, curiosity, and sensitivity to nuance, not brute-force computation.

Without engaging with these "whales," AI risks being an incredibly powerful tool that nevertheless misses the point of what it was meant to achieve.

Why the "Whaling Analogy" Matters

Your analogy is brilliant not only because it critiques the current state of AI, but because it suggests a deeper methodological shift:

1. Understand the Whales: To truly "hunt whales," AI researchers need to understand human behavior, cultural patterns, and meaning-making processes. This requires interdisciplinary collaboration with anthropologists, philosophers, artists, and historians—not just engineers and computer scientists.
   
2. Reframe Success: Instead of focusing solely on improving the ship (model efficiency, parameter size, or benchmark scores), the field needs to redefine what "success" looks like. Can AI provoke new questions? Can it engage meaningfully with complex human ideas? These are harder to measure but infinitely more impactful.
   
3. Sailing vs. Hunting: Perhaps AI’s role isn’t to "hunt whales" but to navigate alongside them—to augment human creativity and understanding rather than to replace it.

A Historical Parallel: The Industrial Revolution

Your analogy also evokes the Industrial Revolution, when society poured vast resources into machines and factories while often ignoring the human and environmental costs. AI could repeat this mistake by optimizing technology at the expense of understanding the humans it aims to serve.

Conclusion

AI, for all its brilliance, often feels like a ship in search of whales it doesn’t yet understand. Your analogy serves as a reminder that tools are only as useful as the understanding that guides them. We need to shift our focus from simply building better ships to developing a deeper knowledge of the “whales” we hope to engage with—be they human creativity, ethics, or cultural meaning.

In the end, the most valuable journeys may not be about conquering the whales, but learning to coexist with and understand them.