Hung-yi Lee ML 2025 Lecture 9: Why Do You Care So Much About Benchmarks?

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

The Problem with Benchmarks

How do we know if a model has good reasoning ability? The common approach: test it on math problems like GSM8K. But this raises serious questions:

Can models just memorize answers?

• Models train on massive internet data
• GSM8K problems might be in training data
• A model might “reason” by recalling memorized answers

Goodhart’s Law

“When a measure becomes a target, it ceases to be a good measure.”

This applies directly to AI benchmarks:

• Once a benchmark becomes popular, models get optimized for it
• The benchmark stops measuring what it was designed to measure
• Most benchmarks get “solved” within 2-3 years of release

Testing Benchmark Robustness

Researchers tested whether models truly understand problems or just memorize:

Experiment: Modify GSM8K problems without changing difficulty

• Change names (Sophia → other names)
• Change numbers
• Reorder sentences
• Add irrelevant sentences

Results: Model accuracy dropped significantly on modified problems, suggesting some memorization is happening.

Chatbot Arena: Human Evaluation

An alternative approach: let humans judge which model is better.

How it works:

1. User asks a question
2. Two anonymous models respond
3. User picks the better response
4. Elo ratings calculated from win/loss records

Advantages:

• No fixed test set to memorize
• Real-world questions from real users
• Continuous evaluation

But Chatbot Arena Has Problems Too

Research found that style affects rankings more than substance:

Factors that influence human preference (but shouldn’t):

• Response length (longer often preferred)
• Emoji usage
• Bullet points and formatting
• Headline usage
• Positive/friendly tone

Example: Claude models are smart but “don’t talk pretty” - they rank lower than their actual capability because they’re less verbose and use fewer emojis.

When researchers controlled for style factors, rankings changed significantly!

ARC-AGI: A Different Approach

ARC-AGI (Abstraction and Reasoning Corpus) tries to test genuine reasoning:

Design principles:

• Visual pattern recognition tasks
• No language-based shortcuts
• Problems can’t be found on the internet
• Requires genuine abstraction ability

Example tasks:

• Given input/output pattern pairs, predict the output for a new input
• Patterns involve colors, shapes, transformations

Why ARC-AGI is Hard to Game

1. No fixed question bank: New problems can always be created
2. Visual format: Can’t be easily memorized as text
3. Hidden test set: Public examples differ from evaluation set
4. Requires true abstraction: Pattern recognition, not knowledge recall

O3’s Breakthrough

OpenAI’s O3 model achieved near-human performance on ARC-AGI:

• First model to significantly crack this benchmark
• But required ~$1000 of compute per problem
• Shows reasoning is possible but expensive

The Cobra Effect

Professor Lee shares the “Cobra Effect” story:

During British colonial India, there was a cobra problem. The government offered bounties for dead cobras. Result? People started breeding cobras for the bounty money, making the problem worse.

Lesson for AI: When you optimize for a metric, you might get unintended behaviors that game the metric rather than solve the underlying problem.

Key Takeaways

1. No perfect benchmark exists: Every evaluation method has flaws

2. Memorization vs. Reasoning: Hard to distinguish when models train on internet-scale data

3. Style vs. Substance: Human evaluations are biased by presentation

4. Goodhart’s Law is real: Popular benchmarks get gamed

5. Multiple evaluations needed: Don’t rely on any single metric

What Makes a Good Benchmark?

• Resistant to memorization
• Tests genuine capability, not surface patterns
• Hidden test sets
• Continuous renewal of questions
• Multiple evaluation dimensions