Hung-yi Lee ML 2025 Lecture 8: When LLMs Think Too Much - Controlling Reasoning Length

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

The Problem: LLMs Think Too Much

Modern reasoning models like DeepSeek-R1 can produce extremely long outputs—often 5,000 to 10,000 tokens of reasoning for a single problem. But is longer reasoning always better?

Key observation: Research shows that longer reasoning doesn’t necessarily mean higher accuracy. In fact, there’s often a negative correlation—longer answers can indicate the model is struggling.

The Giraffe Analogy

Professor Lee uses the giraffe’s evolution as an analogy:

• Giraffes evolved long necks to reach high leaves
• But necks became so long they’re actually disadvantageous—giraffes must bend down to drink water and eat low vegetation
• During famines, short-necked giraffes actually survive better because they need less energy

The lesson: More isn’t always better. The optimal solution uses just enough resources to solve the problem.

Research Findings

Longer ≠ Better

Multiple papers show:

• Reasoning length and accuracy are not positively correlated
• Sometimes they’re negatively correlated
• The correlation might be spurious—harder problems naturally require longer reasoning AND have lower accuracy

The Real Relationship

When controlling for problem difficulty:

• Easy problems: Short reasoning is sufficient
• Hard problems: Longer reasoning is needed
• Forcing uniform length hurts performance on both

Methods to Control Reasoning Length

Method 1: Prompt Engineering (Chain of Draft)

Instead of “think step by step,” use prompts like:

• “Think step by step, but keep each step under 5 words”
• “Provide a draft outline, not detailed reasoning”

Results: Can reduce output length significantly while maintaining most accuracy.

Method 2: RL with Length Penalty

Add length constraints to the RL reward function:

Reward = Correctness - |Target_Length - Actual_Length|

Key insight: Don’t use absolute length limits. Use relative limits based on problem difficulty:

• If reasoning is shorter than average needed → OK
• If reasoning is longer than average needed → Penalty

Method 3: Knowledge Distillation with Short Answers

1. Have a teacher model solve problems multiple times
2. Collect the shortest correct reasoning paths
3. Train student model only on these short examples

Result: Student learns to reason efficiently without verbose explanations.

Method 4: Implicit Chain-of-Thought

Train models to reason internally without outputting the reasoning process:

• Model learns to “think” without writing it down
• Like learning mental math instead of showing work

Surprising finding: Models can maintain accuracy even without explicit reasoning output, suggesting they’ve internalized the reasoning process.

Practical Implications

For Users

• Longer model outputs don’t guarantee better answers
• Consider using models that can control output length based on your compute budget

For Developers

• Don’t just optimize for accuracy—consider efficiency
• The best AI isn’t one that always produces perfect answers, but one that produces good answers within resource constraints

The Best Engineer Analogy

“The best engineer isn’t one who does everything perfectly, but one who does the best possible work within limited resources.”

Experimental Results

Testing on math benchmarks (AIME, GSM8K):

• Models trained with length control can match or exceed verbose models
• 20-40% reduction in output length with minimal accuracy loss
• Works across different model sizes and architectures

Key Takeaways

1. Excess is as bad as deficiency (过犹不及): Too much reasoning can be wasteful
2. Adaptive length is key: Different problems need different amounts of reasoning
3. Efficiency matters: In real applications, compute resources are limited
4. Multiple approaches work: Prompt engineering, RL, and distillation can all help