Hung-yi Lee ML 2025 Lecture 7: How DeepSeek-R1 and Reasoning Models Think Deeply

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

What is Deep Thinking in LLMs?

Recent models like ChatGPT’s O-series, DeepSeek-R1, Claude 3.7 Sonnet’s Extended Thinking, and Gemini Flash Thinking all share a common trait: they produce lengthy reasoning processes before giving final answers. This “deep thinking” behavior involves:

• Breaking down problems into steps
• Self-verification and checking work
• Exploring multiple solution approaches
• Recognizing and correcting errors mid-reasoning

Test-Time Compute: The Key Concept

Test-Time Compute (or Test-Time Scaling) means investing more computational resources during inference rather than just during training. The core insight: by letting models “think longer” at test time, they can solve harder problems.

Key findings:

• More attempts = higher chance of correct answer
• Even weak models can match stronger ones given enough tries
• The “infinite monkey theorem” applies: with enough attempts, models will eventually produce correct answers

Four Methods for Building Reasoning Models

Method 1: Supervised Chain-of-Thought (COT)

The simplest approach: show the model examples of step-by-step reasoning.

Few-shot COT: Give examples in the prompt showing how to break down problems Zero-shot COT: Simply add “Let’s think step by step” to the prompt

Limitation: Requires human-written reasoning examples, which is expensive and time-consuming.

Method 2: Reinforcement Learning (RL)

Train models using only final answer correctness as reward, without requiring reasoning process labels.

How it works:

1. Model generates answer
2. Check if answer is correct
3. Correct → positive reward, Wrong → negative reward
4. Model learns to increase probability of paths leading to correct answers

Key insight from DeepSeek-R1: RL doesn’t teach new abilities—it strengthens behaviors the model already has. The “aha moment” (self-correction, verification) emerges naturally through RL training.

Important finding: RL works better on stronger foundation models. Weak models may not have the latent reasoning ability to strengthen.

Method 3: Process Verification

Instead of only checking final answers, verify each reasoning step.

Process Verifier: A model that scores individual steps, not just final answers

• Can catch errors early before they compound
• Enables tree search through solution space
• More expensive but more reliable

Approaches:

• Best-of-N: Generate N answers, pick the one verifier scores highest
• Majority Vote: Pick the most common answer among multiple attempts
• Tree Search: Explore solution paths, prune bad branches early

Method 4: Imitation Learning (Knowledge Distillation)

Use a strong reasoning model (teacher) to generate training data for weaker models (students).

DeepSeek’s approach:

1. Train R1-Zero using pure RL (produces hard-to-read reasoning)
2. Use R1-Zero to generate 600K+ reasoning examples
3. Human annotators clean up and improve readability
4. Train final R1 model on this curated data

Key finding: A 1B parameter model trained via distillation can outperform an 8B model without reasoning training.

The DeepSeek-R1 Training Pipeline

1. Foundation: Start with DeepSeek-V3 Base
2. Cold Start: Initial supervised fine-tuning with COT data
3. RL Training: Train R1-Zero with correctness reward + format reward
4. Data Generation: R1-Zero generates reasoning examples
5. Human Refinement: Clean up reasoning for readability
6. Final Training: Combine all data, train final R1 model
7. Additional RL: Final RL pass for safety and helpfulness

Interesting Observations

The “Aha Moment”

Models trained with RL spontaneously develop self-correction behavior—they’ll say things like “Wait, let me check that” or “Hmm, that doesn’t seem right” without being explicitly taught.

Reasoning Process Readability

R1-Zero’s raw reasoning is nearly unreadable—jumping between languages, making logical leaps, and containing many errors. Human intervention is needed to make it comprehensible.

Format Rewards Matter

DeepSeek uses special tokens (<think> and </think>) to separate reasoning from final answers. This helps with interface design and allows users to optionally hide the thinking process.

Practical Implications

1. Majority Vote is a strong baseline: Before trying complex methods, try having the model answer multiple times and pick the most common answer

2. Foundation model quality matters: RL can only strengthen existing capabilities, not create new ones

3. Distillation is powerful: Small models can achieve strong reasoning through learning from larger models

4. Test-time compute trades off with training compute: You can invest resources either in training or inference