Hung-yi Lee ML 2025 Lecture 5: The Power and Limits of Pretrain-Alignment

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

The Three Stages of LLM Training

Modern LLM training follows three stages:

1. Pre-training: Learn basic text completion from massive internet data
2. Supervised Fine-Tuning (SFT): Learn to follow instructions with human-provided Q&A pairs
3. RLHF: Learn from human feedback on what responses are better

Stages 2 and 3 together are called alignment - aligning the model’s outputs with human values and needs.

The Illusion of Alignment’s Power

Compare LLaMA-2-7B-base (no alignment) vs LLaMA-2-7B-chat (with alignment):

• Base model: Asked “What is machine learning?”, it answers correctly… then repeats itself endlessly
• Chat model: Gives a well-structured, complete answer

On MT-Bench:

• LLaMA-3-70B (base): 2.7 points
• LLaMA-3-70B-instruct: 8.63 points

Looks like alignment is the MVP and pre-training is just along for the ride, right?

Wrong. Don’t let the scoring system fool you.

Alignment: Adding Eyes to the Dragon

The shocking truth: alignment uses remarkably little data.

• LLaMA-2: Only 27,540 examples for SFT
• LIMA paper: Only 1,000 carefully curated examples
• Homework 5: Only 100 examples (and it still works!)

Alignment is like “adding eyes to the dragon” (画龙点睛) - the dragon’s body is pre-training, but it only flies when you add the eyes. The eyes are small but crucial.

Quality over quantity: LLaMA-2’s team initially bought millions of examples from third parties. They were useless compared to their own 27,000 carefully crafted examples. Hence their section title: “Quality is All You Need.”

The Mystery of Good Alignment Data

What makes alignment data effective? Nobody really knows.

The Ruozhi Ba (弱智吧) Experiment: Researchers tested various data sources for fine-tuning. Zhihu (Chinese Q&A site) with 8,000+ examples improved accuracy from 53% to 63%. But “Ruozhi Ba” (a Chinese forum for absurd questions) with only 240 examples achieved 83%!

Sample Ruozhi Ba questions:

• “I boiled my bank card in a pressure cooker. Why is it still frozen?”
• “I’m 16 years old. Is it normal that I’m under 18?”
• “One jin of iron and one jin of cotton fall into water. Which do you save first?”

Why does this work? The paper’s authors speculate it’s the diversity. But there’s a hidden detail: the answers were generated by GPT-4. So it’s really knowledge distillation.

Knowledge Distillation: Learning from the Teacher

When people didn’t know how to do alignment well, they reverse-engineered ChatGPT:

1. Ask ChatGPT questions
2. Use its answers as training data
3. Fine-tune your model to match

Famous examples:

• Alpaca: LLaMA-7B + ChatGPT distillation, ~$100
• Vicuna: Similar approach
• SkyT1: Qwen-32B + QwQ distillation, $450
• S1: Qwen-2.5-32B + Gemini distillation, 1,000 examples

Note: These cost estimates don’t include data generation costs (querying the teacher model) or data cleaning.

Data Selection: Longer is Better?

AlpacaGas: Filter Alpaca’s 50K examples by having an LLM score quality. Keep only high-scoring examples. Better results with less data.

Long is More for Alignment: Just select the 1,000 longest answers. That’s it. This simple heuristic beat:

• Original Alpaca (50K examples)
• AlpacaGas (LLM-filtered)
• LIMA (1,000 human-curated examples)

Why? Still unclear. But it works.

The Questions Don’t Even Matter

Professor Lee’s lab (Chun-Cheng Hsieh’s work) discovered something wild:

1. Take random sentences from the internet
2. Cut them in half
3. Give the first half to ChatGPT, let it complete the second half
4. Use this as training data

The “questions” aren’t even questions. The “answers” are just text completions. Yet:

• Mistral-7B base: 3.73 → 7.3 (beats official Mistral-7B-instruct at 6.84!)
• LLaMA-3-8B base: 5.5 → 7.0
• LLaMA-3-70B base: 2.7 → 8.0

Why Does This Work? Alignment Changes Less Than You Think

Research shows that aligned vs unaligned models differ less than expected:

Token-level analysis: Compare what token the aligned model outputs vs what the base model would output:

• Unshift: Same top token (most common)
• Marginal: Correct token is 2nd or 3rd choice
• Shift: Correct token not in top 3 (rare)

Most tokens are unshift! The models behave almost identically. The shifted tokens are mostly:

• Greeting phrases (“Thank you for asking”)
• Connectors
• End-of-sequence tokens (crucial!)

The base model’s main problem: it doesn’t know when to stop talking.

Alignment Without Questions

Response Tuning: Train only on answers, no questions at all. Still works! Not quite as good as instruction tuning, but surprisingly effective.

Rule-Based “Alignment”: Without any fine-tuning, just add three rules:

1. Increase end-of-sequence probability (more as response gets longer)
2. Manually adjust some token probabilities
3. Penalize repeated tokens

Result: Base model goes from 2% win rate vs instruction model to 24% win rate. Just from rules!

Self-Alignment: The Model Teaches Itself

Since alignment is easy, models can align themselves:

1. Ask the model a question
2. Generate multiple answers
3. Have the model score its own answers
4. Use RL to increase probability of good answers

This is the Self-Rewarding Model approach (Meta, 2024). It works because alignment doesn’t require learning new knowledge - just adjusting behavior.

What Makes Pre-training Effective?

From “Physics of Language Models Part 3.1”:

The experiment: Create fake biographies. Pre-train a model. Then test if it can answer questions about people it saw during pre-training but not during alignment.

Result: 0% accuracy. The model can’t answer.

Why? If each person appears only once, the model learns wrong associations. It thinks “Takamatsu Tou + Yuushou Girls Academy + Astronomy Club” is ONE entity that is “MyGO’s vocalist.” It doesn’t understand that “Takamatsu Tou” alone is a person with properties.

Solution: Multiple descriptions of the same entity. If the model sees different phrasings of the same information, it learns that “Takamatsu Tou” is an entity with various properties.

Key finding: You don’t need ALL entities to have multiple descriptions. Just having SOME entities with multiple versions teaches the model how to understand entities in general.

The Scale of Pre-training Data

Current models use massive data:

• LLaMA-3: 15T tokens
• DeepSeek-V3: 14.8T tokens

The data wall: A 2022 paper estimated we might exhaust usable internet text by 2028-2030. LLaMA-2 used 1T tokens; LLaMA-3 used 15T - a 10x increase in one year. Internet content doesn’t grow that fast.

Good news: HuggingFace released FineWeb with 15T cleaned tokens. Data is available; compute is the bottleneck.

Data Quality Matters

Textbooks Are All You Need (Microsoft Phi): Training on textbook-style content (with exercises) beats raw code from the internet. But note: these “textbooks” were generated by GPT-3.5, so it might be knowledge distillation again.

Rephrase the Web (Apple): Use a model to rewrite messy web content into cleaner text. Training on rephrased content is 3x more data-efficient.

Deduplication is crucial: Models should see diverse data, not the same content repeatedly. After seeing the same data 4 times, performance starts degrading. After 40 times, learning stops entirely.

The Limits of Alignment

Alignment Can’t Teach New Knowledge

Categorize training data by what the base model knows:

• Highly known: Base model answers correctly
• Maybe known: Correct with right prompting
• Weakly known: Sometimes correct with sampling
• Unknown: Never correct

Intuition says unknown data should help most. Reality:

• Unknown data hurts the most
• Maybe known data helps the most

Training on unknown data doesn’t teach new knowledge - it damages existing capabilities.

Why RL Works

RL doesn’t force the model to learn impossible things. It only reinforces answers the model already generates. RL is about “activating potential,” not “implanting knowledge.”

The Ashes of Pre-training

ROT13 phenomenon: GPT-4 can decode ROT13 (shift letters by 13) but fails at ROT8. Why? ROT13 appears frequently on the internet as a meme. Pre-training data distribution affects capabilities in unexpected ways.

Alignment is a Mask

Using logit lens analysis on GPT-2 base vs DPO-aligned:

• Base model: Starts wanting to output profanity around layer 18
• Aligned model: Suppresses this tendency

But the profanity-related parameters still exist! Alignment just prevents them from being activated. The knowledge is still there, masked but not removed.

The famous meme: The model is a monster that learned everything from the internet. Alignment just puts a mask on it to look presentable.

Key Takeaways

1. Alignment is powerful but limited - it’s adding eyes to the dragon, not building the dragon
2. Quality over quantity - 1,000 good examples beat millions of bad ones
3. Knowledge distillation works - learning from stronger models is effective
4. Alignment changes behavior, not knowledge - it adjusts which tokens get output, not what the model knows
5. Pre-training determines capabilities - alignment can only surface what’s already there
6. The mask can slip - aligned models still contain everything from pre-training