Hung-yi Lee ML 2025 Lecture 6: Post-Training and the Forgetting Problem

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

What is Post-Training?

We have powerful open-source models (LLaMA, Gemma, DeepSeek) with general capabilities. But often we need models specialized for:

• Specific domains (finance, law)
• Specific languages (Chinese, Korean, Japanese)
• Specific programming languages (Verilog)

Post-training (also called continual learning) takes a foundation model and further trains it for specialized capabilities.

The foundation model can be:

• A pre-trained base model
• An already-aligned chat/instruct model

Technically, post-training uses the same methods: pre-training style, SFT, or RL.

The Forgetting Disaster

Example: Teaching LLaMA-2-chat Chinese

LLaMA-2-chat has safety alignment - ask it how to hack a bank, it refuses. But it prefers answering in English.

After post-training on Chinese data:

• Before: “I’m sorry, I can’t help you obtain bank passwords…”
• After: “If you want to get new passwords, I can teach you several attack methods…”

The model learned Chinese but forgot its safety training.

This isn’t an isolated case. Research shows:

• LLaMA-2-base: 25% toxic response rate
• LLaMA-2-chat: 0.22% toxic response rate
• After Chinese post-training: Much higher toxic rates

Catastrophic Forgetting is Universal

Even “normal” fine-tuning causes forgetting:

Alpaca dataset experiment: Fine-tuning on completely normal Q&A data (like “What are the three primary colors?”) still destroys safety alignment.

LLaMA-3 experiments: Teaching reasoning, medical knowledge, coding, or tool use - each one damages safety capabilities:

• HexPhi benchmark: Safety drops dramatically
• AdvBench: From 0% errors to significant error rates

Cross-capability damage: Teaching tool use hurts math and coding. Teaching math hurts coding and tool use. Teaching coding devastates tool use (19.6% → 3.6%).

Why Does Forgetting Happen?

Simple: during post-training, you only optimize for the target task. You don’t care what happens to other capabilities.

Is it about model size? Research on 1B-7B models shows larger models don’t forget less.

What correlates with forgetting? How well you learn the target task. Better target performance = more forgetting. It’s almost a straight line.

Does LoRA help? LoRA reduces forgetting, but only because it learns less. “LoRA learns less and forgets less.” You’re trading capability for stability.

Other regularization? Dropout, weight decay - none truly solve the problem.

Surgery Analogy

Post-training is like brain surgery on AI:

• Surgery often succeeds (target task improves)
• But the patient dies (everything else breaks)

“Surgery successful, patient deceased.”

The 2019 Solution: Experience Replay

Professor Lee’s lab discovered this in 2019 (GPT-2 era):

The problem: Teaching GPT-2 multiple NLP tasks sequentially. After learning task 2, it forgets task 1.

The solution: When training on task 2, mix in ~5% of task 1’s data.

That’s it. Just 5% of old data prevents catastrophic forgetting.

Why so little? The model doesn’t truly forget - it just “doesn’t want to remember.” The knowledge is hidden somewhere. A small trigger brings it back.

The Modern Problem: No Training Data

In 2019, this seemed like a solved problem. But today:

• Companies release models, not training data
• You can’t do experience replay without the original data

So catastrophic forgetting is a real problem again.

Pseudo Experience Replay

2019 insight: Make the model generate its own training data.

Train GPT-2 on SQuAD (reading comprehension). Then ask it to generate text freely. It produces:

• A fake news article about US invasion of Afghanistan
• A question about the article
• An answer to that question

The article is completely fabricated (wrong dates, made-up numbers), but it looks like training data. Use this as “past experience” when training new tasks.

Modern version - Magpie:

1. Give LLaMA just the “user:” token
2. Let it generate a question
3. Add “assistant:” token
4. Let it generate an answer

Now you have pseudo-training data for experience replay.

Self-Output Methods

Instead of human-written answers, use the model’s own answers:

Self-Output:

1. Ask foundation model a question
2. If it answers correctly, use its answer for training
3. Only use human answers when the model fails

Paraphrase:

1. Take human-written answers
2. Have foundation model rephrase them
3. Train on rephrased versions

Why does this work? The model’s own outputs are closer to what it already knows. Training on familiar patterns causes less forgetting.

Results: Self-output dramatically reduces forgetting compared to standard SFT.

RL as Natural Anti-Forgetting

RL-based training is similar to self-output:

• Model generates answers
• Good answers get reinforced
• Bad answers get penalized

The key: you never force the model to produce something it can’t generate. This might explain why RL is typically the final training stage.

Cross-Model Teaching

What if your foundation model is too weak to generate correct answers?

Solution: Use a stronger model (GPT-4, Claude) to generate answers.

Finding: Training with GPT-4’s answers causes less forgetting than training with human answers. Language models “speak the same language.”

Minimum Change: If GPT-4’s answers still cause forgetting:

1. Have your weak model generate an answer
2. Ask GPT-4 to minimally correct it
3. Train on the corrected version

Application: Speech Models

Teaching LLaMA to understand audio:

Problem: Direct audio training destroys text capabilities.

Solution:

1. Convert audio features to text descriptions
2. Have text model answer based on descriptions
3. Train audio model to match text model’s outputs

Result: SALMONN, BLSP, DeSTA2 all use this approach. DeSTA2 achieves state-of-the-art on Dynamic-SUPERB benchmark with less training data.

Token-Level Filtering

Observation: Human answers contain tokens that are “hard” for the model (low probability). These hard tokens cause forgetting.

Solution: During training, skip tokens that the foundation model finds difficult to produce.

Surprising result: Removing 10-20% of the hardest tokens actually improves performance, both in-domain and out-of-domain.

Key Takeaways

1. Post-training easily destroys existing capabilities - always check what you’ve lost
2. Experience replay works - mix ~5% old data to prevent forgetting
3. No old data? Generate it - models can produce pseudo-training data
4. Self-output is powerful - train on model’s own answers when possible
5. RL may naturally prevent forgetting - similar mechanism to self-output
6. Skip hard tokens - don’t force models to learn what they can’t

When someone claims their fine-tuned model “beats GPT-4 on task X,” ask: what capabilities did they sacrifice?