Hung-yi Lee ML 2025 Lecture 11: Model Merging - Equipping Your Foundation Model with Task Vectors

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

What is Model Merging?

Model Merging combines multiple fine-tuned models into one without any additional training. It’s like equipping your character in an RPG with different weapons and armor!

The Key Insight: Task Vectors

When you fine-tune a foundation model on a task:

• Original model parameters: θ
• Fine-tuned parameters: θ_A
• Task Vector: τ_A = θ_A - θ

This task vector represents the “skill” learned during fine-tuning.

The Magic of Task Vectors

Addition: Combining Skills

If you have:

• τ_A: Chinese language ability
• τ_B: Safety alignment

You can create a model with BOTH abilities:

θ_new = θ + τ_A + τ_B

No training required!

Subtraction: Removing Skills

Want to remove a capability? Subtract the task vector:

θ_new = θ - τ_B  // Remove skill B

Analogy: A = B implies C = D

If you know the relationship between tasks:

• Task A → Task B (e.g., English base → English chat)
• You can apply the same transformation to Task C:

τ_D = τ_C + (τ_B - τ_A)

Real-World Examples

Example 1: Chinese LLaMA with Alignment

• LLaMA-2-base: English foundation model
• LLaMA-2-Chat: English with safety alignment
• Chinese LLaMA: Chinese capability but no alignment

Solution:

τ_alignment = LLaMA-2-Chat - LLaMA-2-base
Chinese_aligned = Chinese_LLaMA + τ_alignment

Result: A Chinese model with safety alignment!

Example 2: Creating a “Saint” Model

1. Fine-tune a model on profanity data (learns to curse)
2. Calculate the “profanity vector”
3. Subtract it from a normal model
4. Result: A model that refuses to use profanity!

Example 3: Vision + Code Review

• Model A: Can see images (vision capability)
• Model B: Can review code (but text-only)

Merge them → A model that can review code from screenshots!

Why Does This Work?

The Hypothesis

Different tasks modify different parameters in the neural network. If tasks don’t overlap much in which parameters they change, merging works well.

When Merging Fails

• Tasks modify the same parameters → Interference
• Too many tasks merged → Degradation
• Incompatible foundation models → Failure

Advanced Techniques

DARE (Drop And REscale)

Problem: Different fine-tuned models change many overlapping parameters.

Solution:

1. Randomly drop some parameter changes
2. Rescale remaining changes
3. Merge the sparse task vectors

This reduces interference between tasks!

TIES (Trim, Elect Sign, Merge)

Another technique to reduce conflicts:

1. Trim small changes (likely noise)
2. Resolve sign conflicts
3. Merge remaining changes

The Task Vector Store Vision

Imagine a future where:

1. Foundation models are shared publicly
2. People create and share Task Vectors for specific skills
3. You can “shop” for capabilities and equip your model
4. No training needed - just add vectors!

Like an RPG equipment store for AI!

Applications Beyond Text

Speech Recognition

Problem: No real audio data for specialized domains (legal, medical meetings)

Solution:

1. Use TTS to synthesize domain-specific audio
2. Fine-tune on synthetic data
3. Calculate “Synthetic2Real” task vector
4. Apply to real domain → Works surprisingly well!

Key Factors for Success

1. Same Foundation Model: Models must start from the same base
2. Same Architecture: Network structure must match
3. Non-overlapping Changes: Tasks should modify different parameters
4. Larger Models Work Better: More parameters = less overlap

Limitations

• Not guaranteed to work
• Can find many failure cases
• Generalization is limited
• Still an active research area

Key Takeaways

1. Task Vectors are powerful: Represent learned skills as parameter differences

2. No training needed: Just arithmetic on parameters

3. Modular AI is possible: Mix and match capabilities

4. DARE/TIES help: Reduce interference between merged models

5. Future potential: Task Vector marketplaces could democratize AI customization