Fine-Tuning Regimes: Different Fine-Tuning Methods Define Distinct CL Problems

Paper: 2604.21927 Authors: Lucas Page, Luca Zancato, et al. Categories: cs.AI, cs.LG

Problem

Current CL benchmarks evaluate algorithms without considering how they’re fine-tuned. This leads to misleading conclusions:

• Different fine-tuning methods (full fine-tuning, LoRA, adapters) fundamentally change what the CL problem is
• Same benchmark + same algorithm = different conclusions depending on fine-tuning method
• Current research conflates “algorithm design” with “fine-tuning protocol”

Key Insight

The paper proposes that fine-tuning method choice IS part of the CL problem definition. Three regimes:

1. Full Fine-tuning: All parameters updated → hardest CL problem
2. LoRA/Adapters: Only low-rank matrices → easier but still non-trivial
3. Prompt Tuning: Only soft prompts → nearly trivial for many tasks

Implications

Fine-tuning Method	CL Problem	Research Focus
Full	Catastrophic forgetting	Algorithm design
LoRA	Task interference	Rank selection
Prompt	Template learning	Prompt engineering

Same data, different conclusions! Benchmark conclusions depend on taskification.

Takeaways

• Fine-tuning method choice fundamentally changes the CL problem
• Researchers must specify fine-tuning protocols in benchmark evaluations
• “Best” algorithm may depend entirely on which fine-tuning regime you’re in