Shared LoRA Subspaces for Almost Strict Continual Learning

Paper: 2602.06043 Categories: cs.LG, cs.CL

Abstract

Continual learning in large language models faces the fundamental challenge of catastrophic forgetting - when learning new tasks degrades performance on previously learned ones. This paper introduces Shared LoRA Subspaces, a method that leverages the low-rank structure of LoRA adapters to enable almost strict continual learning, where knowledge from previous tasks is preserved while efficiently learning new ones.

The Problem: Catastrophic Forgetting

When fine-tuning LLMs sequentially on multiple tasks:

• Learning task $T_2$ overwrites weights learned for task $T_1$
• Performance on earlier tasks degrades significantly
• Naive solutions require storing separate models per task

Traditional Approaches

Method	Storage	Forgetting	Efficiency
Full fine-tuning	High	Severe	Low
Separate LoRA per task	Linear growth	None	Moderate
Regularization methods	Low	Moderate	High

Key Contributions

• Shared Subspace Discovery: Identifies common low-rank subspaces across tasks that can be reused
• Almost Strict Continual Learning: Achieves near-zero forgetting while maintaining plasticity
• Parameter Efficiency: Sublinear growth in parameters as tasks increase
• Theoretical Foundation: Proves conditions under which shared subspaces preserve task performance

Methodology

LoRA Recap

Standard LoRA decomposes weight updates as:

$\Delta W = BA$

Where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ with rank $r \ll \min(d, k)$.

Shared Subspace Formulation

Instead of independent LoRA adapters per task, decompose as:

$\Delta W_t = B_{\text{shared}} A_t + B_t A_t^{\text{task}}$

Where:

• $B_{\text{shared}}$ captures common structure across tasks
• $A_t$ projects task-specific information into shared space
• $B_t A_t^{\text{task}}$ handles task-specific residuals

Subspace Identification

The shared subspace is identified by analyzing the principal components of LoRA updates across tasks:

$B_{\text{shared}} = \text{SVD}_r\left(\sum_{i=1}^{t-1} B_i A_i\right)$

This captures the dominant directions of adaptation that generalize across tasks.

Orthogonal Projection for Forgetting Prevention

To prevent forgetting, new task updates are projected orthogonally to the shared subspace:

$\Delta W_t^{\perp} = (I - B_{\text{shared}} B_{\text{shared}}^T) \Delta W_t$

This ensures new learning does not interfere with previously learned representations.

Almost Strict Continual Learning

The method achieves “almost strict” continual learning:

$\mathcal{L}_{T_i}(\theta + \Delta\theta_t) \leq \mathcal{L}_{T_i}(\theta + \Delta\theta_i) + \epsilon$

Where $\epsilon$ is bounded and decreases as the shared subspace better captures task structure.

Theoretical Guarantee

Under mild assumptions on task similarity:

$\|\Delta W_{\text{shared}}\|_F \leq \sqrt{\sum_{i=1}^t \|\Delta W_i\|_F^2 / t}$

The shared representation grows sublinearly with the number of tasks.

Results

Benchmark Performance

On continual learning benchmarks with LLaMA-based models:

Method	Avg. Accuracy	Forgetting	Parameters
Sequential LoRA	72.3%	15.2%	$O(t \cdot r)$
EWC + LoRA	76.8%	8.4%	$O(t \cdot r)$
Shared LoRA (ours)	81.2%	1.3%	$O(r + t \cdot r')$

Where $r' \ll r$ for task-specific residuals.

Scaling Behavior

• Forgetting remains below 2% even after 10+ sequential tasks
• Shared subspace stabilizes after 3-4 tasks
• Parameter efficiency improves with more tasks

Why This Matters

1. Practical Deployment: Single model can handle multiple tasks without degradation
2. Resource Efficiency: Avoids storing separate adapters per task
3. Knowledge Transfer: Shared subspace enables positive transfer between related tasks
4. Theoretical Grounding: Provides guarantees on forgetting bounds

Takeaways

1. LoRA subspaces can be shared: Common low-rank structure exists across diverse tasks
2. Orthogonal projection prevents forgetting: New learning in orthogonal directions preserves old knowledge
3. Sublinear parameter growth: Shared subspace amortizes storage across tasks
4. Almost strict is achievable: Near-zero forgetting with bounded theoretical guarantees
5. Practical for LLM deployment: Enables efficient multi-task models without separate fine-tuning