Spectral and Subspace Dynamics of LoRA: Why Reinforcement Learning Demands Different Optimization Than SFT

Paper: 2611.08414
Authors: Chenruo Liu, Zihan Wang, Yijun Dong, Qi Lei (New York University, Institute of Science and Technology Austria)
Categories: cs.LG, stat.ML, cs.AI

The Gap

In post-training foundation models, practitioners commonly chain two distinct stages: Supervised Fine-Tuning (SFT) on curated demonstrations, followed by Reinforcement Learning with Verifiable Rewards (RLVR / GRPO / PPO) on self-generated reasoning rollouts.

To conserve GPU VRAM, Low-Rank Adaptation (LoRA) is standard practice across both stages, adapting frozen weights $W_0$ via low-rank factor matrices: $\Delta W_t = \frac{\alpha}{r} B_t A_t, \quad B_t \in \mathbb{R}^{d_o \times r}, \, A_t \in \mathbb{R}^{r \times d_i}, \quad r \ll \min(d_o, d_i)$

However, machine learning practitioners routinely use identical optimizer defaults (standard AdamW, fixed learning rates, and uniform scaling) across both SFT and RL.

This one-size-fits-all approach ignores a fundamental mystery: Why does LoRA behave so erratically between SFT and RL?

1. The Spectral Mystery: In SFT, models benefit from higher rank utilization and moderate learning rates. In RL, models often suffer from training instability unless trained with tiny learning rates, yet the geometric reason has remained unknown.
2. The Missing Geometric Diagnostic: While prior work studied differences in output logits or full-weight updates, no one had tracked the low-rank increment $\Delta W_t$ itself—specifically its effective rank (singular value spread) and singular subspace rotation (which input/output directions are selected).
3. Conflating Sampling with Optimization: Is RL’s distinct behavior caused by on-policy exploration, reward hacking, or the mathematical formulation of relative policy gradients?

                     THE UNNOTICED DIVERGENCE IN LORA POST-TRAINING

  Supervised Fine-Tuning (SFT):
  Gold completions ──> Coherent gradients ──> Expands effective rank (SVD spread)
                                              Stable subspace orientation

  Reinforcement Learning (GRPO / RLVR):
  Sampled rollouts ──> Group-mean baseline ──> Collapses effective rank!
                       subtraction (r_i - r̄)  Rotates singular subspaces aggressively!

  Result: Applying identical AdamW configurations causes spectral collapse in RL
          and under-utilizes capacity in SFT.

---

The Increment

One sentence: NYU researchers track the singular spectrum and subspaces of LoRA updates, proving theoretically and empirically that group-mean baseline subtraction ($r_i - \bar{r}$) inherently tilts RL dynamics toward spectral concentration and rapid subspace rotation, and introduce stage-tailored optimizers: full spectral flattening for SFT, and orthogonal subspace projection flattening for RL.

                    SPECTRAL FLATTENING IN SFT VS RL

  SFT-Tailored LoRA Update:
  Flatten the full update to distribute capacity across all rank directions:
  B_{t+1} = B_t + η M( D_t^B )              [M(·): Norm-preserving Muon transform]
  A_{t+1} = A_t + η D_t^A                   [Adam direction for factor A]

  RL-Tailored LoRA Update (GRPO):
  Project onto current adapter span P_t = B_t(B_t^T B_t)^† B_t^T;
  Flatten ONLY the orthogonal component to guide subspace rotation:
  B_{t+1} = B_t + η [ P_t D_t^B + M((I - P_t) D_t^B) ]

---

Core Framework & Mechanisms

The authors construct an experimental and theoretical bridge dissecting the four-stage continuum between SFT and RL: $\text{SFT} \longrightarrow \text{Rejection Sampling (RFT)} \longrightarrow \text{RFT with Group Baseline (RFT-B)} \longrightarrow \text{GRPO}$

1. Two Key Diagnostic Lenses

For each LoRA adapter $\Delta W_t = \frac{\alpha}{r} B_t A_t$, the authors compute its singular value decomposition $\Delta W_t = U_t \Sigma_t V_t^\top$ and track:

1. Effective Rank (Roy & Vetterli entropy of normalized squared singular values): $\text{erank}(t) = \exp\left( -\sum_{i=1}^r \frac{\sigma_{i, t}^2}{\sum_j \sigma_{j, t}^2} \log \frac{\sigma_{i, t}^2}{\sum_j \sigma_{j, t}^2} \right) \in [1, r]$ Low $\text{erank}$ indicates that singular values are concentrated onto a tiny handful of dominant directions (rank collapse).
2. Subspace Overlap (measuring how fast input/output subspaces rotate): $\text{ovlp}_L(t_1, t_2) = \frac{1}{r} \| U_{t_1}^\top U_{t_2} \|_F^2 \in [0, 1], \quad \text{ovlp}_R(t_1, t_2) = \frac{1}{r} \| V_{t_1}^\top V_{t_2} \|_F^2 \in [0, 1]$ A rapid decay in overlap signifies fast geometric rotation of the adapted subspace.

2. The Smoking Gun: Group-Mean Baseline Subtraction

By comparing RFT (rejection sampling without baseline) against RFT-B (rejection sampling with group-mean reward subtraction $r_i - \bar{r}$):

• SFT and RFT behave identically: at intermediate learning rates ($5 \times 10^{-5}$ to $5 \times 10^{-4}$), effective rank expands steadily, and subspace rotation is slow.
• RFT-B and GRPO behave identically: effective rank relentlessly collapses at every single learning rate, and singular subspaces rotate significantly faster.
• The Mathematical Cause: In the paper’s quadratic multi-index model, baseline subtraction suppresses the shared task-relevant components ($\vartheta_a < 1/2$) while amplifying the contrastive noise components ($\vartheta_o > 1/2$). This tilts the balance from coherent fitting toward gradient noise, concentrating energy onto the single strongest eigenvalue and rotating the remaining coordinates.

3. Dynamics-Informed Optimizer Design

Armed with this geometric understanding, the authors design stage-specific optimizer updates using norm-preserving spectral flattening $M(D) := \frac{\|D\|_F}{\sqrt{\text{rank}(D)}} D (D^\top D)^{-1/2}$:

• For SFT: Apply $M(\cdot)$ to the full SGD update of factor $B$. This prevents premature spectral collapse, forcing all $r$ singular directions to absorb demonstration knowledge evenly.
• For RL: In GRPO, the model needs to rotate its subspace to discover novel reasoning paths, but must not destabilize existing within-span progress. The algorithm computes the orthogonal projection $P_t = B_t (B_t^\top B_t)^\dagger B_t^\top$ and applies $M(\cdot)$ strictly to the orthogonal component $(I - P_t) D_t^B$: $B_{t+1} = B_t + \eta \left[ P_t D_t^B + M\left( (I - P_t) D_t^B \right) \right]$

---

Key Quantitative Results

The methods were evaluated across 8 foundation models (including Llama-3.2-1B/3B, Qwen2.5-3B, and Qwen2.5-Math-7B) on challenging mathematical reasoning benchmarks (GSM8K and MATH).

1. Dynamics Comparison on Mathematical Reasoning (GSM8K)

Training Stage & Setup	Optimizer Configuration	Effective Rank Trend ($\text{erank}$)	Subspace Rotation Speed	Preferred Learning Rate
Supervised Fine-Tuning (SFT)	Standard AdamW	Increases ($\text{erank} \uparrow$ after step 25)	Slow ($\text{ovlp} \approx 0.85$)	Moderate ($5 \times 10^{-4}$)
Rejection Sampling (RFT)	Standard AdamW	Increases ($\text{erank} \uparrow$)	Slow ($\text{ovlp} \approx 0.82$)	Moderate ($5 \times 10^{-4}$)
RFT + Group Baseline (RFT-B)	Standard AdamW	Collapses ($\text{erank} \downarrow$ across all LRs)	Fast ($\text{ovlp} \approx 0.35$)	Low ($5 \times 10^{-5}$)
GRPO (Online RL)	Standard AdamW	Collapses ($\text{erank} \downarrow$ across all LRs)	Very Fast ($\text{ovlp} \approx 0.28$)	Low ($5 \times 10^{-5}$)
SFT (Ours)	Full Spectral Flattening	Maximally Expanded ($\text{erank} \to r$)	Controlled	Moderate ($5 \times 10^{-4}$)
GRPO (Ours)	Orthogonal Subspace Flattening	Preserved Active Subspace	Optimally Guided Rotation	Low ($5 \times 10^{-5}$)

2. Downstream Performance Gains

• SFT Mathematical Accuracy: Applying full spectral flattening to factor $B$ during SFT yielded the highest pass@1 accuracy across tested Llama and Qwen checkpoints, beating vanilla LoRA and standard Muon by up to 2.4 percentage points.
• RL Exploration & Stability: Under GRPO, orthogonal subspace flattening consistently prevented policy collapse, maintaining steady reward growth and outperforming vanilla GRPO-LoRA across both base and instruction-tuned models.
• Validation of the Root Cause: The experiments prove that RFT-B perfectly mirrors GRPO’s geometric trajectory, conclusively isolating baseline subtraction—not online data sampling—as the mathematical driver of spectral concentration.

---

Practical Implications

1. Never Copy SFT Hyperparameters to RL: The empirical habit of using identical LoRA ranks, learning rates, and optimizers across SFT and RL is fundamentally flawed. RL with baseline subtraction operates in a high-rotation, rank-collapsing regime that requires lower learning rates and targeted orthogonal exploration.
2. Spectral Surgery for Reinforcement Learning: Rather than blindly increasing nominal LoRA rank $r$, practitioners should monitor effective rank $\text{erank}(t)$. If effective rank collapses to 1 or 2 during GRPO, nominal rank capacity is being wasted.
3. Targeted Subspace Exploration: In reasoning agents, applying spectral conditioning specifically to the orthogonal subspace $(I - P_t) D_t^B$ offers a principled way to encourage policy exploration without overwriting previously mastered reasoning patterns.

                   LORA 后训练阶段未被察觉的几何动力学分裂

  监督微调阶段 (SFT):
  黄金专家数据 ──> 梯度连贯稳定 ──> 有效秩持续扩展 (奇异值能量均匀分布)
                                  奇异子空间旋转极其缓慢平稳

  强化学习阶段 (GRPO / RLVR):
  自采样探索轨迹 ──> 组均值基线扣除 (r_i - r̄) ──> 奇异谱断崖式聚拢 (有效秩严重塌缩!)
                                                 奇异子空间剧烈超速旋转!

  核心痛点：使用同一套通用 AdamW 优化器，会导致 RL 阶段过早发生谱聚拢塌缩，
           而在 SFT 阶段又无法充分激活 LoRA 的全部秩容量。

                    SFT 与 RL 分立的动力学定制优化方案

  SFT 定制 LoRA 更新策略：
  对全量梯度做谱平坦化，促使奇异值能量均匀铺展到所有低秩方向：
  B_{t+1} = B_t + η M( D_t^B )              [M(·): 保范数 Muon 正交奇异谱平坦化算子]
  A_{t+1} = A_t + η D_t^A                   [A 矩阵保留 Adam 自适应动量更新]

  RL 定制 LoRA 更新策略 (针对 GRPO):
  构建当前适配器列空间的正交投影算子 P_t = B_t(B_t^T B_t)^† B_t^T;
  严格仅对“子空间之外的正交增量”实施谱平坦化，定向引导子空间探索而不破坏已有记忆：
  B_{t+1} = B_t + η [ P_t D_t^B + M((I - P_t) D_t^B) ]