PEFT-Arena: Rethinking Parameter-Efficient Finetuning Through the Stability-Plasticity Lens

Paper: 2605.28819 Authors: Yangyi Huang, Ruotian Peng, Zeju Qiu, Jiale Kang, Yandong Wen, Bernhard Schölkopf, Weiyang Liu Categories: cs.LG, cs.CL

The Gap

The field has embraced parameter-efficient finetuning (PEFT) methods like LoRA, adapters, and prefix tuning as the standard for adapting large language models. Evaluations focus almost exclusively on downstream task accuracy—does the model solve the target task better? But there’s a blind spot: nobody systematically measures what the model forgets during finetuning. The pretrained model arrives with broad capabilities (reasoning, world knowledge, instruction following). After PEFT, it might ace your sentiment classification task but lose the ability to write coherent essays or answer factual questions.

Prior work treats PEFT as a pure optimization problem: minimize task loss under parameter constraints. This paper reframes it as a stability-plasticity dilemma—the classic neuroscience trade-off between learning new information (plasticity) and retaining old knowledge (stability). Existing benchmarks don’t measure this trade-off, so we can’t tell which PEFT methods are actually better at balancing adaptation and retention.

Problem: PEFT evaluated only on task accuracy
   |
   v
Observation: Models forget pretrained capabilities
   |
   v
Hypothesis: PEFT methods differ in stability-plasticity profiles
   |
   v
Method: PEFT-Arena benchmark (task perf + retention metrics)
   |
   v
Evidence: Orthogonal finetuning dominates Pareto frontier
   |
   v
Analysis: Geometric view explains why (weight/activation space)
   |
   v
Conclusion: Retention matters; current checkpoints overshoot

The Increment

One sentence: Before this paper, PEFT was evaluated on task accuracy alone; after, we have a benchmark and geometric framework showing that orthogonal methods retain pretrained capabilities better, and that standard training often overshoots the optimal stability-plasticity operating point.

Core Mechanism

PEFT-Arena is a dual-axis benchmark. On one axis, it measures downstream task performance (the usual metric). On the other, it measures general capability retention across multiple dimensions: instruction following, world knowledge, reasoning, and safety alignment. For each PEFT method (LoRA, adapters, prefix tuning, orthogonal finetuning, etc.), the benchmark produces a stability-plasticity profile—a curve showing how task performance trades off against capability retention as you vary hyperparameters like learning rate or rank.

The analysis operates in two geometric spaces. In weight space, the authors decompose PEFT updates using singular value decomposition (SVD) of the pretrained weight matrices. They show that different PEFT parameterizations interact differently with the pretrained singular value structure—some methods align updates with top singular vectors (high forgetting), others spread updates orthogonally (low forgetting). In activation space, they track how finetuning distorts the representations of general-capability inputs. Forgetting correlates with non-isometric distortion: when the geometry of activation space gets warped, the model loses its ability to process inputs it previously handled well.

Input: Pretrained LLM + PEFT method + target task
   |
   v
[PEFT-Arena Benchmark]
   |
   +---> Task Performance (accuracy, F1, etc.)
   |
   +---> Retention Metrics:
         - Instruction following (IFEval)
         - World knowledge (MMLU)
         - Reasoning (GSM8K, BBH)
         - Safety (ToxiGen)
   |
   v
Stability-Plasticity Profile (Pareto curve)
   |
   v
[Geometric Analysis]
   |
   +---> Weight Space: SVD decomposition
   |     (How updates interact with pretrained structure)
   |
   +---> Activation Space: Representation distortion
         (Isometry preservation vs. forgetting)
   |
   v
Insight: Orthogonal methods dominate; checkpoints overshoot

Think of PEFT like renovating a house. The pretrained model is a fully furnished home with plumbing, electricity, and structural integrity. Your target task is adding a new room (say, a home office). Different PEFT methods are different renovation strategies:

• LoRA is like adding furniture and decorations to existing rooms—you’re modifying the space without changing the walls, but you might block doorways or cover up outlets (forgetting).
• Adapters are like building a small addition that connects to the main house—the new room is separate, but you still need to route utilities through the existing structure.
• Orthogonal finetuning is like building the new room on stilts above the house—it doesn’t touch the foundation or interfere with existing rooms. The plumbing and wiring stay intact.

The benchmark measures two things: how well the new room works (task performance) and whether the rest of the house still functions (retention). The geometric analysis is like hiring a structural engineer to explain why some renovation strategies damage the foundation while others don’t. Weight space analysis checks if you’re drilling into load-bearing walls (top singular vectors). Activation space analysis checks if the flow of people and utilities through the house (representations) is still smooth or now has bottlenecks and detours (non-isometric distortion).

Key Concepts

• Stability-Plasticity Dilemma: In neuroscience, this is the trade-off between learning new information and retaining old memories. A perfectly plastic system learns everything instantly but forgets everything just as fast. A perfectly stable system never forgets but can’t learn anything new. Real brains—and now, we argue, PEFT methods—must balance these extremes. In the PEFT context, plasticity is how quickly the model adapts to the target task, and stability is how well it retains pretrained capabilities like reasoning or world knowledge. The dilemma is that most optimization pressures push toward plasticity (lower task loss) at the expense of stability (forgetting). A good PEFT method should reach high task performance without catastrophic retention loss—it should sit on the Pareto frontier of this trade-off.

• Orthogonal Finetuning: Standard finetuning updates weights in whatever direction reduces task loss fastest. Orthogonal finetuning constrains updates to be orthogonal to the top singular vectors of the pretrained weight matrices. Why does this help? Pretrained weights have structure: the top singular vectors capture the most important patterns learned during pretraining (general language understanding, world knowledge). If you update along these directions, you’re overwriting that structure—forgetting. If you update orthogonally, you’re adding new information in “unused” dimensions, leaving the pretrained structure intact. It’s like writing in the margins of a book instead of crossing out the text. The method requires computing SVD of weight matrices and projecting gradient updates onto the orthogonal subspace, which adds computational cost but yields better retention.

• Non-Isometric Representation Distortion: An isometry is a transformation that preserves distances and angles—like rotating or translating an object without stretching or squashing it. In activation space, each input produces a representation (a point in high-dimensional space). If finetuning is isometric, it moves these points around but preserves their relative geometry—inputs that were similar before are still similar after. Non-isometric distortion warps the space: some regions get compressed, others stretched. The paper shows that forgetting correlates with this distortion. When general-capability inputs (e.g., reasoning questions) land in distorted regions, the model can’t process them correctly anymore. Retention requires keeping the activation geometry roughly isometric for non-task inputs. Orthogonal methods achieve this by constraining weight updates to preserve the pretrained representation structure.

Framework Shift

Before (mainstream approach):        After (this paper):

PEFT Method                          PEFT Method
    |                                    |
    v                                    v
Optimize Task Loss                   Optimize Task Loss
    |                                    |
    v                                    +---> Measure Retention
Evaluate Task Accuracy                   |
    |                                    v
    v                                Stability-Plasticity Profile
Done                                     |
                                         +---> Geometric Analysis
                                         |     (Weight + Activation Space)
                                         v
                                     Understand Trade-offs
                                         |
                                         v
                                     Improve Operating Point

From single-axis evaluation (task accuracy) to dual-axis evaluation (task accuracy + retention), the core shift is recognizing that PEFT is not just an optimization problem but a knowledge preservation problem.

Expert Assessment

Problem choice: This is a real gap. The field has been flying blind—optimizing for task performance without measuring what’s lost. The stability-plasticity framing is borrowed from neuroscience, but the application to PEFT is novel and well-motivated. It sits at a critical juncture: as models get larger and more capable, preserving pretrained knowledge becomes more important, not less. The problem will only grow in relevance.

Method maturity: The benchmark design is solid—multiple retention dimensions, diverse tasks, fair comparisons. The geometric analysis is where the paper shines. Connecting forgetting to singular value structure and representation distortion is insightful, not just empirical pattern-matching. However, the orthogonal finetuning method itself isn’t new (prior work exists); the contribution is showing *why it works better through the geometric lens. The path-wise rewinding idea (rolling back to earlier checkpoints) is clever but underdeveloped—it’s a case study, not a full method.

Experimental integrity: Baselines are fair and comprehensive (LoRA, adapters, prefix tuning, full finetuning, orthogonal variants). The retention metrics span multiple capability dimensions, which is thorough. One concern: the tasks are mostly NLP benchmarks (sentiment analysis, NLI, QA). It’s unclear if the findings generalize to other domains like code generation or multimodal tasks. The Pareto frontier analysis is convincing, but the paper doesn’t deeply explore *why some tasks cause more forgetting than others—task characteristics matter, and that’s left underexplored.

Writing quality: The paper is dense but well-structured. The geometric analysis sections (weight space, activation space) are the hardest to parse—more intuitive explanations or visualizations would help. The related work section is thorough but could be trimmed; the space would be better spent on deeper ablations (e.g., how does retention vary with task difficulty or dataset size?). The path-wise rewinding section feels tacked on—it’s interesting but doesn’t integrate cleanly with the rest of the narrative.

Verdict: Strong accept — This paper reframes PEFT evaluation in a way that will influence how the community thinks about finetuning. The benchmark is immediately useful, and the geometric analysis provides actionable insights. It’s not perfect (some ideas are underdeveloped, generalization is unclear), but the core contribution is strong enough to shift the conversation.

Takeaways

1. Evaluate retention, not just task performance: If you’re deploying a finetuned model, test it on general-capability benchmarks (MMLU, GSM8K, instruction following) before and after finetuning. If retention drops significantly, you’ve overfit to the task at the expense of broader utility.

2. Orthogonal finetuning is underused: If you care about retention, try orthogonal methods (e.g., OFT, BOFT). They’re more expensive (SVD computation), but the retention gains are real. For production systems where pretrained capabilities matter, the cost is worth it.

3. Checkpoints overshoot: Standard training runs until task performance plateaus, but the paper shows that earlier checkpoints often have better retention with only slightly worse task performance. Try early stopping based on a retention metric, or use path-wise rewinding to roll back to a better operating point.

4. Geometric intuition transfers: The idea that updates aligned with top singular vectors cause forgetting applies beyond PEFT. If you’re doing any kind of finetuning or continual learning, think about how your updates interact with the pretrained structure. Orthogonality is a general principle for preserving prior knowledge.

5. Stability-plasticity is a design axis: When designing PEFT methods, explicitly consider the trade-off. Don’t just minimize task loss—add a retention term to the objective, or constrain updates to preserve pretrained geometry. The field has been optimizing for plasticity alone; there’s low-hanging fruit in balancing it with stability.