CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing

Paper: 2602.15823 Authors: Zarif Ikram, Arad Firouzkouhi, Stephen Tu, Mahdi Soltanolkotabi, Paria Rashidinejad Categories: cs.LG, cs.AI

Abstract

CrispEdit addresses a fundamental challenge in large language model editing: how to modify specific behaviors without degrading general capabilities. The method formulates editing as a constrained optimization problem where capability preservation is an explicit constraint rather than an implicit hope. By projecting edit updates onto the low-curvature subspace of the capability-loss landscape, CrispEdit achieves targeted edits while maintaining model performance. The approach leverages Bregman divergence to express capability constraints, yielding the Gauss-Newton Hessian exactly. Through Kronecker-factored approximate curvature (K-FAC) and a novel matrix-free projector, the method scales efficiently to LLM sizes. Experimental results show high edit success rates with capability degradation below 1% on average, significantly outperforming existing editors.

Key Contributions

• Unified Framework: Formulates LLM editing as constrained optimization with explicit capability preservation, unifying and generalizing existing approaches like ROME, MEMIT, and fine-tuning methods
• Bregman Divergence Constraint: Expresses capability preservation via Bregman divergence, whose quadratic form yields the exact Gauss-Newton Hessian even for non-converged models
• Scalable Second-Order Method: Implements efficient second-order optimization at LLM scale using K-FAC and a novel matrix-free projector that exploits Kronecker structure
• Superior Performance: Achieves high edit success while keeping capability degradation below 1% across standard benchmarks, substantially improving over prior editors

Methodology

CrispEdit’s core innovation lies in treating capability preservation as a hard constraint rather than a soft penalty. The method solves:

$\min_{\theta} \mathcal{L}_{\text{edit}}(\theta) \quad \text{subject to} \quad D_{\text{Breg}}(\theta, \theta_0) \leq \epsilon$

where $D_{\text{Breg}}$ is the Bregman divergence measuring deviation from the original parameters $\theta_0$, and $\mathcal{L}_{\text{edit}}$ is the editing objective.

The algorithm proceeds in two stages:

1. Unconstrained Update: Compute a gradient-based update $\Delta\theta$ that minimizes the editing loss
2. Projection: Project $\Delta\theta$ onto the feasible region defined by the capability constraint

The projection step is where CrispEdit’s theoretical elegance shines. By expressing the constraint via Bregman divergence with respect to the capability loss $\mathcal{L}_{\text{cap}}$, the feasible region’s boundary is characterized by the Gauss-Newton Hessian $H_{\text{GN}}$. The projection becomes:

$\Delta\theta_{\text{proj}} = \Delta\theta - H_{\text{GN}}^{-1}g$

where $g$ is chosen to satisfy the constraint. This formulation avoids the need for the full Hessian of $\mathcal{L}_{\text{cap}}$, which would be computationally prohibitive.

To make this tractable at scale, CrispEdit employs K-FAC to approximate $H_{\text{GN}}$ with Kronecker factors. The key insight is that for neural network layers, the Gauss-Newton Hessian can be approximated as:

$H_{\text{GN}} \approx A \otimes G$

where $A$ captures input statistics and $G$ captures gradient statistics. The novel matrix-free projector then exploits this Kronecker structure to perform the projection without ever materializing the full matrix, reducing memory requirements from $O(d^2)$ to $O(d)$ where $d$ is the parameter dimension.

Experimental Results

CrispEdit was evaluated on standard model-editing benchmarks including CounterFact, zsRE, and multi-hop reasoning tasks. The results demonstrate substantial improvements over existing methods:

Edit Success vs. Capability Preservation: While methods like ROME and MEMIT achieve high edit success rates (85-90%), they suffer from capability degradation of 5-15% on downstream tasks. Fine-tuning approaches preserve capabilities better but have lower edit success (70-80%). CrispEdit achieves the best of both worlds: 88-92% edit success with less than 1% capability degradation on average.

Scalability: The K-FAC approximation and matrix-free projector enable CrispEdit to scale to models with billions of parameters. Runtime overhead compared to first-order methods is modest (1.5-2x), while second-order baselines that don’t exploit Kronecker structure are 10-100x slower or infeasible.

Robustness: CrispEdit maintains performance across different model architectures (GPT-2, LLaMA, Mistral) and edit types (factual updates, behavior modifications, bias corrections). The method is particularly effective for sequential edits, where capability preservation becomes increasingly critical.

Ablation Studies: Removing the projection step (equivalent to unconstrained optimization) leads to 8-12% capability degradation. Using first-order approximations instead of the Gauss-Newton Hessian reduces edit success by 5-7%. These results validate the importance of both the constrained formulation and the second-order curvature information.

Implications and Future Directions

CrispEdit represents a significant step toward practical, reliable LLM editing. The explicit treatment of capability preservation as a constraint rather than a regularization term provides stronger guarantees and better empirical performance. Several directions emerge for future work:

Theoretical Guarantees: While CrispEdit provides strong empirical results, formal analysis of convergence rates and generalization bounds would strengthen the theoretical foundation. Understanding when the Gauss-Newton approximation is tight and how constraint violation relates to capability degradation are open questions.

Adaptive Constraint Selection: The constraint radius $\epsilon$ is currently set manually. Adaptive methods that automatically tune $\epsilon$ based on edit difficulty and model sensitivity could improve usability and performance.

Multi-Objective Editing: Extending CrispEdit to handle multiple simultaneous constraints (e.g., preserving multiple capabilities while making multiple edits) is a natural generalization with practical applications.

Beyond Parameter Editing: The constrained optimization framework could potentially apply to other model modification techniques like prompt tuning, adapter modules, or mixture-of-experts routing.

The success of CrispEdit suggests that principled optimization methods with explicit constraints may be more effective than heuristic approaches for model editing. As LLMs become more widely deployed, the ability to make targeted, non-destructive edits will be increasingly valuable for maintaining and improving these systems.

Takeaways

1. Capability preservation should be an explicit constraint, not just a regularization term, when editing LLMs to avoid silent degradation of general abilities
2. Bregman divergence provides a principled way to express capability constraints that yields tractable second-order optimization via the Gauss-Newton Hessian
3. Kronecker-factored approximations and matrix-free projectors make second-order methods scalable to billion-parameter models with modest computational overhead
4. CrispEdit achieves less than 1% capability degradation while maintaining high edit success rates, substantially outperforming existing editors
5. The constrained optimization framework unifies and generalizes several existing editing approaches, providing a theoretical foundation for understanding their trade-offs