Quantization-Robust LLM Unlearning via Low-Rank Adaptation

Paper: 2602.13151 Authors: João Vitor Boer Abitante, Joana Meneguzzo Pasquali, Luan Fonseca Garcia, Ewerton de Oliveira, Thomas da Silva Paula, Rodrigo C. Barros, Lucas S. Kupssinskü Categories: cs.LG, cs.CL

Abstract

Large language models deployed in production environments typically undergo post-training quantization (PTQ) to reduce memory footprint and inference costs. However, this paper reveals a critical problem: aggressive low-bit quantization can completely erase the effects of machine unlearning, causing models to revert to their pre-unlearning state. The authors demonstrate that standard full-parameter fine-tuning for unlearning produces parameter changes too subtle to survive 4-bit quantization. Their solution leverages Low-Rank Adaptation (LoRA) to concentrate unlearning updates into trainable adapter modules while freezing the base model, ensuring these updates remain effective post-quantization. Experiments on Llama-2-7B using the MUSE benchmark show substantial improvements: up to 7.93 points in 4-bit utility and significant reductions in privacy leakage while maintaining strong forgetting guarantees.

Key Contributions

• Identification of quantization vulnerability: Demonstrates that standard unlearning methods fail under aggressive PTQ, with parameter changes being too small to survive 4-bit quantization
• LoRA-based solution: Proposes using Low-Rank Adaptation to concentrate unlearning updates in trainable adapters, making them robust to quantization
• Comprehensive evaluation: Tests multiple unlearning algorithms (NPO, GA) with retention methods (GDR, KLR) on MUSE dataset (BOOKS and NEWS domains)
• Quantifiable improvements: Shows up to 7.93 point utility gains and substantial privacy leakage reduction (e.g., PrivLeak from -25.68 to -5.86) under 4-bit quantization

The Quantization-Unlearning Conflict

Machine unlearning aims to remove specific knowledge from trained models, addressing privacy concerns and regulatory requirements like GDPR’s “right to be forgotten.” However, practical LLM deployment demands quantization for efficiency. This creates a fundamental tension: unlearning typically modifies a small subset of parameters with subtle changes, while aggressive quantization rounds weights to a limited set of discrete values.

The paper demonstrates this conflict empirically. When standard full-parameter unlearning is applied followed by 4-bit PTQ, the quantization process effectively “rounds away” the unlearning updates. The model’s behavior reverts to its pre-unlearning state, defeating the purpose of the unlearning procedure. This happens because the magnitude of parameter changes induced by unlearning is often smaller than the quantization bin width in 4-bit representations.

The core insight is that quantization acts as a lossy compression that disproportionately affects small, distributed parameter changes—exactly the type produced by standard unlearning methods.

Methodology: LoRA for Quantization-Robust Unlearning

The proposed solution exploits the structure of Low-Rank Adaptation. Instead of fine-tuning all model parameters, LoRA freezes the pre-trained weight matrix $W_0$ and introduces trainable low-rank decomposition matrices $A$ and $B$ such that the effective weight becomes:

$W = W_0 + BA$

where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ with rank $r \ll \min(d,k)$.

For unlearning, this architecture offers a crucial advantage: the unlearning updates are concentrated entirely in the adapter matrices $A$ and $B$, which remain in full precision even when the base model $W_0$ is quantized. The quantization process becomes:

$W_{quantized} = Q(W_0) + BA$

where $Q(\cdot)$ represents the quantization function. Since $BA$ is not quantized, the unlearning signal is preserved regardless of how aggressively $W_0$ is compressed.

The authors evaluate this approach with multiple unlearning algorithms:

• Negative Preference Optimization (NPO): Optimizes the model to assign lower probability to forget samples
• Gradient Ascent (GA): Directly maximizes loss on forget data

Combined with retention strategies:

• Gradient Difference Retention (GDR): Balances forgetting and retention gradients
• KL Retention (KLR): Maintains distribution similarity on retain data

Experimental Results and Analysis

Testing on Llama-2-7B with the MUSE benchmark reveals striking improvements. On the BOOKS domain with NPO+GDR, LoRA increases 4-bit utility from 50.17 to 58.10 (7.93 point gain). For NEWS domain with GA+GDR, utility improves from 40.06 to 44.82 (4.76 point increase).

Privacy metrics show even more dramatic improvements. The PrivLeak metric measures how much the model’s predictions on forget data deviate from a reference model (ideal value: 0). For GA+KLR on BOOKS, standard unlearning yields PrivLeak of -25.68 under 4-bit quantization (indicating severe privacy leakage), while LoRA achieves -5.86—much closer to the ideal.

Critically, these utility and privacy gains come without sacrificing forgetting effectiveness. Metrics like VerMem (verbatim memorization) and KnowMem (knowledge memorization) remain near zero, confirming that the targeted information is successfully removed.

The results demonstrate that LoRA doesn’t just preserve unlearning under quantization—it often outperforms full-parameter methods even in full precision scenarios, likely due to the regularization effect of the low-rank constraint.

Implications for Production Deployment

This work has immediate practical implications for organizations deploying LLMs under privacy constraints. Many real-world scenarios require both efficient inference (necessitating quantization) and the ability to remove user data on request (requiring unlearning). Previously, these requirements were in tension.

The LoRA-based approach enables a viable workflow:

1. Train base model and quantize for deployment
2. When unlearning is needed, apply LoRA adapters
3. Keep base model quantized while adapters remain full precision
4. Deploy with minimal memory overhead (adapters are small)

The memory cost is minimal since LoRA adapters typically add less than 1% of parameters. For a 7B parameter model with rank $r=8$, the adapter overhead is negligible compared to the 4× memory savings from 4-bit quantization.

This also opens possibilities for modular unlearning: different adapters could handle different unlearning requests, allowing fine-grained control over what knowledge is removed without retraining the entire model.

Takeaways

1. Standard unlearning methods fail under aggressive quantization because parameter changes are too small to survive discretization
2. LoRA concentrates unlearning updates in full-precision adapters, making them robust to base model quantization
3. On Llama-2-7B with MUSE, LoRA improves 4-bit utility by up to 7.93 points while dramatically reducing privacy leakage
4. The approach enables practical deployment scenarios where both efficiency (quantization) and privacy (unlearning) are required
5. LoRA’s low-rank constraint may provide beneficial regularization, sometimes outperforming full-parameter methods even without quantization