SPQ: An Ensemble Technique for Large Language Model Compression

Paper: 2602.18420 Authors: Jiamin Yao, Eren Gultepe Categories: cs.CL

Abstract

This paper introduces SPQ (SVD-Pruning-Quantization), an ensemble compression technique for large language models that strategically combines three complementary methods: variance-retained singular value decomposition for attention layers, activation-based pruning for MLP layers, and 8-bit post-training quantization across all linear layers. The key insight is that different architectural components exhibit different redundancy patterns, making them amenable to different compression strategies. When applied to LLaMA-2-7B, SPQ achieves up to 75% memory reduction (from 13.48 GB to 6.86 GB) while actually improving perplexity on WikiText-2 from 5.47 to 4.91. The method outperforms single-technique baselines and competitive methods like GPTQ and SparseGPT in both memory efficiency and inference speed, achieving up to 1.9x throughput improvement over GPTQ.

Key Contributions

• An ensemble compression framework that applies different techniques to different model components based on their structural characteristics
• Variance-retained SVD for attention projections that decomposes weight matrices into low-rank factors while preserving critical variance
• Activation-based pruning for MLP layers that removes redundant neurons based on their contribution to model outputs
• Empirical demonstration that combining complementary compression techniques outperforms individual methods at matched compression ratios
• Comprehensive evaluation showing memory reduction, perplexity improvement, and preserved accuracy on downstream tasks (C4, TruthfulQA, GSM8K)

Methodology: Layer-Aware Compression Strategy

The SPQ framework operates on the principle that different layers in transformer architectures have different redundancy characteristics. The attention mechanism, with its projection matrices ($W_Q$, $W_K$, $W_V$, $W_O$), benefits from low-rank decomposition since these projections often capture lower-dimensional semantic relationships. For a weight matrix $W \in \mathbb{R}^{m \times n}$, SVD decomposes it as:

$W = U\Sigma V^T \approx U_r\Sigma_r V_r^T$

where $r \ll \min(m,n)$ is the reduced rank determined by variance retention thresholds.

In contrast, MLP layers exhibit neuron-level redundancy where entire neurons contribute minimally to the output. The pruning strategy computes importance scores based on activation magnitudes across calibration data, removing neurons below a threshold while maintaining model capacity. Finally, 8-bit quantization applies uniformly across all linear layers, mapping floating-point weights to discrete values:

$W_q = \text{round}\left(\frac{W - \min(W)}{\max(W) - \min(W)} \times 255\right)$

This layered approach ensures each compression technique targets the inefficiency it handles best, avoiding the limitations of one-size-fits-all methods.

Experimental Results and Analysis

The experiments on LLaMA-2-7B reveal several important findings. First, at matched compression ratios, SPQ consistently outperforms single-method baselines. For instance, at approximately 50% compression, SPQ achieves perplexity of 4.91 on WikiText-2, compared to 5.23 for SVD-only, 5.67 for pruning-only, and 5.89 for quantization-only approaches. This validates the hypothesis that complementary techniques capture different types of redundancy.

Second, SPQ demonstrates superior memory-accuracy tradeoffs compared to established baselines. Against GPTQ (7.16 GB, perplexity 5.47), SPQ achieves 6.86 GB with perplexity 4.91—both better memory efficiency and better perplexity. Compared to SparseGPT (7.89 GB, perplexity 5.68), the advantages are even more pronounced.

Third, downstream task performance remains robust. On TruthfulQA, SPQ maintains 38.2% accuracy compared to 38.5% for the uncompressed model. On GSM8K, accuracy is 14.6% versus 14.8% baseline. The C4 validation perplexity shows similar trends. This suggests that SPQ’s compression preserves the knowledge and reasoning capabilities encoded in the original model.

Fourth, inference throughput improvements are substantial. SPQ achieves 1.9x speedup over GPTQ in token generation, attributed to reduced memory bandwidth requirements and more efficient matrix operations on compressed representations. This makes SPQ particularly attractive for deployment scenarios where both memory and latency matter.

Implications for LLM Deployment

SPQ addresses a critical bottleneck in deploying large language models: the memory wall. With models like LLaMA-2-7B requiring over 13 GB in FP16 format, deployment on consumer hardware or edge devices becomes impractical. By reducing this to under 7 GB while maintaining or improving quality, SPQ opens new deployment possibilities.

The ensemble approach also provides flexibility. Practitioners can adjust the compression ratio of each component independently based on their specific constraints. For instance, in scenarios with limited memory but abundant compute, one might apply aggressive SVD and pruning with lighter quantization. Conversely, for inference-optimized deployments, heavier quantization might be preferred.

The method’s post-training nature is another practical advantage. Unlike methods requiring retraining or fine-tuning, SPQ can be applied to existing pretrained models without access to original training data or compute resources. This democratizes model compression, making it accessible to researchers and practitioners who lack the resources to train large models from scratch.

Finally, the superior performance compared to GPTQ—a widely adopted quantization method—suggests that the field should explore ensemble approaches more systematically. The success of SPQ indicates that the future of model compression may lie not in perfecting individual techniques but in intelligently combining them.

Takeaways

1. Ensemble compression methods that combine complementary techniques (SVD, pruning, quantization) outperform single-method approaches at matched compression ratios
2. Layer-aware compression strategies that apply different techniques to different architectural components (attention vs. MLP) are more effective than uniform approaches
3. SPQ achieves 75% memory reduction on LLaMA-2-7B (13.48 GB → 6.86 GB) while improving WikiText-2 perplexity from 5.47 to 4.91
4. Compared to GPTQ, SPQ provides better memory efficiency (6.86 GB vs. 7.16 GB), better perplexity, and 1.9x inference speedup
5. Downstream task accuracy remains robust across TruthfulQA, GSM8K, and C4 benchmarks, indicating preserved model capabilities
6. Post-training compression without fine-tuning makes the method practical and accessible for deployment scenarios
7. The success of ensemble approaches suggests future research should focus on intelligently combining compression techniques rather than optimizing individual methods in isolation