ROCKET: Training-Free Model Compression via Knapsack Optimization and Sparse Factorization

Paper: 2602.11008 Authors: Ammar Ali, Baher Mohammad, Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Stamatios Lefkimmiatis Categories: cs.LG, cs.AI, cs.CL

Abstract

ROCKET introduces a training-free model compression framework that achieves state-of-the-art results by combining two innovative techniques. The method formulates layer-wise compression as a multi-choice knapsack problem to optimally allocate compression budgets across layers, minimizing reconstruction error while meeting target model sizes. It then applies a single-step sparse matrix factorization inspired by dictionary learning, using only a small calibration dataset to sparsify weights based on activation-weight sensitivity and updating the dictionary via closed-form least squares. This approach bypasses iterative optimization and backpropagation entirely. ROCKET demonstrates superior performance across various architectures at 20-50% compression rates, retaining over 90% of original performance at 30% compression without fine-tuning, and achieving near-original performance when combined with lightweight fine-tuning.

Key Contributions

• Knapsack-based compression allocation: Formulates optimal layer-wise compression as a multi-choice knapsack problem, enabling principled budget distribution that minimizes global reconstruction error
• Single-step sparse factorization: Introduces a dictionary learning-inspired approach that sparsifies weights based on activation sensitivity and updates dictionaries in closed form, eliminating iterative optimization
• Training-free operation: Achieves compression using only a small calibration set without requiring backpropagation or gradient computation
• Strong empirical results: Retains 90%+ performance at 30% compression without fine-tuning, and achieves near-original performance with minimal fine-tuning (e.g., compressing Qwen3-14B to 8B parameters)

Methodology

ROCKET operates in two main phases: compression budget allocation and sparse matrix factorization.

Phase 1: Knapsack Optimization for Budget Allocation

The method treats each layer as an item in a multi-choice knapsack problem, where each layer can be compressed at different levels (e.g., no compression, 20%, 40%, 60%). For a target global compression budget $B$, ROCKET solves:

$\min_{\{c_i\}} \sum_{i=1}^{L} E_i(c_i) \quad \text{subject to} \quad \sum_{i=1}^{L} S_i(c_i) \leq B$

where $E_i(c_i)$ is the reconstruction error for layer $i$ at compression level $c_i$, and $S_i(c_i)$ is the resulting size. This formulation allows ROCKET to allocate more compression to layers that tolerate it better while preserving critical layers.

Phase 2: Sparse Matrix Factorization

For each layer’s weight matrix $W \in \mathbb{R}^{m \times n}$, ROCKET decomposes it as $W \approx DC$ where $D \in \mathbb{R}^{m \times k}$ is a dense dictionary and $C \in \mathbb{R}^{k \times n}$ is a sparse coefficient matrix. The process:

1. Sensitivity-based sparsification: Using calibration data, compute activation-weight sensitivity scores to identify important coefficients
2. Sparse coding: Retain only top-$s$ coefficients per column in $C$ based on sensitivity
3. Dictionary update: Solve $D^* = \arg\min_D \|W - DC\|_F^2$ in closed form via least squares: $D = WC^T(CC^T)^{-1}$

This single-step approach avoids the alternating optimization typical in dictionary learning, making it extremely fast.

Experimental Results

ROCKET demonstrates consistent improvements across multiple model families and compression rates:

Performance Retention: At 30% compression without fine-tuning, ROCKET retains over 90% of the original model’s performance across benchmarks. This significantly outperforms baseline methods like SVD-based factorization, structured pruning, and dynamic compression techniques.

Compression Range: The method shows strong results across 20-50% compression rates, with graceful degradation as compression increases. At 20% compression, performance loss is often negligible (< 2%), while at 50% compression, the method still maintains 75-80% of original performance.

Fine-tuning Synergy: When combined with lightweight fine-tuning, ROCKET achieves remarkable recovery. The paper demonstrates compressing Qwen3-14B (14 billion parameters) to an 8B-parameter model, then fine-tuning with just 30 million tokens to achieve performance nearly matching the original Qwen3-8B model. This suggests ROCKET creates a strong initialization that requires minimal adaptation.

Architecture Generalization: Results span transformer-based language models of various sizes, demonstrating the method’s applicability across different architectures and scales.

Technical Insights

Several design choices contribute to ROCKET’s effectiveness:

Global vs. Local Optimization: By formulating compression as a global knapsack problem rather than applying uniform compression per layer, ROCKET exploits the heterogeneous sensitivity of different layers. Early layers and attention mechanisms often require less compression than later feed-forward layers.

Activation-aware Sparsification: Unlike magnitude-based pruning, ROCKET’s sensitivity metric considers both weight magnitudes and their interaction with typical activations from the calibration set. This produces more informed sparsity patterns that preserve functional behavior.

Closed-form Updates: The single-step dictionary update via least squares is computationally efficient and avoids local minima issues that plague iterative alternating optimization. This also makes the method deterministic and reproducible.

Calibration Efficiency: ROCKET requires only a small calibration dataset (typically a few hundred samples) to compute sensitivity scores, making it practical for scenarios where full training data is unavailable or expensive to process.

Takeaways

1. Training-free compression can achieve competitive results with trained methods when using principled optimization frameworks like knapsack allocation
2. Global budget optimization across layers significantly outperforms uniform per-layer compression strategies
3. Activation-aware sensitivity metrics provide better sparsification than magnitude-based approaches alone
4. Single-step closed-form factorization can match or exceed iterative optimization methods while being orders of magnitude faster
5. The combination of aggressive compression with minimal fine-tuning (30M tokens) offers a practical path to model size reduction without extensive retraining
6. ROCKET’s training-free nature makes it particularly valuable for compressing proprietary or large-scale models where full retraining is prohibitive