SLORR: Making Neural Networks Compressible During Training with Minimal Overhead

Paper: 2607.08754 Authors: David González-Martínez, Shiwei Liu Categories: cs.LG, cs.AI

The Gap

Low-rank factorization is one of the most natural ways to compress neural networks — decompose a big weight matrix into two smaller ones. The problem? Modern architectures (ResNets, ViTs, LLMs) are not naturally low-rank. If you just factorize post-training, you lose significant accuracy. So researchers have developed “in-training” regularizers that nudge weights toward low-rank structure during training. But existing approaches come with painful baggage: methods like LR-SGD require full SVDs of weight matrices every step (O(n^3) cost), some like FWSVD introduce additional trainable parameters that alter the architecture, and others like AdaLR rely on stateful cached quantities that complicate implementation and memory management.

SLORR asks: what if you could get the compressibility benefits of in-training regularization without any of these costs — no SVDs, no architecture changes, no state?

Problem: Models resist low-rank compression after training
    |
    v
Existing regularizers help, but require SVD / modify arch / cache state
    |
    v
Assumption: We can approximate low-rank-inducing penalties with GPU-friendly ops
    |
    v
Method: SLORR - apply Hoyer or Nuclear Norm directly on weights
        with cheap forward/backward approximations
    |
    v
Evidence: <8% overhead on ImageNet, <1% on LLMs
        compressed models preserve accuracy far better than baselines
    |
    v
Conclusion: State-of-the-art compressibility with negligible training cost

The Increment

One sentence: Before this paper, making models compressible during training required expensive SVDs or architectural surgery; after this paper, you can do it with a simple regularizer and almost zero overhead.

Core Mechanism

SLORR’s insight is deceptively simple. Instead of directly computing expensive matrix decompositions during training, it regularizes the weight matrices using two proxies for “low-rank-ness”: the Hoyer sparsity of the singular values (which measures how concentrated the spectrum is) and the nuclear norm (the sum of singular values, which is a convex relaxation of rank). Both encourage the weight matrix to have most of its “energy” in a few principal directions — exactly what you want for low-rank factorization.

The key engineering trick is in how SLORR computes these regularizers efficiently. The nuclear norm requires the sum of singular values, which naively needs an SVD. SLORR instead uses a stochastic approximation: multiply the weight matrix by a random Gaussian matrix, compute the Frobenius norm of the result, and use that as an unbiased estimator. For Hoyer sparsity, they approximate the singular value distribution using the matrix’s power iteration trace estimates. Both approximations come with theoretical guarantees on their quality, and both are fully differentiable with cheap backward passes.

Weight Matrix W (original, unmodified)
    |
    +-----> Forward: W = U S V^T (approximated cheaply)
    |              |
    |              v
    |       Approximate singular values
    |              |
    |              v
    |       Compute Hoyer or Nuclear Norm penalty
    |              |
    |              v
    +-----> Add penalty to standard loss
    |
    v
Backward: gradients flow through approximation
          (no SVD in computation graph)
    |
    v
Weight update preserves original architecture

Think of it like tax policy for neural network weights. A government (the optimizer) wants to encourage citizens (weight elements) to consolidate their wealth (singular values) into fewer large accounts (principal components) rather than spreading it evenly. The old approach was to audit every citizen’s full financial portfolio (compute SVD) — accurate but expensive. SLORR instead uses statistical sampling: check a random subset of transactions (random projections) and infer the concentration from that. You lose a tiny bit of precision, but you can run the audit at scale without bankrupting the treasury (your GPU budget). The “tax” (regularization penalty) then gently pushes weights toward concentration, so that when you later apply low-rank factorization (the “withdrawal” — compressing the model), most of the important “wealth” is already in accounts you’re keeping.

Key Concepts

• Hoyer Sparsity: Imagine you have a bag of 100 marbles. If they’re all the same size, nothing is “sparse” — the weight is evenly distributed. If one marble is huge and 99 are tiny, that’s maximally sparse. Hoyer sparsity is a single number between 0 and 1 that captures this: 0 means perfectly uniform, 1 means everything concentrated in one place. In a neural network’s singular values, high Hoyer sparsity means a few singular values dominate — which is exactly what makes a matrix easy to compress via low-rank factorization. SLORR uses this as a differentiable proxy for “how compressible is this weight matrix right now?”

• Stochastic Trace Estimation: How do you measure a property of a huge matrix without doing expensive matrix operations? You cheat — but provably. If you multiply an n×n matrix by a random vector, the expected value of the result’s squared norm equals the matrix’s trace. By using a few random vectors instead of the full identity matrix, you get a good estimate of trace-based quantities (like nuclear norm) at a fraction of the cost. It’s the matrix equivalent of polling: you don’t need to survey everyone, just enough random people to get a reliable answer.

Framework Shift

Before (mainstream approach):          After (this paper):
                                      
Train normally                         Train with SLORR penalty
      |                                     |
      v                                     v
Post-hoc factorization                 Weights already "prepared"
      |                                     |
      v                                     v
Large accuracy drop                    Factorize with minimal loss
      |                                     |
      v                                     v
Fine-tune to recover (if possible)     Deploy directly
      |
      v
Extra training budget needed

From “train then compress then pray” to “train to be compressible,” the core shift is treating low-rank structure as a training objective rather than a post-hoc aspiration.

Expert Assessment

Problem choice: This is a genuine gap. In-training low-rank regularization is a real need — the post-hoc compression pipeline is painful and lossy, and existing regularizers are genuinely cumbersome. The problem sits squarely in the “practical engineering for model compression” sweet spot where small improvements matter a lot to practitioners. It’s not a moonshot paper; it’s a “make the existing toolbox actually usable” paper, and the field needs those.

Method maturity: Clever engineering more than deep insight. The individual pieces — Hoyer sparsity, nuclear norm, stochastic trace estimation — are all known. The contribution is assembling them into something that actually works in practice without SVDs. The approximation guarantees are nice but not surprising given the stochastic linear algebra literature. A simpler approach? Maybe just regularizing the Frobenius norm of the weight matrix after mean-centering, but that’s been tried and doesn’t work as well. The authors made reasonable design choices.

Experimental integrity: The baselines are mostly “unregularized then compressed” which is fair but not ambitious. I’d like to see direct comparison against LR-SGD and FWSVD at matched wall-clock time — the paper emphasizes overhead percentage but doesn’t always show apples-to-apples comparisons against competing regularizers on the same hardware. The LLM experiments at 135M and 560M are modest scales; we’re left extrapolating to real production models. Numbers look clean, no obvious cherry-picking, but the evaluation could be tighter.

Writing quality: The paper reads well and the notation is clean. Section 4 (approximation guarantees) is where it gets dense — a practitioner-friendly walkthrough of “here’s why the approximation doesn’t hurt you” would help. The related work section is comprehensive but could better articulate *why each prior method fails in a specific way, not just that it has limitations. The ablation between Hoyer and Nuclear Norm variants deserves more space — practitioners need to know which to choose.

Verdict: weak accept — Solid incremental engineering that solves a real practical problem, but lacks the experimental depth and head-to-head comparisons against competing regularizers that would make it a clear strong accept.

Takeaways

For practitioners: The stochastic trace estimation trick for approximating nuclear norm is immediately transferable. Anytime you need to regularize a matrix toward low-rank structure — in attention heads, in recommendation models, in any large weight tensor — you can now do it without SVD. The <1% overhead claim for LLMs means you can add this as a default training flag with minimal risk.

For researchers: The framing of “compressibility as a training objective” with GPU-friendly proxies is a template. There are other structural properties we might want (block-sparsity, butterfly structure, Tucker decomposition) where similar stochastic approximation strategies could be applied. The paper opens a lane for “structure-aware training” that goes beyond just low-rank.