JACTUS: Unifying Model Compression and Adaptation in a Single Subspace

Paper: 2605.02829 Authors: Jingze Ge, Yun Liu, Xue Geng, Wanqi Dong, Wang Zhe Mark, Min Wu, Xulei Yang Categories: cs.AI

The Gap

The field has two mature strategies for deploying large pretrained models: parameter-efficient fine-tuning (PEFT) keeps the full model frozen and trains small adapters, while low-rank compression reduces model size by approximating weight matrices. The standard practice chains them sequentially—compress first, then add adapters. This decoupling creates a fundamental misalignment: compression chooses directions that preserve pretrained knowledge, but adaptation needs directions aligned with downstream task gradients. The compressed subspace might discard exactly the dimensions the task needs, forcing adapters to compensate inefficiently. Worse, the parameter budget is split arbitrarily between compression and adaptation rather than allocated globally by utility.

Problem: Sequential pipeline wastes budget
   |
   v
Assumption: Compression & adaptation need overlapping but distinct subspaces
   |
   v
Method: Form union of pretrained + task subspaces, compress inside union
   |
   v
Evidence: 89.2% ViT accuracy at 80% params (vs 87.9% full PEFT baseline)
   |
   v
Conclusion: Joint optimization beats decoupled compress-then-adapt

The Increment

One sentence: Before JACTUS, you compressed blindly then adapted; after JACTUS, you peek at the task to compress only what adaptation won’t need.

Core Mechanism

JACTUS operates in three phases. First, it estimates two covariance matrices from a small calibration set: input covariances (capturing data distribution) and pre-activation gradient covariances (capturing task-relevant directions). Second, it constructs a union subspace by taking the top eigenvectors from these covariances and orthogonalizing them against the pretrained weight’s existing subspace. This union explicitly includes both what the pretrained model already knows and what the task will need. Third, it performs low-rank approximation constrained to live inside this union, allocates rank globally across all layers by marginal accuracy gain per parameter, and trains only a compact core matrix while discarding the full frozen weights.

The key insight is projection before approximation. Standard compression approximates weights in the ambient space, then hopes adapters can fix errors. JACTUS first projects into a task-aware subspace, then approximates there. This guarantees the compressed model retains capacity along task-critical directions. The global rank allocation further optimizes the parameter budget: layers that benefit more from extra rank get it, rather than uniform allocation.

Calibration Set
   |
   +---> Input Covariance ----+
   |                          |
   +---> Gradient Covariance -+
                              |
                              v
                    Union Subspace = Orthogonalize(Pretrained + Task)
                              |
                              v
                    Project Weights into Union
                              |
                              v
                    Low-Rank Approx inside Union
                              |
                              v
                    Global Rank Allocation (by marginal gain)
                              |
                              v
                    Train Core Matrix Only

Think of JACTUS as packing a suitcase for a trip. The standard approach (compress-then-adapt) is like packing your clothes without knowing the destination, then buying new outfits when you arrive and realize you brought the wrong things. JACTUS checks the weather forecast (calibration set) before packing. It identifies what you’ll actually need (task gradients), keeps those items, and only compresses the rest. The suitcase size is fixed (parameter budget), but now every item inside is chosen knowing both what you own (pretrained weights) and what you’ll use (task requirements). The global rank allocation is like giving more space to versatile items (layers with high marginal gain) rather than splitting space equally among all clothes.

Key Concepts

• Subspace union: Imagine you have two flashlights pointing in different directions in a dark room. One illuminates the pretrained model’s knowledge (what it learned during pretraining), the other illuminates the task’s requirements (gradients from the downstream objective). Most compression methods only use the first flashlight, so they might throw away objects the second flashlight reveals as important. JACTUS turns on both flashlights simultaneously and keeps everything either light touches. Mathematically, it takes the top eigenvectors from input and gradient covariances, orthogonalizes them against the pretrained weight subspace, and forms their span. This union is the “safe zone” where compression can happen without discarding task-critical directions.

• Projected low-rank approximation: Standard low-rank approximation (like SVD) finds the best rank-k approximation in the full ambient space. JACTUS restricts the search to the union subspace. It’s like being told “you can only rearrange furniture within this room” instead of “you can put furniture anywhere in the house.” This constraint is actually helpful because the room was chosen to contain everything important. The projection ensures the approximation error is orthogonal to task-relevant directions, so the compressed model doesn’t lose capacity where it matters.

• Global rank allocation: Instead of giving every layer the same rank budget (e.g., rank 8 everywhere), JACTUS measures how much accuracy each layer gains per additional parameter and allocates rank accordingly. A layer that improves 2% with 1000 extra parameters gets more rank than a layer that improves 0.5% with the same budget. This is computed greedily: start with minimal rank everywhere, then iteratively give one more rank to whichever layer has the highest marginal gain, until the parameter budget is exhausted. The result is a heterogeneous rank profile tailored to the model and task.

Framework Shift

Before (compress-then-adapt):        After (JACTUS):

Pretrained Weights                   Pretrained Weights
       |                                    |
       v                                    v
  Compress (SVD)                      Calibration Set
       |                                    |
       v                                    v
  Low-Rank Model              Estimate Task Subspace
       |                                    |
   (frozen)                                 v
       |                          Union(Pretrained + Task)
       v                                    |
  Add Adapters                              v
       |                          Compress inside Union
       v                                    |
  Fine-Tune Adapters                        v
                                   Train Core (no frozen weights)

From sequential decoupling to joint optimization, the core shift is treating compression and adaptation as a single budget allocation problem rather than two independent stages.

Expert Assessment

Problem choice: This is a real gap. The compress-then-finetune pipeline is widely used in production (model serving costs matter), but the misalignment between compression objectives and task objectives is well-known and under-addressed. The problem sits at the intersection of efficiency and adaptation, both hot areas, so the timing is good.

Method maturity: The core idea—using task information to guide compression—is elegant and well-motivated. The execution is solid: covariance estimation is standard, orthogonal union is clean, global rank allocation is principled. However, the method requires a calibration set, which adds a dependency. The paper doesn’t deeply explore what happens when calibration data is scarce or mismatched. Also, the comparison against “compress-then-finetune” baselines is somewhat unfair because those baselines use 100% of parameters during finetuning (frozen weights + adapters), while JACTUS discards frozen weights. A fairer comparison would be compress-then-finetune with the same 80% total parameter budget.

Experimental integrity: The baselines are strong (DoRA, LoRA, full finetuning) and the datasets are standard (vision: CIFAR, SVHN, etc.; language: commonsense QA). The numbers are convincing: 89.2% vs 87.9% on ViT, 80.9% vs 79.7% on Llama2. However, the paper reports averages across datasets without showing per-dataset breakdowns or variance, which makes it hard to assess robustness. The ablation studies are present but shallow—no analysis of sensitivity to calibration set size or quality. The claim that JACTUS “avoids retaining full frozen weights” is a deployment advantage, but the paper doesn’t measure actual inference speedup or memory savings in practice.

Writing quality: The abstract and introduction are clear, but the method section is dense and notation-heavy. The paper would benefit from a standalone “algorithm box” summarizing the full procedure in pseudocode. The related work section conflates compression and PEFT literature without clearly delineating what JACTUS inherits from each. The experimental section front-loads tables without enough narrative to guide interpretation. Rewriting Section 4 to lead with insights (“global rank allocation matters most”) before dumping numbers would elevate the whole paper.

Verdict: weak accept — The idea is sound and results are strong, but experimental rigor and presentation need tightening before this becomes a landmark paper.

Takeaways

Practitioners can steal the global rank allocation strategy even outside this specific framework. If you’re deploying low-rank adapters (LoRA, etc.), don’t use uniform rank across layers—profile each layer’s sensitivity and allocate rank by marginal gain. This is cheap to compute (one forward pass per rank increment) and often yields 1-2% accuracy for free.

The broader lesson: when composing two optimization stages (compression + adaptation, quantization + finetuning, pruning + distillation), check if they’re fighting over the same parameter budget. If so, consider joint optimization or at least peek at the second stage’s objective when making first-stage decisions. The “union of subspaces” framing is a clean way to formalize this for linear operations.