DepthWeave-KV: Smarter KV Cache Compression by Sharing Across Layers and Adapting Per Token

Paper: 2607.06523 Authors: Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jesus Olivera Categories: cs.AI

The Gap

Long-context LLM inference is choking on KV cache memory. At 64K tokens, a 32-layer model stores millions of key-value pairs, and existing compression methods treat this as a flat budgeting problem: apply the same compression ratio to every layer and every token. Methods like GQA, quantized caches, and eviction-based approaches (e.g., H{2}O, SnapKV) either reduce dimensionality uniformly or drop tokens based on a single attention score snapshot. The problem: not all tokens are equal. Instruction-bearing tokens and retrieval-critical passages need higher fidelity, while filler context can tolerate aggressive compression. And not all layers are equal — neighboring transformer layers often learn similar representations, creating redundancy that flat methods waste. Nobody has jointly exploited both cross-layer redundancy and per-token sensitivity in a single, training-free compression scheme.

Problem: KV cache is the memory bottleneck at long context
    |
    v
Prior work: Uniform compression (same budget per layer, per token)
    |
    v
Limitation: Sacrifices retrieval-critical tokens; ignores cross-layer redundancy
    |
    v
Assumption: Layers share structure; tokens vary in importance
    |
    v
Method: Cross-layer shared basis + token-adaptive residuals + online routing
    |
    v
Evidence: 8.3x compression, near-full quality on LongBench / Needle / L-Eval
    |
    v
Conclusion: Adaptive, cross-layer factorization beats uniform compression

The Increment

One sentence: Before this paper, KV cache compression was a flat per-layer, per-token budgeting problem; after this paper, it is a structured, cross-layer factorization problem with token-level adaptive rank allocation — and it works without retraining.

Core Mechanism

DepthWeave-KV has three interlocking components. First, cross-depth residual factorization: instead of storing full key and value vectors independently at each layer, the method computes a shared low-rank channel basis from pairs (or groups) of neighboring layers. Each layer’s actual K/V state is reconstructed as the shared basis plus a lightweight per-layer residual that captures what the basis misses. This exploits the empirical observation that adjacent transformer layers often produce highly correlated representations.

Second, a token-conditional depth router decides how much reconstruction budget each token gets. The router inspects attention statistics (entropy, peak scores) to classify tokens: instruction-carrying and retrieval-critical tokens get higher-rank residuals, while generic context tokens get aggressively compressed. This is not a one-shot decision — the router runs online during generation.

Third, calibration-free error tracking via attention-output probes monitors reconstruction quality in real time. The probes compare what the compressed cache would produce against an error signal, and feed this back to the router so it can reallocate rank on the fly. No retraining, no offline calibration dataset — everything adapts during inference.

A fused CUDA kernel ties it all together: basis lookup, residual dequantization, and attention projection happen in a single pass, minimizing memory traffic.

Layer i   K,V --+                   +--> Attention Projection
                |                   |
                +--> Shared Basis B-+--> + Residual R_i (low-rank, quantized)
                                    |
Layer i+1 K,V --+                  +--> + Residual R_{i+1}
                |                  |
                +--> Basis B ------+
                                   
                                    ^
                                    |
                        Token Router (rank allocation)
                                    ^
                                    |
                        Error Probe (attention-output signal)

Think of this like a restaurant kitchen with a shared stock system. The kitchen maintains a few master stocks (the shared low-rank bases) — a rich chicken stock, a vegetable fond — that underpin dozens of dishes across multiple stations (layers). Each dish (token at a layer) starts from the shared stock but gets its own finishing touches: a splash of lemon, a pinch of herbs (the per-token residual). The head chef (depth router) walks the line and decides: this dish is the house special — give it the full garnish treatment; that dish is a side — the stock alone is fine. And there’s a tasting station (error probe) where dishes are sampled before they go out: if something tastes flat, the chef adjusts the garnish in real time. The genius is that you don’t need a separate recipe for every dish from scratch — you share the expensive base and customize only where it matters.

Key Concepts

• Cross-Layer Residual Factorization: Imagine two adjacent floors of an office building share the same structural frame (beams, columns). Each floor still has its own furniture and layout, but the heavy lifting — the skeleton — is shared. In transformers, neighboring layers often learn very similar internal representations. Instead of storing the full K/V matrix at both layer 5 and layer 6, you store one shared “skeleton” (low-rank basis) and two tiny “furniture” residuals that capture the differences. This is where the memory savings come from — the skeleton is compact, and the residuals are small.

• Token-Conditional Depth Routing: Not every sentence in a 64K context window deserves the same memory budget. A passage that answers a question is critical; boilerplate legal text is not. The router looks at how “peaked” each token’s attention pattern is (low entropy = highly focused attention = likely important) and assigns a reconstruction rank accordingly. Think of it like a triage nurse: walk in with a cold, you get ibuprofen; walk in with chest pain, you get the full cardiac workup. The router does this triage per-token, per-generation-step.

• Calibration-Free Online Adaptation: Most compression methods need a calibration dataset to set their parameters, or they fix the compression scheme before inference. DepthWeave-KV avoids this by probing the attention output during generation itself. If the compressed cache is producing significantly different attention results than expected, the error signal tells the router to increase rank for the offending tokens. It’s like a thermostat that adjusts heating based on the actual room temperature, not a pre-programmed schedule.

Framework Shift

Before (mainstream):                   After (DepthWeave-KV):

Layer 5: [full K][full V]              Layer 5:  [shared basis B]
Layer 6: [full K][full V]              Layer 6:  [shared basis B]
Layer 7: [full K][full V]              Layer 7:  [shared basis B]
  ...                                    ...
Each layer stores independently.       + per-layer residual (small, quantized)
Uniform budget per token.              + per-token rank (router allocates)
Static compression.                    + online error probe (adapts during gen)

From flat per-layer storage to structured cross-layer factorization with adaptive token-level allocation, the core shift is treating KV cache compression as a shared-subspace reconstruction problem rather than an independent quantization problem.

Expert Assessment

Problem choice: This is a real gap. KV cache memory is the dominant bottleneck for long-context inference, and the field has mostly pursued either layer-agnostic eviction or uniform quantization. The insight that neighboring layers share exploitable structure and that tokens vary in reconstruction sensitivity is well-grounded in prior empirical observations (e.g., layer similarity studies, attention entropy analysis). The paper positions itself at the intersection of two known phenomena that nobody had combined in a training-free system — that’s genuine, not manufactured.

Method maturity: Clever engineering with a solid core insight. The cross-layer basis sharing is not entirely novel (similar ideas appear in weight-sharing literature), but applying it to the KV cache with token-adaptive residuals and online routing is a meaningful synthesis. The fused CUDA kernel suggests the authors actually care about wall-clock performance, not just FLOP counts — that’s a good sign. One concern: the router’s reliance on attention entropy as a proxy for token importance is heuristic. It works in practice, but it’s not principled — a adversarial context could potentially fool it.

Experimental integrity: The benchmarks are comprehensive and appropriate — LongBench for general long-context tasks, Needle-in-a-Haystack for retrieval, L-Eval for long-form generation, plus QA and summarization. The 8.3x compression with near-full quality claim is strong, and the 72.8 tokens/sec throughput at 64K context shows real system-level gains. Baselines include prior compressed cache methods. The numbers look credible, though I’d want to see more ablation on the router specifically — how much of the gain comes from cross-layer sharing vs. adaptive routing? The abstract doesn’t mention this breakdown, which is a yellow flag.

Writing quality: The abstract is dense but well-structured — it names the problem, the method, the mechanism, and the results in a logical flow. Without seeing the full paper, the main risk is that the method section could become a wall of notation. The ASCII diagram I drew above is what the paper should aspire to in clarity. If the authors rewrote the experimental analysis to clearly separate the contribution of each component (basis sharing, routing, error tracking), the paper would be significantly stronger.

Verdict: weak accept — Solid system paper combining known insights into a practical, training-free compression scheme with real throughput gains, but the ablation story needs sharpening and the router heuristic needs more principled justification.

Takeaways

Three things worth stealing:

1. Cross-layer redundancy is free memory. If you’re designing any system that processes stacked transformer layers, check whether adjacent layers’ internal states are correlated enough to share a representation. This idea transfers to weight sharing, activation caching, and even speculative decoding state management.

2. Token-level adaptive budgets beat uniform budgets. The triage principle — spend more resources where errors are more costly — is obvious in theory but rarely implemented cleanly in KV cache systems. The attention-entropy proxy for token importance is a practical, cheap heuristic that any KV eviction or compression scheme could adopt.

3. Online error probes eliminate calibration datasets. If you can measure reconstruction quality during inference itself (even approximately), you can adapt on the fly. This is a design pattern that transfers to quantization, pruning, and any lossy compression in an inference pipeline.