Tucker Attention: Unifying Efficient Attention Through Tensor Decomposition

Paper: 2603.30033 Authors: Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer Categories: cs.LG, cs.AI

The Gap

Attention mechanisms in transformers are memory hogs. Multi-headed attention (MHA) stores separate weight matrices for every head, which scales poorly. Recent fixes like Group-Query Attention (GQA) and Multi-head Latent Attention (MLA) reduce parameters through ad-hoc factorizations — GQA groups heads together, MLA compresses across embedding dimensions. But these methods emerged from engineering intuition, not principled approximation theory. Nobody knows what these factorizations actually approximate, what ranks they achieve, or how they relate to each other.

The gap: we have a zoo of efficient attention variants with no unifying theory. Each method is a point solution. We can’t compare their approximation quality, can’t interpolate between them, and can’t reason about what’s being traded off.

Problem: Memory explosion in MHA
   |
   v
Assumption: Attention weights have low-rank structure
   |
   v
Method: Tucker decomposition of 3D weight tensor
   |
   v
Evidence: 10x fewer parameters, comparable metrics
   |
   v
Conclusion: GQA/MLA are special cases of Tucker factorization

The Increment

One sentence: Before this paper, efficient attention methods were isolated tricks; after, they’re points in a continuous tensor decomposition space with interpretable rank budgets.

Core Mechanism

Standard MHA stores a 3D tensor of weights: (num_heads × head_dim × model_dim). Each head gets its own projection matrix. Tucker Attention factorizes this tensor into three smaller matrices plus a tiny core tensor. Think of it as compressing along three axes simultaneously: across heads, across head dimensions, and across model dimensions.

The factorization works like this: instead of storing the full weight tensor W, you store three factor matrices (U, V, S) and a small core tensor G. To reconstruct any slice of W, you multiply through the core: W ≈ G ×₁ U ×₂ V ×₃ S. The ranks of U, V, S control how much information you keep along each axis. Set the head-axis rank low, you get GQA. Set the embedding-axis rank low, you get MLA. Tune all three, you get Tucker Attention.

Standard MHA:
  Input ---[W_q]---> Q (full rank, all heads independent)
         |
         [W_k]---> K
         |
         [W_v]---> V

Tucker Attention:
  Input ---[S]---[Core G]---[U]---> Q (factored)
         |                    |
         [S]---[Core G]---[V]---> K
         |                    |
         [S]---[Core G]---[U]---> V
         
  Core G: tiny (r_h × r_d × r_m)
  Factor matrices: thin (heads×r_h, dim×r_d, model×r_m)

Here’s the structural metaphor: imagine MHA as a library where every book (head) has its own complete encyclopedia (weight matrix). Tucker Attention is like replacing those encyclopedias with a shared index system. You have three catalogs (the factor matrices U, V, S) and one master reference book (the core tensor G). To look up information for any head, you consult all three catalogs to find the right page in the master reference. The catalogs are thin — they just point to compressed knowledge. The master reference is tiny because it only stores the essential cross-references. GQA is like having multiple heads share the same catalog entry. MLA is like compressing the master reference along one dimension. Tucker does both, plus compresses the catalogs themselves.

Key Concepts

• Tucker Decomposition: Imagine you have a 3D block of numbers (a tensor). Tucker decomposition is like finding three thin slices (one along each dimension) plus a tiny core block, such that when you multiply them back together in a specific way, you approximately reconstruct the original block. It’s the 3D generalization of matrix SVD. The “ranks” are how thick you make each slice — thicker means more accurate reconstruction but more parameters. In attention, the three dimensions are: which head, which position in the head, which position in the model. Tucker lets you compress all three simultaneously with independent rank budgets.

• Rank Budget: In low-rank approximation, rank is how many independent patterns you keep. A rank-1 matrix can only represent one direction of variation. Rank-10 can represent ten independent patterns. In Tucker Attention, you have three rank budgets: r_h (how many independent head patterns), r_d (how many independent within-head patterns), r_m (how many independent model-dimension patterns). GQA only budgets r_h (groups heads). MLA only budgets r_m (compresses embeddings). Tucker budgets all three, which is why it needs fewer total parameters for the same approximation quality.

Framework Shift

Before (GQA/MLA):                After (Tucker Attention):
Compress one axis at a time      Compress all axes jointly
                                 
MHA: [H x D x M] tensor          Tucker: [H x D x M] tensor
  |                                |
  +--GQA--> group H                +---> [r_h x r_d x r_m] core
  |                                      + 3 factor matrices
  +--MLA--> compress M             
                                   GQA = Tucker(r_h=small, r_d=D, r_m=M)
Isolated tricks                   MLA = Tucker(r_h=H, r_d=D, r_m=small)
No shared framework               MHA = Tucker(r_h=H, r_d=D, r_m=M)

From axis-specific compression to joint tensor factorization, the core shift is treating attention weights as a 3D object with exploitable structure along all dimensions simultaneously.

Expert Assessment

Problem choice: Real gap. Memory is the bottleneck for long-context transformers, and the proliferation of ad-hoc attention variants (GQA, MLA, GLA, etc.) genuinely lacks theoretical grounding. This paper asks the right question: what are these methods actually doing in approximation-theoretic terms?

Method maturity: Elegant insight, not brute force. Tucker decomposition is classical (1960s), but applying it to attention weight tensors is non-obvious because attention has never been framed as a 3D approximation problem. The realization that GQA and MLA are special cases is genuinely clarifying. However, the paper doesn’t explore why Tucker specifically — other tensor decompositions (CP, tensor train) might work too.

Experimental integrity: Baselines are fair. They compare against GQA and MLA on equal footing (same model size, same training setup). The LLM experiments (1.3B parameters) and ViT experiments are reasonable scale. One concern: they only report validation perplexity and accuracy, not downstream task performance or inference speed. The “order of magnitude fewer parameters” claim holds for the attention layer, but attention is ~30% of total model parameters, so the end-to-end savings are more modest than the headline suggests.

Writing quality: The paper front-loads tensor notation without building intuition first. Section 2 would benefit from a concrete numerical example (e.g., “here’s a 2×2×2 tensor, here’s its Tucker decomposition with ranks (1,1,1)”). The related work section is thorough but reads like a literature dump. The key insight — that GQA/MLA are rank-constrained Tucker — should be stated in the abstract, not buried in Section 4.

Verdict: weak accept — Solid theoretical contribution that unifies existing methods, but experimental validation is narrow and the practical impact is incremental rather than transformative.

Takeaways

For practitioners: if you’re designing a new efficient attention variant, start with Tucker decomposition and set rank budgets based on where your model has redundancy. Don’t invent a new factorization from scratch — you’re probably rediscovering a special case of Tucker.

For researchers: the “view attention as a 3D tensor” framing is transferable. Other transformer components (feedforward layers, embeddings) might also benefit from joint multi-axis compression instead of axis-by-axis tricks.

Concrete technique: when debugging why your efficient attention variant underperforms, compute the actual achieved ranks along each axis (via SVD of unfolded tensors). If one axis has much lower rank than your budget allows, you’re wasting parameters — reallocate the rank budget.