VideoMLA: Compressing Video Diffusion Memory by 92.7% with Low-Rank Latent Attention

Paper: 2605.30351 Authors: Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Hoda Eldardiry, Pinar Yanardag Categories: cs.CV, cs.AI

The Gap

Long-form autoregressive video diffusion models generate videos by repeatedly predicting the next frame, maintaining a sliding-window KV cache to remember recent context. Current methods (CogVideoX, Pyramid Flow, Allegro) all use the same per-head KV cache layout inherited from language models: each attention head stores its own full-dimensional keys and values for every cached token. Recent innovations have focused on which tokens to cache or how to encode their positions, but the per-head storage layout itself—the dominant contributor to memory and latency—has remained untouched.

The problem: generating minute-scale videos requires caching thousands of tokens across dozens of layers. At standard dimensions (32 heads × 128 dims per head), this becomes the bottleneck. Prior work assumed this layout was necessary because attention heads need independent key/value representations. No one had tested whether video diffusion actually requires this redundancy.

Problem: Long video generation hits memory wall
   |
   v
Assumption: Per-head KV cache layout is necessary
   |
   v
Method: Replace per-head KV with shared low-rank latent + decoupled RoPE
   |
   v
Evidence: 92.7% memory reduction, quality maintained, throughput +23%
   |
   v
Conclusion: Video attention doesn't need per-head KV redundancy

The Increment

One sentence: Before this paper, video diffusion cached full per-head keys and values; after, a single shared low-rank latent plus positional encoding achieves the same quality at 7.3% of the memory cost.

Core Mechanism

VideoMLA replaces the standard multi-head attention KV cache with two shared components. First, instead of storing separate 128-dimensional keys and values for each of 32 heads (4096 dims total per token), it stores a single 512-dimensional “content latent” shared across all heads. Second, instead of baking positional information into per-head keys, it maintains a separate 512-dimensional “positional key” that encodes 3D spatiotemporal position using RoPE (Rotary Position Embedding).

During attention computation, each head projects the shared content latent into its own key and value subspaces using learned down-projection matrices. The positional key is similarly projected per-head and added to the content-derived key. Queries interact with these reconstructed keys normally. The critical insight: by factoring out position and compressing content into a shared bottleneck, the model eliminates per-head redundancy while preserving the ability to specialize attention patterns.

Standard Multi-Head KV Cache:
Head 1: [K1: 128d] [V1: 128d]  \
Head 2: [K2: 128d] [V2: 128d]   |-- 4096d per token
  ...                           |
Head 32:[K32:128d] [V32:128d] /

VideoMLA Shared Cache:
[Content Latent: 512d] [Position Key: 512d]  <-- 1024d per token
         |                      |
         v                      v
    Down-project          Down-project
    per-head              per-head
         |                      |
         +----------------------+
         v
    [K_h: 128d] [V_h: 128d] for each head h

Think of it like a library system. The standard approach is like giving each reader their own complete copy of every book (per-head KV cache). VideoMLA instead maintains one master copy in the archive (content latent) and one index card system tracking where each book sits on the shelf (positional key). When a reader needs a book, the librarian (down-projection) pulls the relevant pages from the master copy and combines them with the location info. Each reader still gets a personalized view, but the library doesn’t need 32 redundant copies of everything. The compression works because most of the information in those 32 copies was identical—only the “personalization” (the projection) differs per reader.

Key Concepts

• Low-rank latent compression: In linear algebra, a matrix is “low-rank” if most of its information concentrates in a few dominant directions, like how a pancake is mostly flat (2D) even though it exists in 3D space. VideoMLA bets that the keys and values across all attention heads share most of their information—they’re redundant. Instead of storing 32 separate 128-dimensional vectors (a 32×128 matrix), it stores one 512-dimensional vector (a compressed representation) and reconstructs the full matrix on-the-fly using learned projections. The paper shows this works even though pretrained video attention isn’t naturally low-rank: the training process learns to *use the low-rank budget efficiently, adapting within the bottleneck rather than requiring the data to already fit it. Concrete example: imagine 32 people describing the same scene. Instead of recording all 32 descriptions verbatim (per-head storage), you write one detailed summary (latent) and 32 short notes on what each person emphasized (projections). You lose some nuance but save 90% of the storage.

• Decoupled positional encoding: Standard attention bakes position into the keys themselves—a key for token 5 is fundamentally different from a key for token 500 because position is mixed into the representation. VideoMLA separates “what is this token about” (content latent) from “where is this token in space-time” (positional key). This decoupling has two benefits: (1) the content latent can be shared across all positions without position-specific redundancy, and (2) the positional key can use 3D-RoPE to encode height, width, and time dimensions independently. Think of it like a filing system: instead of writing “Q2 2023 Sales Report - Northeast Region” on every page (position baked in), you write “Sales Report” on the pages and keep a separate index that says “this document is Q2 2023, Northeast” (position decoupled). The content stays clean and reusable.

Framework Shift

Before (per-head KV cache):
Token 1 --> [H1_K, H1_V] [H2_K, H2_V] ... [H32_K, H32_V]
Token 2 --> [H1_K, H1_V] [H2_K, H2_V] ... [H32_K, H32_V]
  ...
Token N --> [H1_K, H1_V] [H2_K, H2_V] ... [H32_K, H32_V]
            ^                                           ^
            |---- Each head stores full K,V per token --|

After (VideoMLA):
Token 1 --> [Content_Latent] [Position_Key]
Token 2 --> [Content_Latent] [Position_Key]
  ...                |              |
Token N --> [Content_Latent] [Position_Key]
                     |              |
                     v              v
            Project per-head --> [H1_K,V] ... [H32_K,V]
            ^                                          ^
            |-- Shared storage, per-head reconstruction --|

From redundant per-head storage to shared compressed storage with on-demand reconstruction, the core shift is factoring out redundancy and deferring specialization to computation time rather than memory time.

Expert Assessment

Problem choice: Real gap. Minute-scale video generation is a clear frontier, and memory is the binding constraint. The paper correctly identifies that prior work optimized around the KV layout rather than questioning it. This sits at the intersection of scaling (longer videos) and efficiency (practical deployment)—both high-value directions.

Method maturity: Clever adaptation, not invention. MLA was introduced for language models (DeepSeek-V2), and this paper is the first to apply it to video diffusion. The insight is recognizing that the method transfers despite video attention not being naturally low-rank (the paper’s spectral analysis is the most interesting contribution). The approach is clean: no architectural gymnastics, just swap the cache layout. However, the paper doesn’t explore *why the bottleneck works—it shows empirically that training fills the rank budget, but the mechanism remains a black box.

Experimental integrity: Baselines are fair (CogVideoX, Pyramid Flow, Allegro—all recent SOTA). The VBench evaluation is comprehensive. The 92.7% memory reduction is measured correctly (per-token KV size). One concern: the throughput improvement (1.23×) is modest given the memory savings, suggesting the bottleneck shifted elsewhere (likely computation or bandwidth). The paper doesn’t ablate latent dimension thoroughly—512d is chosen to match DeepSeek-V2, but is this optimal for video? The spectral analysis is rigorous, but the claim that “MLA determines effective rank from initialization” needs more mechanistic explanation.

Writing quality: The spectral analysis section (why MLA works despite non-low-rank attention) is the paper’s strongest contribution but feels rushed. Expanding this with ablations on initialization strategies and training dynamics would elevate the work from “MLA works for video” to “here’s why compression works even when the data doesn’t cooperate.” The related work section undersells the novelty—this is the first MLA application outside language models, which deserves emphasis.

Verdict: weak accept — Solid empirical contribution with a surprising negative result (pretrained attention isn’t low-rank, yet compression works). The method is practical and the memory savings are significant. However, the paper is more “successful application” than “deep insight”—it shows *that MLA works for video but not fully why. The spectral analysis hints at interesting training dynamics but doesn’t close the loop.

Takeaways

Transferable technique: The decoupled position encoding (separate content and position keys) is worth stealing for any domain with structured positional information (3D data, graphs, time series). It’s cleaner than baking position into representations and enables better compression.

Counterintuitive insight: Compression can work even when the data isn’t naturally compressible—the bottleneck forces the model to learn a compressed representation during training. This suggests that architectural constraints can act as implicit regularizers. Practitioners working on memory-constrained models should consider whether their architecture is *too expressive, allowing the model to memorize redundancy rather than compress it.

Practical heuristic: When optimizing memory, look for redundancy across parallel structures (heads, layers, experts) before optimizing within a single structure. The per-head KV cache was the elephant in the room because everyone assumed heads needed independence. Question inherited layouts.

Caveat: The modest throughput gain (1.23×) despite 92.7% memory reduction suggests that memory isn’t always the bottleneck—measure your actual bottleneck before assuming compression will speed things up. The win here is enabling longer rollouts within a fixed memory budget, not necessarily faster generation.