Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention

Paper: 2603.21957 Authors: Junhao Du, Jialong Xue, Anqi Li, Jincheng Dai, Guo Lu Categories: cs.CV

The Gap

Video-LLMs process massive visual token volumes—a single video can generate thousands of tokens. Existing compression methods (LLaVA-PruMerge, VideoChat2) use two-stage spatiotemporal compression: first compress frames spatially, then compress across time. This approach assumes spatial and temporal dimensions are separable and uses different metrics for each stage (attention weights for spatial, similarity for temporal).

Under extreme compression (retaining <5% tokens), this separability assumption breaks down. The two-stage pipeline can’t rebalance when one dimension needs more tokens than allocated. A frame with critical evidence might get pruned spatially before temporal selection even sees it. The result: lost visual evidence essential for answering questions.

Problem: Two-stage compression loses critical tokens
   |
   v
Assumption: Spatial/temporal are separable dimensions
   |
   v
Method: Unified global selection pool + text-aware merging
   |
   v
Evidence: 2% retention -> 90.1% performance (vs 85% baseline)
   |
   v
Conclusion: Joint optimization beats staged decomposition

The Increment

One sentence: Before this paper, extreme video compression required choosing between spatial and temporal fidelity; after, a unified selection pool lets tokens compete globally regardless of dimension.

Core Mechanism

The method operates in two phases. First, during visual encoding, all spatiotemporal tokens enter a single global pool. Each token gets scored by combining attention weights (contribution) and semantic similarity to neighbors (redundancy). Tokens are ranked globally—a highly informative token from frame 50 can beat mediocre tokens from frame 1. Top-scoring tokens are kept; the rest are clustered by similarity and merged into representative tokens that refill the pool to a target size.

Second, inside the LLM, after the model generates text queries, a text-aware merging pass happens. Tokens are re-scored based on relevance to the query (via cross-attention between text and visual tokens). Low-relevance tokens get merged again, performing secondary compression that’s question-specific.

Visual Encoder Output (thousands of tokens)
   |
   v
[Global Scoring: attention * (1 - similarity)]
   |
   +---> High score tokens -----> [Keep]
   |                                |
   +---> Low score tokens -----> [Cluster] -> [Merge] -> [Refill]
   |                                              |
   v                                              v
Compressed Pool (2% of original) ---------> LLM Input
                                                  |
                                                  v
                                    [Text Query Generated]
                                                  |
                                                  v
                              [Text-Aware Re-scoring]
                                                  |
                                                  v
                              [Secondary Merge] -> Final Tokens

Think of this like airport security screening. Traditional methods are like having separate lines for domestic (spatial) and international (temporal) passengers, with fixed quotas per line. If the international flight is nearly empty but domestic is packed, you can’t rebalance—people get stuck.

This paper creates one unified security line. Every passenger (token) gets a priority score based on urgency (attention weight) and whether they’re traveling with a large group (similarity to neighbors—redundant). High-priority solo travelers go straight through. Low-priority groups get consolidated onto shuttle buses (clusters) to save space. Then, once you know which gate they’re heading to (text query), you do a second screening to merge passengers going to irrelevant gates. The key: global competition means the most important passengers always get through, regardless of which flight they’re on.

Key Concepts

• Spatiotemporal Separability Assumption: Traditional methods treat video as “compress each frame, then compress across frames”—like packing a suitcase by folding each shirt perfectly, then stacking them. This assumes the best local packing (per-frame compression) leads to best global packing (full video). But under extreme space constraints, you might need to sacrifice one shirt’s folding to fit another critical item. The assumption fails when the budget is too tight for sequential optimization.

• Global Token Pool: Instead of pre-allocating “20% for spatial, 10% for temporal,” all tokens compete in one pool. Imagine a university admissions office that stops separating applicants by major and just ranks everyone by merit. A brilliant physics student doesn’t lose their spot because the physics quota filled—they compete against all applicants. Here, a token from a critical frame at timestamp T=50 can beat tokens from T=1 if it scores higher globally.

• Text-Aware Merging: After the LLM sees the compressed video and generates a query like “What color is the car?”, the method re-scores tokens based on relevance to “car” and “color.” Tokens showing background trees get merged away; tokens showing the car stay. It’s like a librarian who first gives you a condensed book, then after hearing your research question, highlights only the relevant paragraphs and summarizes the rest into footnotes.

Framework Shift

Before (two-stage):                After (unified):

Frame 1 -> [Spatial] -+            All Tokens -> [Global Pool]
Frame 2 -> [Spatial] -+                |
Frame 3 -> [Spatial] -+                v
    ...                |          [Unified Scoring]
Frame N -> [Spatial] -+                |
         |                             v
         v                        [Keep Top K]
   [Temporal Merge]                    |
         |                             v
         v                        [Cluster & Refill]
   Compressed Tokens                   |
                                       v
                                  Compressed Tokens
                                       |
                                       v
                                  [Text-Aware Merge]
                                       |
                                       v
                                  Final Tokens

Fixed stage budgets            Dynamic global allocation
Sequential optimization        Joint optimization
Stage-specific metrics         Unified scoring function

From staged decomposition to global competition, the core shift is treating token selection as a single allocation problem rather than a pipeline of independent compressions.

Expert Assessment

Problem choice: Real gap. Video-LLMs hitting deployment barriers due to token costs is well-documented (LLaVA-NeXT, VideoChat2 both cite this). The focus on ultra-low retention (<5%) is timely—edge devices and real-time applications need this. Not manufactured.

Method maturity: Clever insight with solid execution. The unified pool idea is simple but non-obvious—most prior work stayed in the two-stage paradigm. The text-aware merging is a natural extension once you have the LLM’s query. However, the clustering-based merging feels like it could be simpler—why not just weighted averaging of low-score tokens? The paper doesn’t ablate this choice clearly.

Experimental integrity: Baselines are fair (LLaVA-PruMerge, FastV, VideoChat2). Numbers look solid across multiple benchmarks (MSVD-QA, MSRVTT-QA, ActivityNet-QA). The 90.1% performance retention at 2% tokens is impressive but needs context—what’s the variance? One red flag: no comparison with learned compression methods (e.g., trainable token pruning). The “plug-and-play” constraint is reasonable but limits the ceiling.

Writing quality: Section 3.2 (method) is dense—too many details crammed into notation-heavy paragraphs. The intuition gets buried. Figure 2 helps but arrives late. If they rewrote Section 3.2 to lead with the airport security analogy, then formalize, the paper would be 30% clearer. Also, the related work section undersells how different this is from prior work—readers might miss the novelty.

Verdict: weak accept — Solid contribution with clear practical value, but method presentation needs work and experimental scope could be broader (missing learned baselines, variance analysis).

Takeaways

Practitioners can steal the “global pool” framing for any multi-stage compression problem. If you’re doing hierarchical pruning (spatial then temporal, or layer-wise in transformers), ask: does my stage budget allocation make sense, or should I let all candidates compete globally? The unified scoring function (attention × inverse-similarity) is a simple heuristic worth trying before building complex learned pruners.

The text-aware merging idea transfers to any multimodal system where you have a query after initial compression—re-score and re-compress based on query relevance. This is essentially late fusion of task information into compression.

One subtle technique: the “cluster and refill” approach to preserve information from pruned tokens. Instead of discarding low-score tokens, merge them into representatives and add back to the pool. This is cheaper than keeping all tokens but avoids the information cliff of hard pruning. Useful for any token reduction pipeline.