Tango: Smarter Token Pruning for Efficient Video LLMs

Paper: 2604.09547 Authors: Shukang Yin, Sirui Zhao, Hanchao Wang, Baozhi Jia, Xianquan Wang, Chaoyou Fu, Enhong Chen Categories: cs.CV

The Gap

Video LLMs are slow because they process too many visual tokens. Token pruning emerged as the fix: throw away unimportant tokens, keep the model fast. Two camps formed: attention-based selection (keep tokens the model looks at most) and similarity-based clustering (merge similar tokens). Both sound reasonable. Both have hidden flaws.

Attention-based methods use top-k selection: rank tokens by attention score, keep the top k. Problem: attention distributions are spatially multi-modal (multiple hotspots) and long-tailed (few tokens get most attention, many get scraps). Top-k blindly cuts at a threshold, ignoring this structure. You might keep 10 tokens from one hotspot and zero from another equally important region.

Similarity clustering merges tokens that look alike. Problem: direct clustering fragments the video into tiny, scattered clusters. When you pool these fragments, you get distorted representations—like averaging GPS coordinates from random street corners and claiming it represents the city center.

Problem: Video LLMs too slow (too many tokens)
   |
   v
Assumption: Can prune tokens without losing info
   |
   +---> Method 1: Attention selection (top-k)
   |        |
   |        v
   |     Flaw: Ignores multi-modal distribution
   |
   +---> Method 2: Similarity clustering
            |
            v
         Flaw: Creates fragmented clusters
   |
   v
Tango: Diversity-driven selection + ST-RoPE
   |
   v
Evidence: 98.9% performance with 10% tokens
   |
   v
Conclusion: Structure-aware pruning > naive ranking

The Increment

One sentence: Before Tango, token pruning treated attention as a flat ranking and similarity as a clustering problem; after Tango, pruning respects attention’s spatial structure and preserves geometric relationships through position embeddings.

Core Mechanism

Tango has two components working in tandem. First, diversity-driven token selection replaces top-k with a smarter strategy. Instead of just picking the highest attention scores, it divides the spatial grid into regions and ensures each region contributes tokens proportionally to its total attention mass. If one region has 30% of the attention, it gets 30% of the kept tokens. This preserves multi-modal structure.

Second, Spatio-temporal Rotary Position Embedding (ST-RoPE) injects geometric priors into the similarity space. Standard clustering treats tokens as abstract vectors—two tokens could be visually similar but spatially distant, and clustering wouldn’t care. ST-RoPE encodes each token’s position (frame number, x, y coordinates) using rotary embeddings, so nearby tokens naturally cluster together. When you pool a cluster, you’re averaging a coherent spatial region, not random fragments.

Input Video Tokens
   |
   v
[Attention Scores] -----> Diversity-Driven Selection
   |                         |
   |                         v
   |                   Spatial Grid Division
   |                         |
   |                         v
   |                   Region-Proportional Sampling
   |                         |
   v                         v
[Token Features] -------> ST-RoPE Encoding
   |                         |
   v                         v
Similarity Matrix -----> Locality-Aware Clustering
   |                         |
   v                         v
Cluster Pooling --------> Pruned Tokens

Think of it like organizing a photo album. The naive approach (top-k) is like keeping only the brightest photos, which might all be from one sunny day, losing the story arc. Diversity-driven selection is like ensuring each chapter (region) of your trip gets representation proportional to how many good shots you took there. ST-RoPE is like keeping photos in chronological and spatial order—you don’t mix beach photos with mountain photos just because the colors look similar. When you create a summary collage (pooling), each cluster represents a coherent moment, not a random grab bag.

Key Concepts

• Multi-modal attention distribution: Imagine a heat map of where people look in a video frame. It’s not a single bright spot—it’s multiple hotspots (faces, moving objects, text). Top-k selection is like saying “keep the 10 brightest pixels” which might all come from one hotspot, ignoring others. Multi-modal means the distribution has multiple peaks, and you need to sample from all of them, not just the tallest.

• Rotary Position Embedding (RoPE): Standard position encoding adds a vector to each token: token_1 gets [1,0,0], token_2 gets [0,1,0]. RoPE instead rotates the token’s feature vector in high-dimensional space based on its position. Why? Because rotation preserves relative distances—if token A and B are neighbors, their rotated vectors stay close. When you compute similarity, nearby tokens naturally score higher. It’s like encoding GPS coordinates not as numbers but as angles on a compass: nearby locations point in similar directions.

• Cluster fragmentation: When you cluster tokens by similarity alone, you might group “all blue pixels” together even if they’re from different objects in different frames. This creates tiny, scattered clusters. Pooling them is like making a smoothie from ingredients picked randomly across your kitchen—the result doesn’t represent any coherent dish. ST-RoPE prevents this by making spatial neighbors more similar, so clusters become coherent regions.

Framework Shift

Before (mainstream approach):        After (Tango):

Attention Scores                     Attention Scores
      |                                    |
      v                                    v
  Top-k Ranking                      Spatial Grid
      |                                    |
      v                                    v
  Keep Highest                       Region-Proportional
      |                              Selection
      |                                    |
      v                                    v
Token Features                       Token Features
      |                                    |
      v                                    v
Similarity Matrix                    ST-RoPE Encoding
      |                                    |
      v                                    v
Direct Clustering                    Locality-Aware
      |                              Clustering
      v                                    |
Fragmented Clusters                        v
                                     Coherent Clusters

From flat ranking to structure-aware sampling, the core shift is treating spatial and temporal geometry as first-class constraints, not afterthoughts.

Expert Assessment

Problem choice: Real gap. Video LLMs are genuinely bottlenecked by token count, and existing pruning methods do leave performance on the table. The observation about multi-modal attention and cluster fragmentation is sharp—these aren’t manufactured problems, they’re visible in attention visualizations and cluster quality metrics.

Method maturity: Clever insight, not brute force. Diversity-driven selection is conceptually simple (proportional sampling) but addresses a real oversight in top-k. ST-RoPE is elegant—reusing rotary embeddings for spatial encoding is a natural fit. However, the paper doesn’t explore whether simpler fixes (like stratified sampling without full grid division) would suffice. The method feels well-motivated but possibly over-engineered for the gain.

Experimental integrity: Baselines are fair (compares against recent token pruning methods like FastV, LLaVA-PruMerge). The 98.9% performance retention at 10% tokens is impressive, but the paper tests primarily on LLaVA-OV. Generalization claims would be stronger with more diverse architectures. The 1.88x speedup is honest (includes overhead), not just theoretical FLOP reduction. No obvious red flags, but ablations could be more granular—hard to tell if diversity-driven selection or ST-RoPE contributes more.

Writing quality: The abstract and intro are crisp. The method section gets dense—ST-RoPE’s mathematical formulation could use a worked example. The related work section is thorough but reads like a checklist. If I were revising, I’d rewrite Section 3.2 (ST-RoPE) with a visual walkthrough showing how position encoding changes similarity scores for nearby vs distant tokens. That one change would make the paper 30% more accessible.

Verdict: weak accept — Solid contribution with clear improvements over baselines, but incremental rather than paradigm-shifting. The ideas are sound and the results hold up, but the method’s complexity raises questions about whether simpler variants were adequately explored.

Takeaways

Practitioners can steal the diversity-driven selection strategy for any attention-based pruning task—it’s not video-specific. The core idea (sample proportionally from spatial regions instead of global top-k) applies to image transformers, document attention, or any domain where attention has spatial structure. Implementation is straightforward: grid the input, sum attention per region, allocate tokens proportionally.

ST-RoPE’s insight—that position embeddings can guide clustering—transfers to any scenario where you’re merging similar items but want to preserve locality. Think: merging database records (prefer merging entries from the same time period), compressing sensor data (cluster nearby readings), or even text summarization (keep sentences from different sections, not just the most salient ones).

The broader lesson: when pruning or compressing, respect the structure of your data’s distribution. Flat rankings and naive clustering ignore geometry. Adding lightweight structural priors (spatial grids, position embeddings) often costs little but preserves much.