One Token per Frame: Extreme Video Compression for Long-Form Understanding

Paper: 2604.14149 Authors: Zheyu Zhang, Ziqi Pang, Shixing Chen, Xiang Hao, Vimal Bhat, Yu-Xiong Wang Categories: cs.CV

The Gap

Vision-language models (VLMs) for video understanding hit a hard wall: each frame explodes into tens or hundreds of tokens, and LLMs have fixed context windows. Existing approaches like VideoChat, LLaVA-Video, and LongVA use heuristic compression (uniform sampling, clustering, pooling) to fit videos into context limits. The problem? Heuristics are blind to what matters. They compress before knowing the question, lose temporal coherence, and still can’t handle truly long videos densely.

The specific boundary: prior work achieves ~10-20 tokens per frame at best, forcing sparse sampling (1 FPS or less for hour-long videos). This creates a trilemma: sample densely and run out of tokens, sample sparsely and miss events, or use heuristics and lose information.

Problem: Long video (1000s frames) + Limited context (4K-32K tokens)
   |
   v
Assumption: Compression must be (1) learnable not heuristic
            (2) progressive across LLM layers (3) query-aware
   |
   v
Method: LP-Comp (token-level) + QC-Comp (frame-level)
   |
   v
Evidence: 1 token/frame, 2-4x denser sampling, +3.3% on LVBench
   |
   v
Conclusion: Extreme compression enables dense long video understanding

The Increment

One sentence: Before this paper, VLMs compressed videos heuristically before processing; after, compression is learned progressively through LLM layers and conditioned on the question, achieving 1 token per frame.

Core Mechanism

The method has two compression stages working in tandem. First, LP-Comp (Learnable Progressive Compression) operates at the token level within each frame. Instead of heuristically pooling frame tokens before the LLM, it inserts lightweight compression modules between LLM layers. Each module learns to merge tokens progressively—early layers preserve detail, deeper layers aggressively compress. By the final layer, each frame is squeezed to exactly one token. This is supervised: the model is trained to predict which tokens to merge using ground-truth labels from attention patterns.

Second, QC-Comp (Question-Conditioned Compression) operates at the frame level. After LP-Comp produces one token per frame, QC-Comp selects which frames to keep based on the question. It uses the LLM’s internal attention scores—specifically, how much the question tokens attend to each frame token—as a relevance signal. Frames with high attention survive; others are dropped. Crucially, to avoid LLM’s position bias (over-attending to start/end), videos are split into short segments and local attention is computed within each segment.

Input: Video frames (N frames, each ~100 tokens)
   |
   v
[LP-Comp: Token-level compression]
   Layer 1-8:  100 tokens/frame -> 50 tokens/frame
   Layer 9-16: 50 tokens/frame  -> 10 tokens/frame
   Layer 17+:  10 tokens/frame  -> 1 token/frame
   |
   v
Intermediate: N frames, 1 token each
   |
   v
[QC-Comp: Frame-level compression]
   Split into segments -> Compute local attention
   -> Rank frames by question relevance
   -> Keep top K frames
   |
   v
Output: K frames (K << N), 1 token each -> LLM final layers

Think of it like packing for a trip with a tiny suitcase. LP-Comp is folding each piece of clothing (frame) into the smallest possible bundle—you start by folding loosely (early layers), then tighter and tighter (deeper layers) until each item is vacuum-sealed to a single compact cube. QC-Comp is then deciding which cubes to actually pack based on your itinerary (the question)—if you’re going to the beach, you keep swimwear cubes and ditch the winter coat cubes. The segment-based attention is like checking each cube’s relevance within its category (tops, bottoms, accessories) rather than comparing a swimsuit directly to a winter coat, which would be biased by their positions in your closet.

Key Concepts

• Progressive compression across LLM layers: Traditional VLMs compress video tokens before feeding them to the LLM—it’s a one-shot, all-or-nothing operation. Progressive compression instead spreads the compression across the LLM’s depth. Early layers see high-resolution token representations and learn semantic groupings. Middle layers merge related tokens. Deep layers perform aggressive compression. Why does this work? Because LLM layers naturally form a hierarchy of abstraction—early layers capture low-level features, deep layers capture high-level semantics. By aligning compression with this hierarchy, the model can make informed decisions about what to discard at each level. Concrete example: In a cooking video, early layers might preserve tokens for “hand,” “knife,” “onion” separately; middle layers merge “hand + knife” into “cutting action”; final layers compress the entire frame to “food preparation.”

• Attention-based frame selection with position bias mitigation: LLMs exhibit position bias in long contexts—they over-attend to the first and last tokens, even if middle tokens are more relevant. This is catastrophic for video understanding: if you ask “What happened in the middle of the video?” the model might still focus on the opening and closing frames. The solution: split the video into short segments (e.g., 10 frames each) and compute attention locally within each segment. Now, within each segment, the “first” and “last” positions rotate, so every frame gets a fair chance to be selected based on actual relevance, not position. Example: In a 100-frame video, instead of computing attention over all 100 frames (where frames 1, 2, 99, 100 dominate), split into 10 segments of 10 frames each, compute attention within each segment, then aggregate. Frame 50 can now be selected if it’s relevant to the question, even though it’s in the middle.

• Supervised compression tuning: The compression modules (which tokens to merge, which frames to keep) need to learn what’s important. But “importance” is task-dependent—what matters for action recognition differs from what matters for question answering. Supervised compression tuning trains these modules using a small amount of labeled data (2.5% of the full fine-tuning dataset) where the model learns to compress in a way that preserves task-relevant information. The supervision signal comes from comparing compressed vs. uncompressed representations on downstream tasks. Think of it like training a summarizer: you don’t just teach it to shorten text, you teach it to shorten text while preserving the answer to specific questions.

Framework Shift

Before (mainstream approach):        After (this paper):

Video frames                         Video frames
     |                                    |
     v                                    v
[Heuristic compression]              [LLM Layer 1-8]
(pooling/sampling)                        |
     |                                    v
     v                               [LP-Comp module]
[Fixed tokens]                       (merge tokens)
     |                                    |
     v                                    v
[LLM processes]                      [LLM Layer 9-16]
     |                                    |
     v                                    v
[Output]                             [LP-Comp module]
                                     (merge more)
                                          |
                                          v
                                     [LLM Layer 17+]
                                          |
                                          v
                                     [1 token/frame]
                                          |
                                          v
                                     [QC-Comp: frame selection]
                                          |
                                          v
                                     [Output]

Compression: Before LLM              Compression: During + After LLM
Decision: Blind to content           Decision: Informed by layers + query

From blind pre-compression to informed progressive compression, the core shift is making compression a learned, query-aware process integrated into the model’s reasoning, not a preprocessing step.

Expert Assessment

Problem choice: Real gap. Long video understanding is a genuine bottleneck—hour-long videos are increasingly common (lectures, surveillance, movies), and current VLMs choke on them. The trilemma (dense sampling vs. context limits vs. information loss) is not manufactured. However, the problem sits in a crowded space—many recent papers (LongVA, VideoChat2, LLaMA-VID) tackle the same issue. The novelty is in the solution approach, not the problem identification.

Method maturity: Clever insight with some brute force. The progressive compression idea is elegant—it aligns with how LLMs naturally process information hierarchically. The attention-based frame selection is smart. But the position bias mitigation (splitting into segments) feels like a workaround rather than a principled solution—it’s patching an LLM limitation rather than addressing it. Also, the method requires a separate supervised compression tuning stage, which adds complexity. A simpler approach might be end-to-end training with a compression loss, but the authors don’t explore this.

Experimental integrity: Baselines are fair—they compare against strong recent models (VideoChat-Flash, LLaVA-Video). The numbers are solid: +3.3% on LVBench, improvements on EgoSchema, VideoMME. However, there’s a red flag: the supervised compression tuning uses only 2.5% of data, yet yields large gains. This suggests the base model (VideoChat-Flash) was undertrained or poorly tuned for long videos. The gains might be partially from better training, not just better compression. Also, the paper doesn’t report inference speed—extreme compression should speed things up, but no numbers are provided.

Writing quality: The paper is dense and assumes familiarity with recent VLM literature. The method section jumps between token-level and frame-level compression without clearly delineating them first—a reader unfamiliar with the space will get lost. The ablation studies are thorough, but buried in the appendix. If the authors rewrote Section 3 to first give a high-level overview (two-stage compression: token then frame), then dive into each stage separately, the paper would be far more accessible. Also, the name “XC” (extreme compression) is generic and unmemorable.

Verdict: Weak accept — solid contribution to a real problem, but incremental over recent work and the writing could be clearer. The method is clever but not groundbreaking, and the experimental gains, while consistent, are modest.

Takeaways

Progressive compression across model layers: Don’t compress all at once before your model. If your model has a hierarchical structure (like transformers), insert compression modules at multiple depths. Early layers preserve detail, deep layers compress aggressively. This is applicable beyond video—think long documents, multi-image inputs, or time-series data.

Attention scores as a relevance signal: If you need to select a subset of inputs based on a query, use your model’s internal attention scores. They’re already computed, they’re query-conditioned, and they’re free. Just watch out for position bias in long sequences.

Segment-based processing to mitigate position bias: If your model over-attends to the start/end of long sequences, split the sequence into short segments and process locally. This is a simple, effective hack for any long-context task.

Supervised compression tuning as a separate stage: If your compression is learnable, you can train it with a small amount of labeled data (2-5% of your full dataset) as a separate stage. This is data-efficient and lets you adapt compression to specific tasks without full retraining.