AdaCodec: Predictive Visual Coding for Efficient Video MLLMs

Paper: 2606.02569 Authors: Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan Categories: cs.CV, cs.AI, cs.CL

The Gap

Current video multimodal large language models treat each sampled frame as an independent RGB image, encoding it from scratch regardless of temporal redundancy. This per-frame encoding wastes visual tokens on repeated content—adjacent frames share most objects, background, and layout. The core inefficiency: existing video MLLMs lack a mechanism to distinguish “what’s new” from “what’s already known” across frames. Qwen3-VL-8B using per-frame RGB encoding requires 224k visual tokens for long videos, causing 9.26s time-to-first-token delays.

Problem: Video temporal redundancy ignored
    |
    v
Observation: Adjacent frames share 80%+ content
    |
    v
Hypothesis: Encode full frames only when unpredictable
    |           otherwise send compact deltas
    v
Method: AdaCodec with I-frames + P-tokens
    |
    v
Evidence: 1/7 tokens, same/better performance
    |
    v
Conclusion: Predictive coding >> per-frame encoding

The Increment

One sentence: Before—every video frame encoded independently as RGB; after—full frames only when unpredictable, compact motion/residual codes otherwise.

Core Mechanism

AdaCodec operates like video compression but for MLLM visual tokens. It maintains a predictive model that estimates the next frame from prior context. When prediction quality is high (low residual error), AdaCodec sends only P-tokens encoding motion vectors and small residuals—typically 10-20 tokens per frame. When prediction fails (scene cut, new object), it sends a full I-frame with complete visual tokens—around 256 tokens. A gating mechanism decides frame-by-frame: compute conditional predictive cost; if below threshold, emit P-tokens; if above, emit I-frame.

The architecture has three components: (1) a reference frame encoder producing full visual embeddings, (2) a predictive module estimating inter-frame changes via optical flow and residual prediction, (3) a compact P-token encoder translating motion/residuals into ~15 tokens. Data flows as: reference frame → full encoding → prediction for next frame → compute prediction error → if error low, encode delta as P-tokens; if error high, encode next frame fully as new I-frame.

Frame sequence:  F1 -----> F2 -----> F3 -----> F4
                 |         |         |         |
Predictive cost: LOW ....> LOW ....> HIGH ...> LOW
                 |         |         |         |
Action:          I-frame   P-tokens  I-frame   P-tokens
                 (256 tok) (15 tok)  (256 tok) (15 tok)
                    |         |         |         |
                    v         v         v         v
                 +-------------------------------------+
                 |   MLLM Context Window (562 tokens) |
                 +-------------------------------------+

Think of AdaCodec as a storyteller who knows when you’re already following along. For the opening scene, she describes everything—the room, the people, the lighting (I-frame). As the conversation continues and people just shift in their chairs, she says “John leans left, Mary nods” (P-tokens)—you fill in the rest from memory. When someone new walks in, she gives you the full picture again (new I-frame). The predictive model is her theory of your mental state; the gating threshold is her judgment of when your mental image has drifted too far from reality. Motion vectors are “John leans left”—directional updates. Residuals are “but his shirt is now wrinkled”—details the motion doesn’t capture. Without this mechanism, she’d describe the entire scene every second, even when nothing changes.

Key Concepts

• Conditional predictive cost: The expected encoding length needed to represent a frame given prior context. Low cost means the frame is highly predictable from what came before—a static camera with small object motion. High cost signals unpredictability—scene cuts, new objects, camera jumps. AdaCodec measures this by computing how well a learned predictor reconstructs the frame from previous embeddings; reconstruction error correlates with encoding cost. Concrete example: in a lecture video where the speaker stands still, frames 2-100 have low predictive cost (only lip movement); when the slide changes at frame 101, cost spikes.

• P-tokens vs I-frames: Borrowed from video codecs. An I-frame is a full, independently decodable visual encoding (256 tokens capturing the entire frame). A P-token is a compact delta encoding (15 tokens) describing changes relative to a predicted frame—motion direction and prediction residuals. The MLLM reconstructs the frame by applying P-tokens to its predicted state. Example: if frame N shows a person at position (100,50) and frame N+1 shows them at (105,50), the P-token encodes motion vector (+5,0) rather than re-encoding the entire person.

• Gating mechanism: The decision rule determining I-frame vs P-token encoding. AdaCodec computes a predictive error score for each frame; if score > threshold τ, emit I-frame; else emit P-tokens. The threshold τ controls the tradeoff: lower τ means more I-frames (higher quality, more tokens), higher τ means more P-tokens (lower tokens, potential quality loss). Training learns both the predictor and the optimal τ per dataset. Example: on long interview videos, τ is set high (mostly P-tokens); on action movies with frequent cuts, τ is lower (more I-frames).

Framework Shift

Before (per-frame RGB):              After (AdaCodec):

Frame 1 ---> [Encoder] ---> 256 tok  Frame 1 ---> [Encoder] ---> 256 tok (I)
                             |                      |
Frame 2 ---> [Encoder] ---> 256 tok                v
                             |        Frame 2 ---> [Predictor] ---> error low
Frame 3 ---> [Encoder] ---> 256 tok                |            |
                             |                     v            v
Frame 4 ---> [Encoder] ---> 256 tok             [P-encoder]   15 tok (P)
                                                    |
Total: 1024 tokens                   Frame 3 ---> [Predictor] ---> error high
All frames independent                             |            |
                                                   v            v
                                                [Encoder]    256 tok (I)

                                     Total: 527 tokens
                                     Dependencies exploited

From uniform per-frame encoding to adaptive predictive coding, the core shift is: pay full cost only when the past doesn’t buy you anything.

Expert Assessment

Problem choice: Real gap. Video temporal redundancy has been exploited in codecs for decades, but video MLLMs ignored it because the dominant paradigm treats vision encoders as black boxes operating per-frame. This paper brings a codec-style interface to the MLLM level—obvious in retrospect, but genuinely underexplored. It sits at the intersection of efficient video understanding and MLLM scaling, both hot areas.

Method maturity: The core idea is clean—predict, measure error, gate. The execution is competent but not groundbreaking. The predictive module uses standard optical flow estimation plus a learned residual encoder. The gating threshold is set via grid search per benchmark rather than learned end-to-end, which feels like a missed opportunity. A simpler approach might be fixed I-frame intervals (like H.264 GOP structure), but adaptive gating does outperform fixed patterns in their ablations. No red flags, but also no deep novelty beyond the application context.

Experimental integrity: Baselines are fair—they compare against Qwen3-VL-8B with matched total token budgets and also at equal per-frame sampling. The 1/7 token reduction claim holds across 11 benchmarks, with consistent wins on long-video tasks. Numbers look credible: 32k AdaCodec beats 224k baseline on all five long-video benchmarks. Time-to-first-token improvement (9.26s → 1.62s) is measured correctly. One concern: the predictive model adds computational overhead during encoding, but they don’t report wall-clock encoding time, only inference latency. Also, all experiments are on Qwen3-VL; generalization to other video MLLMs is claimed but not demonstrated.

Writing quality: The paper is readable and well-motivated. The introduction clearly articulates the redundancy problem. The method section is detailed enough to reproduce. However, the related work section conflates video compression with MLLM visual encoding without clarifying why codec techniques haven’t been adopted before—was it a technical barrier or just an oversight? The ablation study (Section 4.4) is thorough, showing the impact of gating threshold and P-token length. The weakest section is 4.1 (implementation details)—it buries key architectural choices in a dense paragraph. Rewriting that section with a clear component breakdown and a table of hyperparameters would improve clarity.

Verdict: weak accept — Solid engineering contribution with clear practical value, but limited conceptual novelty; the idea is “obvious once stated,” and the execution is competent but doesn’t push boundaries.

Takeaways

1. Adaptive gating for token budgets: The predictive cost → threshold → encoding decision pattern applies beyond video. Any sequential input with temporal correlation (audio, time-series, multi-turn dialogues) could use a similar mechanism: send full context only when prediction fails, otherwise send deltas. Concrete: for multi-turn chat, encode the first message fully, then encode subsequent messages as “topic shifts” or “continuation tokens.”

2. Metric before mechanism: AdaCodec defines the predictive cost metric first, then builds the encoding around it. This is transferable: for any sequential task, define a predictability score (e.g., next-token perplexity, embedding distance), then let it drive your compression or caching strategy.

3. Exposure of codec primitives to MLLMs: Video codecs hide I-frames and P-frames inside binary streams. AdaCodec exposes them as differentiable tokens the MLLM can attend to. This suggests a broader pattern: MLLMs benefit from explicit representations of temporal structure (keyframes, deltas) rather than treating all frames uniformly. Practitioners working on long-context video could experiment with tagged tokens (e.g., [KEYFRAME], [DELTA]) to give the model explicit temporal cues.