AutoGaze: Teaching Video Models to Look Before They See

Paper: 2603.12254 Authors: Baifeng Shi, Stephanie Fu, Long Lian, Hanrong Ye, David Eigen, Aaron Reite, Boyi Li, Jan Kautz, Song Han, David M. Chan Categories: cs.CV

The Gap

Video understanding with MLLMs has a dirty secret: the models treat every pixel in every frame as equally important. A 1K-frame 4K-resolution video produces millions of visual tokens, and the ViT and LLM chew through all of them regardless of how much of the video is a static background, a slow pan, or a talking head that hasn’t moved in 30 seconds.

Prior work attacked this in a few ways. Token merging (ToMe, FastV) prunes tokens inside the ViT or LLM after they’ve already been computed — you’ve already paid the cost. Frame sampling heuristics (uniform sampling, motion-based selection) operate at the frame level and throw away entire frames, which is too coarse for high-resolution detail. Spatial attention maps from the LLM have been used to guide which regions matter, but that’s a chicken-and-egg problem: you need to run the expensive model to know what to look at.

The gap is upstream: nobody has built a cheap, trainable module that decides before the ViT runs which patches are worth processing, at multiple spatial scales, across time.

Problem: Long 4K video -> millions of tokens -> ViT/LLM too slow
    |
    v
Assumption: Most spatiotemporal patches are redundant;
            a small subset can reconstruct the video
            within a bounded error
    |
    v
Method: AutoGaze -- lightweight autoregressive patch selector
        trained with next-token prediction + RL
        selects multi-scale patches before ViT sees anything
    |
    v
Evidence: 4x-100x token reduction, up to 19x speedup,
          67.0% VideoMME, +10.1% on new HLVid benchmark
    |
    v
Conclusion: Pre-ViT patch selection is a viable and
            superior alternative to post-hoc pruning

The Increment

One sentence: Before this paper, video MLLMs had no cheap way to decide what to look at before looking — AutoGaze gives them that ability, making 1K-frame 4K-resolution video tractable for the first time.

Core Mechanism

AutoGaze sits entirely upstream of the ViT. It takes raw video frames (or low-resolution thumbnails) and autoregressively outputs a sequence of patch selections — each selection is a (frame index, spatial location, scale) tuple. The model is trained with two objectives: a reconstruction loss that ensures the selected patches can reproduce the original video within a user-specified error threshold, and a reinforcement learning signal that rewards selecting fewer patches while staying under that threshold. The result is a Pareto-optimal selector: it finds the smallest set of patches that preserves the information the downstream task needs.

The multi-scale aspect is important. AutoGaze doesn’t just pick pixels — it picks patches at different zoom levels. A wide shot of a room might be captured at coarse scale, while the text on a whiteboard in the corner gets a fine-scale patch. This mirrors how a human would skim a video: broad strokes first, then zoom in on what matters.

The autoregressive structure means each selection conditions on previous selections. If AutoGaze already grabbed the left half of frame 42, it knows not to redundantly grab the overlapping region. This sequential dependency is what allows it to avoid duplication across both space and time.

Raw Video Frames
      |
      v
  [AutoGaze]  <-- lightweight, runs BEFORE ViT
      |
      | autoregressive loop:
      |   step 1: select patch (frame, location, scale)
      |   step 2: condition next selection on prior ones
      |   step 3: stop when reconstruction error < threshold
      |
      v
  Sparse Patch Set  (4x-100x fewer tokens)
      |
      v
  [ViT Encoder]  <-- now processes only selected patches
      |
      v
  [MLLM / LLM]
      |
      v
  Answer / Caption

Think of AutoGaze as a skilled film editor doing a rough cut before the director watches the footage. The director (ViT + LLM) is expensive — their time costs money. So the editor (AutoGaze) watches everything first at 2x speed, marks the important moments and close-ups, and hands the director a tight assembly cut. The editor doesn’t need to understand the film deeply; they just need to know what’s redundant. The director then watches only the assembly cut and makes the final call. The key insight is that the editor’s job is much cheaper than the director’s, so front-loading the selection work saves the whole production.

Each component maps cleanly: the raw footage is the 4K video, the editor’s rough cut is the sparse patch set, the director is the ViT/LLM, the editor’s judgment criteria are the reconstruction loss + RL reward, and the “stop when good enough” rule is the user-specified error threshold.

Key Concepts

• Autoregressive patch selection: Instead of scoring all patches independently and picking the top-K (which ignores overlap and redundancy), AutoGaze picks patches one at a time, where each pick depends on what’s already been picked. It’s like packing a suitcase: you don’t decide what to pack by scoring each item in isolation — you pack the first thing, then ask “given what’s already in here, what’s the next most important thing to add?” This sequential dependency is what lets AutoGaze avoid grabbing the same information twice from nearby frames.

• Reconstruction-bounded selection: The stopping criterion isn’t “pick exactly N patches” — it’s “pick patches until you can reconstruct the video within error threshold T.” This is a much more principled formulation. It means the number of selected patches adapts to the video’s actual complexity: a static talking-head video needs very few patches, while a fast-cut action sequence needs more. The user controls the quality-speed tradeoff by setting T.

• Pre-ViT vs post-ViT pruning: Most prior token reduction work happens inside or after the ViT — you compute all the patch embeddings first, then throw some away. AutoGaze prunes before the ViT ever runs, which means you never pay the compute cost for the discarded patches. This is the difference between deciding not to cook a dish vs cooking it and then throwing it out.

Framework Shift

Before (mainstream approach):        After (this paper):

All frames                           Raw frames
    |                                    |
    v                                    v
[ViT: process ALL patches]          [AutoGaze: cheap selector]
    |                                    |
    v                                    v
Millions of tokens                  Sparse patch set (small)
    |                                    |
    v                                    v
[Token pruning / merging]           [ViT: process ONLY selected]
(post-hoc, cost already paid)           |
    |                                    v
    v                               Hundreds of tokens
[LLM]                                   |
    |                                    v
    v                               [LLM]
Answer                                  |
                                        v
                                    Answer

Cost: O(all patches)                Cost: O(selected patches)
Bottleneck: ViT compute             Bottleneck: AutoGaze (cheap)

From reactive pruning to proactive selection, the core shift is: stop paying to look at things you were going to ignore anyway.

Expert Assessment

Problem choice: This is a real gap, not a manufactured one. The compute wall for long high-resolution video is a genuine blocker for deployment, and the prior work on post-hoc token pruning has an obvious inefficiency that this paper correctly identifies. The timing is right — 4K video understanding is becoming a practical requirement, not a benchmark curiosity. The introduction of HLVid as a new benchmark is a smart move that also happens to be self-serving, but the benchmark itself addresses a real evaluation gap.

Method maturity: The core idea is clean and well-motivated. Training a lightweight selector with reconstruction loss + RL is not trivial engineering, and the autoregressive formulation is a genuine insight over independent scoring. That said, the RL training adds complexity and potential instability that the paper doesn’t fully interrogate. The ablation on “what if you just used uniform sampling at the same token budget?” would be the most important baseline, and it’s not clear from the abstract whether that’s included. The multi-scale aspect is sensible but also adds hyperparameters.

Experimental integrity: The 67.0% on VideoMME is a strong number, and the 19x speedup claim is specific enough to be checkable. The +10.1% on HLVid is compelling but needs scrutiny since the authors also created HLVid — that’s not disqualifying, but it warrants independent replication. The 4x-100x token reduction range is suspiciously wide; the 100x end likely applies only to very redundant videos, and the average case matters more for practical deployment. I’d want to see the distribution of compression ratios, not just the extremes.

Writing quality: The abstract buries the most interesting technical contribution — the autoregressive formulation and the reconstruction-bounded stopping criterion — in favor of leading with speedup numbers. The methods section (not visible here but predictable) probably undersells the RL training details, which is where the real engineering difficulty lives. If the authors rewrote the methods section to be more transparent about training stability and failure modes of the RL component, the paper would be significantly more useful to practitioners trying to reproduce it.

Verdict: weak accept — strong empirical results and a clean problem framing, but the self-referential benchmark and underspecified RL training details leave enough uncertainty that it needs a careful read before trusting the numbers.

Takeaways

The reconstruction-bounded stopping criterion is directly stealable. Any time you’re doing adaptive computation — deciding how much processing to apply to an input — framing the stopping rule as “stop when you can reconstruct the input within error T” is cleaner than “stop after K steps.” It gives you a principled quality-speed knob and makes the compute budget adaptive to input complexity rather than fixed.

The pre-processor pattern is also worth generalizing: if you have an expensive model and a cheap proxy, train the proxy to filter inputs for the expensive model rather than pruning inside the expensive model. This applies beyond video — document understanding, multi-image reasoning, any setting where the input has spatiotemporal redundancy.

The autoregressive selection with sequential conditioning (each pick depends on prior picks) is a useful inductive bias for any selection problem where items have overlapping information content. It’s a better fit than independent scoring whenever the value of an item depends on what else you’ve already selected.