Do Vision Language Models Need to Process Image Tokens?

Paper: 2604.09425 Authors: Sambit Ghosh, R. Venkatesh Babu, Chirag Agarwal Categories: cs.CV

The Gap

VLMs have become the default architecture for multimodal tasks—you throw a visual encoder at an LLM, stack them together, and it works. But nobody’s really asked: do we need all those layers of visual processing? The assumption baked into every VLM is that image tokens need to flow through the entire transformer stack, getting refined at each layer. This is expensive—image tokens are dense and numerous. Yet the field has never systematically measured whether this computational cost is actually buying you anything. Are visual representations meaningfully changing as they go deeper, or do they stabilize early and then just… sit there? This paper fills that gap by measuring what actually happens to visual representations across depth.

Current Paradigm:
Image -> Encoder -> [Deep Visual Processing] -> LLM -> Output
         (assumed necessary)

Question:
Is all that deep processing actually doing work?
Or do visual reps stabilize early?

This Paper:
Measure representation dynamics across layers
-> Find visual reps converge quickly
-> Show deeper layers add little value
-> Propose task-dependent truncation

The Increment

One sentence: Before this paper, we assumed all visual depth was necessary; after, we know visual representations stabilize early and deeper processing is often redundant.

Core Mechanism

The authors measure three properties of visual representations as they flow through VLM layers: entropy (information content), intrinsic dimensionality (how many dimensions actually matter), and trajectory curvature (how much the representation is changing). They find that all three metrics stabilize quickly—entropy plateaus, dimensionality compresses, and the rate of change flattens. Meanwhile, textual representations keep evolving throughout the stack.

Then they do the key experiment: truncate visual processing at different depths and measure task performance. Single-token predictions (like image classification) stay robust even with shallow visual depth. But multi-token generation (like visual question answering) needs sustained visual access. Under deterministic decoding, truncating visual depth perturbs the reasoning trajectory more than the final answer, suggesting image tokens shape how the model reasons, not just what it concludes.

Layer 1    Layer 5    Layer 10   Layer 20
Visual:    ▲▲▲▲▲      ▲▲▲▲▲      ▲▲▲▲▲      ▲▲▲▲▲
           (changing) (stable)   (stable)   (stable)

Text:      ▲▲▲▲▲      ▲▲▲▲▲      ▲▲▲▲▲      ▲▲▲▲▲
           (changing) (changing) (changing) (changing)

Key insight: Visual reps converge early, text keeps evolving

Think of it like a conversation where one person (visual) makes their point early and then mostly repeats themselves, while the other person (text) keeps building on it. You could cut the first person’s microphone after they’ve made their point—the conversation still works, but it changes the flow of how the second person develops their ideas.

Key Concepts

• Representation Convergence: A representation has converged when its statistical properties stop changing meaningfully. Here, visual representations reach a “bounded-complexity regime” where entropy stabilizes and dimensionality compresses. This isn’t about the values becoming identical—it’s about the *structure of the representation becoming fixed. Analogy: a river that stops carving new channels and just flows in the same riverbed.

• Intrinsic Dimensionality: Not all dimensions in a high-dimensional representation are equally important. Intrinsic dimensionality measures how many dimensions you actually need to capture the meaningful variation. A 1024-dimensional vector might only need 50 dimensions to preserve its essential information. When this number stops shrinking across layers, the representation has stopped compressing—it’s done its structural work.

• Task-Dependent Necessity: The paper shows that whether you need deep visual processing depends on what you’re asking the model to do. Single-token tasks (classification) are forgiving; multi-token tasks (generation) are demanding. This suggests visual tokens serve different roles at different stages of decoding, not a uniform role throughout.

Framework Shift

Before (mainstream VLM design):
Image -> Dense Encoder -> [All Layers Process Visuals] -> LLM
         (visual tokens flow through entire stack)
         Assumption: Deeper = Better

After (this paper's insight):
Image -> Dense Encoder -> [Early Layers] -> [Shallow Truncation OK]
                                            (task-dependent)
         Reality: Visual reps stabilize early, deeper layers
                  add marginal value for many tasks

Core shift: From “visual processing is uniformly essential at all depths” to “visual processing is task-dependent and front-loaded.”

Expert Assessment

Problem choice: This is a real gap. The field has been building deeper and deeper VLMs without questioning whether the visual pathway actually needs that depth. It’s a classic case of “we can, so we do” without measuring whether we should. The timing is good—as VLMs get larger and more expensive, efficiency questions matter more.

Method maturity: The approach is straightforward and honest. They measure representation properties (entropy, dimensionality, curvature) that are standard in representation learning, then do ablations by truncating depth. No fancy tricks, just systematic measurement. The simplicity is actually a strength—it’s hard to argue with the findings because there’s no methodological complexity to hide behind.

Experimental integrity: The baselines are fair. They test on multiple VLMs (LLaVA, etc.) and multiple tasks. The key experiment—truncating visual depth and measuring performance—is the right experiment to run. One thing to watch: the paper focuses on deterministic decoding, which is cleaner for analysis but doesn’t capture the full picture of how these models behave in practice with sampling. Still, the findings are consistent enough that this limitation doesn’t undermine the core claims.

Writing quality: The paper is clear and well-structured. The main weakness is that the implications for architecture design could be sharper. The authors show that visual depth can be reduced, but they don’t propose a concrete new architecture or training procedure. It’s more of a “here’s what we found” paper than a “here’s what to do about it” paper. A section on practical design recommendations would strengthen it.

Verdict: Weak accept — The paper makes a solid empirical contribution that challenges a widespread assumption in VLM design. The findings are interesting and the experiments are sound, but the practical impact is limited because the paper doesn’t propose concrete architectural changes. It’s the kind of paper that should influence how people think about VLM design, but it stops short of showing the path forward.

Takeaways

• Measure before optimizing: The paper’s core lesson is to measure what’s actually happening in your model before assuming all components are necessary. Apply this to any deep architecture—measure representation dynamics, find where things stabilize, and question whether you need all that depth.

• Task-dependent efficiency: Not all tasks have the same computational requirements. If you’re building a system that does both classification and generation, you might want conditional computation that uses different depths for different tasks. This paper gives you a framework for identifying where those boundaries are.

• Reasoning trajectory vs. final output: The finding that truncating visual depth perturbs reasoning more than final answers is useful for interpretability work. It suggests that intermediate representations matter for understanding *how a model reasons, even if they don’t change the final answer. This could inform how you design probes or interventions for model analysis.

• Entropy and dimensionality as diagnostic tools: Using entropy, intrinsic dimensionality, and trajectory curvature to characterize representation evolution is a portable toolkit. You can apply these metrics to any multimodal or multi-stage model to identify where processing stabilizes.