Gaze Heads: How VLMs Look at What They Describe

Paper: 2606.14703 Authors: Rohit Gandikota, David Bau Categories: cs.CV, cs.CL, cs.LG

The Gap

Existing mechanistic interpretability work in VLMs has focused on locating object-specific neurons or probing attention heads for factual recall.
But no prior method explains how the model aligns its language generation with a spatial region of the image in real time.
Attempts to steer VLM outputs (via prompt engineering, finetuning, or latent editing) treat the model as a black box and require retraining or large datasets.
This paper asks: is there a compact, causally responsible set of attention heads that controls which part of the image is being described?
They use comic strips – where narrative order is spatially laid out – as a controlled testbed, and find that a few hundred heads (fewer than 9% of all heads) act as “gaze heads” that track the described region.

[Problem]  
VLM description mechanism is opaque  
   |  
   v  
[Assumption]  
A small set of heads correlates attention with described region  
   |  
   v  
[Method]  
Correlation of attention weights to current-description tokens  
   |  
   v  
[Evidence]  
Redirecting those heads forces description of a different region  
   |  
   v  
[Conclusion]  
Gaze heads are a causal inference-time control lever

The Increment

One sentence: Before this paper, steering a VLM to describe a specific image region required prompt engineering or finetuning; after this paper, a single attention-mask intervention on <9% of heads achieves 83.1% accuracy with zero retraining.

Core Mechanism

The method has three steps: identification, intervention, and control.

First, identification: the authors use a dataset of comic strips where each panel’s description naturally refers to a specific spatial location. For each generated token during inference, they compute the correlation between an attention head’s weights over image tokens and a binary mask of the “correct” panel being described. Heads with consistently high correlation scores across many images are selected as “gaze heads” (top 100 by default).

Second, intervention: at inference time, the gaze heads’ attention scores are modified by adding a large constant to the attention logits for tokens belonging to the desired image region (i.e., a mask). This forces the head to attend predominantly to that region.

Third, control: the same intervention works continuously – if the mask is changed mid-generation, the model finishes describing the current panel and moves to the new one within a few tokens.

   Input Image          Text prompt  
       |                     |  
    (ViT encoder)     (Language model backbone)  
       |                     |  
   Image tokens         +-------...-------+
       |                |   all heads     |
       +------concat----| (100s of heads) |
                        |                 |
                        +-------+---------+
                                |
                     [Correlation scoring]
                                |
                   Top-100 gaze heads identified
                                |
              At inference: apply mask to logits
              of these heads to redirect attention
                                |
                        Output description

Structural metaphor: The Spotlight Director

Imagine a theater stage (the image) with multiple actors (image regions).
The VLM is a director who calls out lines (generates tokens).
Most stagehands (regular attention heads) scatter light equally across the whole stage.
But a few key stagehands – the “gaze heads” – actively follow the director’s gaze: they always point their spotlight at the actor currently speaking.

If you grab those few spotlights and physically turn them toward a different actor, the director will immediately start describing that actor’s scene.
Grabbing random spotlights does nothing – they don’t track the narrative.
And grabbing all spotlights confuses the director because the entire stage is artificially lit.

The mask is your hand on the spotlight handle: a simple, physical lever to change what the model “looks at.”

Key Concepts

• Gaze head: An attention head whose attention over image tokens correlates with the spatial region being described. Identified by computing, over many tokens, the pointwise correlation between the head’s attention distribution and a binary mask of the correct region. The top 100 heads (out of ~1100 in a 7B model) are kept.

• Attention redirection: During forward pass of the language model, before softmax, add a large positive constant (e.g., 100) to the attention logits of image tokens that belong to the *target region. After softmax, attention is forced to concentrate on that region. This is a zero-training, inference-time operation.

• Zero-shot control: No finetuning, no prompt tricks. The intervention directly modifies the internal computation of a known causal component. This contrasts with latent space steering (e.g., reading/writing control vectors) which requires collecting contrastive pairs.

Framework Shift

Before (mainstream approach):  
Input image + "Describe the left panel" -> [VLM black box] -> output  
All steering is done via input manipulation (prompt engineering) or output selection.

After (this paper):  
Input image + "Describe it" -> [VLM with identified gaze heads] -> output  
                                |  
                           Mask applied to top-100 heads  
                           (internal causal intervention)

The core shift is: from external prompt engineering (treating the model as a black box) to internal causal control (operating on a discovered mechanism).

Expert Assessment

Problem choice: Real and timely. As VLMs are deployed in production, the ability to steer which part of an image they describe without retraining has direct practical value. The gap between “we know attention heads exist” and “we can use them as control handles” was clearly articulated.

Method maturity: Clever but simple. The correlation-based identification is straightforward, yet the paper’s key contribution is demonstrating that these heads are *causally sufficient for steering. A more rigorous approach might use automated causal discovery, but the current method’s simplicity is a virtue.

Experimental integrity: Baselines are fair (random heads, all heads, no intervention). The 83.1% accuracy on comics is convincing. However, the natural image (COCO) experiment reports only a single example, not a quantitative comparison. The paper would be stronger with systematic evaluation on diverse image types. No red flags, but the lack of statistical tests is a minor weakness.

Writing quality: Clear, well-structured, with good figures. The methods section could be more explicit about the exact correlation formula. The ablation study (different head counts, different model sizes) is well done but buried in the appendix. Moving it to the main body would strengthen the paper.

Verdict: weak accept – solid mechanistic discovery with a practical control method, but limited quantitative evaluation on natural images and a narrow task scope (description only).

Takeaways

• Actionable technique: You can find gaze heads in your own VLM by computing attention-region correlation on any dataset with spatial annotations (e.g., caption-region pairs). The correlation metric is cheap (a few forward passes).

• Inference-time steering: The attention-mask intervention is a general method for any task where you want to force a VLM to focus on a specific spatial region – e.g., visual question answering about a particular object without mentioning it in the prompt.

• Mechanistic interpretability as engineering tool: This paper demonstrates that understanding internal mechanisms (gaze heads) directly yields a control lever. The same approach could be applied to other “tracking” behaviors (e.g., tracking a moving object in video VLMs).