Retrieval Heads Meet Vision: 1.7% of Attention Heads Do the Grounding

Paper: 2608.27417 Authors: Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung Categories: cs.CV

The Gap

A VLM can locate the image region a text prompt refers to, and route the corresponding visual evidence into its output. This works, and the internal mechanism behind it was not understood. That is the gap: a capability that is relied on — grounding is the basis of every visual question about a specific object — with no account of which components implement it.

The obvious analogy is retrieval heads in language models, a small subset of attention heads causally responsible for copying information from context to output. If VLMs have an analogous mechanism for visual retrieval, it would explain grounding in terms the field already understands, and it would make the capability inspectable. But the analogy is a hypothesis, and it could fail: vision introduces an encoder, a projector and a different input structure, any of which could spread the function diffusely.

   THE OBSERVED CAPABILITY, THE MISSING MECHANISM

   VLM: locate the region a prompt refers to
        route the corresponding visual evidence to output
        -> works, and grounding underpins every
           object-specific visual question
        |
        v
   [GAP] no account of which components implement it
        |
        v
   HYPOTHESIS: retrieval heads, as established for TEXT
     - a small subset of attention heads
     - causally responsible for copying context into output
        |
        v
   the analogy COULD FAIL for vision:
     an encoder, a projector, a different input structure
     -> any of which might spread the function diffusely

The Increment

One sentence: Before this paper, VLM grounding had no mechanistic account; after it, visual retrieval heads are identified — about 1.7-2.6% of attention heads whose masking costs up to 80 percentage points of grounding accuracy — and they show properties the text analogue does not.

Core Mechanism

The first contribution is how to find them, and it is a methodological result rather than a lucky guess. The authors recast existing head-scoring methods under a unified design space over three axes — query tokens, key aggregation, and cross-sample aggregation — and then determined which point in that space most reliably identifies causal heads. The answer: scoring attention from output prediction tokens, with a sum over the ground-truth referent region. Two choices are doing the work there. Taking attention from prediction tokens focuses on the heads that contribute to what is actually output, not to some intermediate representation. Aggregating over the ground-truth referent region grounds the score in the correct answer rather than in whatever the model happens to attend to.

Then the causal evidence, which is what makes the label “retrieval head” earned rather than assigned. Across eleven VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. Twenty heads out of however many — a small intervention with a catastrophic and specific effect — is the causal-sparse signature that justifies calling them retrieval heads.

And then the part that goes beyond replicating the text analogy. Three properties are reported that were not previously established:

They generalise across visual reference tasks. The heads were discovered through bounding-box prediction, yet they remain causal on attribute, spatial, counting and visual-math benchmarks. So what was found is not a bounding-box module. It is a mechanism for routing visual evidence that a box-prediction task happens to expose, and it is used by tasks that never produce a box.

They are functionally specific. Masking them preserves output format while corrupting localization. That is a strong dissociation, because it rules out a boring explanation — if the heads were simply needed for producing well-formed output, corrupting them would break the format too. The function is specifically localization.

They are architecturally shared. They transfer causally across VLMs that share an LLM backbone but differ in vision encoder, projector and instruction tuning. This is the most surprising result. One would expect visual grounding to live in whatever adapts vision to language — the projector. Instead, the mechanism appears to reside in the language backbone side and survives changes to the vision stack, which suggests the grounding function is implemented in a place that is more stable across model families than the vision components are.

   FINDING THEM (a design-space result)

   recast head-scoring methods under three axes:
     query tokens | key aggregation | cross-sample aggregation
        |
        v
   the point that works:
     score attention FROM OUTPUT PREDICTION TOKENS,
     summed over the GROUND-TRUTH REFERENT REGION
        |
        v
   causal evidence (11 VLMs, 5 referring-expression benchmarks)
     mask top 20 VRHs   -> grounding accuracy -80 points
     mask 20 RANDOM     -> little effect
     -> 1.7-2.6% of heads, causally sparse

   BEYOND THE TEXT ANALOGUE
     GENERALISE: discovered via bounding-box prediction, yet
       remain causal on attribute / spatial / counting /
       visual-math benchmarks
     FUNCTIONALLY SPECIFIC: output FORMAT preserved while
       LOCALISATION is corrupted
     ARCHITECTURALLY SHARED: transfer causally across VLMs that
       share an LLM backbone but differ in vision encoder,
       projector and instruction tuning

Think of it as tracing which few switches in a control room actually steer the camera. The room has hundreds of switches; flipping them at random changes nothing you can measure. But a small set turns out to be the ones that move the camera, and flipping those sends the view somewhere else entirely. Two details from the study make this more than a metaphor. The switches were found by watching which ones light up when the camera ends up correctly pointed — not which ones light up during the operation generally. And when you turn them off, the camera’s power indicator still works and its menus still render: only the aim is wrong. That dissociation is what tells you the switches are for aiming rather than for the machine running at all. The third finding is that this switch panel is the same across two control rooms wired quite differently on the input side — the aiming function lives on the side of the system that has less to do with how the pictures arrive.

Key Concepts

• Visual Retrieval Heads (VRHs): about 1.7-2.6% of attention heads that are causally responsible for grounding textual descriptions to image regions. The percentage is what makes them a mechanism rather than a general property of attention.
• Discovering causal heads through a design space: recasting head-scoring across query tokens, key aggregation and cross-sample aggregation, and identifying the combination that finds causal heads. The specific answer — prediction tokens, summed over the ground-truth region — matters less than the fact that it was searched for rather than assumed.
• The causal-sparse-universal triad: sparse (a small subset), causal (masking them has a large specific effect), universal (consistent across models and benchmarks). It is the pattern established for text retrieval heads, and replicating it for vision is what licenses the analogy.
• Functional specificity as evidence: preserved output format with corrupted localization. Without this, “these heads matter” would be consistent with a much less interesting explanation — that they are needed for generating any valid output at all.

Framework Shift

Before (capability observed, mechanism unknown):
  VLM grounds a text prompt to an image region
  -> understood behaviourally, not mechanically
  -> "grounding" is a black box inside the model

After (mechanism located and characterised):
  1.7-2.6% of heads, found via a searched design space
  masking top 20 -> -80 points; random 20 -> nothing
  generalise beyond bounding boxes
  specific: format intact, localisation broken
  shared across VLM families with different vision stacks

From treating visual grounding as a black box that behaviourally works, to locating it in a small, causally verified set of heads with a defined functional profile, the core shift is that the capability can now be inspected, pruned and reasoned about at the level of components.

Expert Assessment

Problem choice: Very good, and it picks the right analogy to test rather than to assume. Retrieval heads are one of the better-established mechanisms in language model interpretability, so asking whether vision has an analogue is well posed — and it is a question that could have come back negative.

Method maturity: The design-space formulation is the methodological strength, because it turns “which scoring method finds the heads” from a choice of convention into a search whose answer is reported. The evidence standard is also right: discovering heads by one task and then checking causal effect on tasks that share no output format is a genuine generalisation test, and the functional-specificity dissociation is a properly conceived control against a deflationary explanation.

Experimental integrity: The numbers are structured to be falsifiable. An 80-point drop from masking 20 heads, against near-nothing from masking 20 random heads, is a large effect with a matched control. Eleven VLMs and five benchmarks is a breadth that makes single-model artefacts unlikely, and the cross-architecture transfer test is the strongest single result because it comes with a prediction one would not have made. The honest limitation is that all of this is behavioural plus causal masking; there is no account of how the heads route information, so the label describes a function located, not a mechanism explained.

Writing quality: The paper is organised around the triad and then the three novel properties, which is a clear structure and puts the new claims where they can be assessed. Because the finding is about a small subset of heads, a visualisation of where VRHs occur — depth, layer type, head index — would be disproportionately useful, letting a reader see the structure rather than only its consequences.

Verdict: strong accept — a well-controlled identification of a sparse causal mechanism for visual grounding, with three properties that go beyond replicating the text analogue and one cross-architecture result that is genuinely unexpected.

Takeaways

• Search the scoring design space before adopting a convention. Which statistic finds causal heads is an empirical question with a reportable answer, not a matter of habit.
• Test causal heads on tasks that share no output format with the discovery task. That is what separates a general mechanism from a module for one benchmark.
• Use functional specificity as a control. Preserved format with broken localisation rules out the deflationary explanation; without it, “these heads matter” is weak.
• Look for mechanisms on the stable side of the architecture. Grounding here transferred with the LLM backbone rather than with the vision stack, which is not where one would have expected to find it.