Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models

Paper: 2606.28273
Authors: Niclas Lietzow, Danielle Bitterman, Carsten Eickhoff, William Rudman, Michal Golovanevsky
Categories: cs.CL

The Gap

Existing work on perception-knowledge conflict in VLMs (e.g., visual question answering with absurd images, biased prompts) has described the behavioral outcome: sometimes the model trusts the image, sometimes it falls back on memorized facts. But no one had opened the black box at the component level to ask which internal parts cause the conflict resolution. Prior studies relied on probing classifiers, attention visualizations, or global feature attribution – correlational tools that cannot distinguish cause from effect. The field knew that conflict happens, not how it happens mechanistically.

This paper bridges that gap by combining causal intervention (activation patching at three granularities) with ablation of specific components. They don’t just observe – they break and fix to prove causality.

[Perception-Knowledge Conflict in VLMs]
                     |
                     v
[Existing work: behavioral description only]
                     |
                     v
[Gap: no component-level causal account]
                     |
                     v
[Method: activation patching + ablation]
        |                        |
        v                        v
[Find sparse causal circuit]  [Asymmetric causal structure]
        |                        |
        +------------------------+
                     |
                     v
[Conclusion: visual default, prior override via attention heads]

The Increment

One sentence: Before this paper, we knew VLMs could be fooled by conflicting knowledge; after this paper, we know exactly which attention heads (2.5–4.8%) in the second half of the network are causally responsible for pulling answers from memory when vision disagrees – and that visual grounding is the default, not the override.

Core Mechanism

The authors apply activation patching (also called interchange interventions) across three levels of granularity: the residual stream, individual attention heads, and MLP sublayers. Activation patching works by recording the model’s internal state on a “source” input, then replacing a specific layer’s activations during a forward pass on a “target” input. If the output changes toward the source’s output, that component is causally implicated.

They then perform ablation – setting the activation of a component to zero (or mean) – to test whether removing it flips the model’s behavior. The key trick: they compare two kinds of prompts. A vision-default prompt asks the model to describe what it sees (e.g., “What color is the strawberry?” when the image shows a blue strawberry). A prior-knowledge prompt asks for the fact (e.g., “What color is a strawberry?” while also showing the blue image). The model must reconcile the conflict. By ablating the identified heads, they find that 68–96% of prior-knowledge answers switch to visual answers, but only 0.8–7.5% of visual answers switch to knowledge answers. That’s the asymmetry.

[Input image + prompt] --> [Residual Stream]
                              |
                    +-----------------+
                    |                 |
                [Patch point]     [Ablation mask]
                    |                 |
                    v                 v
            [Attention Heads]    [MLP Sublayers]
              (22-24 layers)     (2 layers per block)
                    |                 |
                    +--------+--------+
                             |
                  [Output logits]
                   /          \
          [visual answer]   [knowledge answer]

Structural Metaphor: The Judge and the Specialist Witnesses

Think of the VLM as a courtroom trial. The residual stream is the shared whiteboard where all evidence is recorded. The attention heads are witnesses: they read from the whiteboard and write new facts. Most witnesses are default witnesses – they honestly report what they see in the image (the visual evidence). A small group of specialist witnesses (2.5–4.8% of the total) have memorized world knowledge. When the prompt asks for a fact (e.g., “What color is a strawberry?”), the judge (the decoder) calls these specialists. They override the visual evidence on the whiteboard by writing “red” even though the image shows blue. The authors’ activation patching is like temporarily switching a specialist’s testimony with a different witness’s – if the verdict changes, that specialist was causal. Ablation is like removing the specialist entirely – the jury (the model) can only rely on the default witnesses and thus trusts the image. The asymmetry: removing a specialist flips the verdict from knowledge-based to image-based almost always, but adding a specialist (or removing a default witness) rarely flips from image-based to knowledge-based. That’s because the court defaults to visual evidence unless a specialist is explicitly present.

Key Concepts

• Activation Patching: A causal intervention technique. You run the model on input A (e.g., blue strawberry + fact prompt) and record the activation of a specific layer (say, head 12). Then you run the model on input B (blue strawberry + vision prompt) but at the same layer you *replace the activation with the recorded one from input A. If the output changes to match input A’s answer, that layer is causally necessary for the knowledge answer. This is like performing a brain transplant on one neuron at a time.

• Asymmetric Causal Structure: The finding that the identified attention heads are necessary for knowledge-based answers but not for vision-based answers. This means visual grounding is the model’s default mode – the residual stream naturally encodes visual features. The small set of “prior” heads act as an override that can write conflicting knowledge. Symmetry would mean both directions depend on distinct circuits; asymmetry implies a hierarchical relation.

• Causal Circuit: A set of components (attention heads, MLP neurons) that form a minimal, necessary pathway for a behavior. The authors show that the prior-grounded answers depend on a sparse circuit: only a few heads in the later layers. This contrasts with dense, distributed circuits often assumed for complex reasoning.

Framework Shift

The old view: conflict resolution in VLMs is a black-box behavior described by overall accuracy or saliency maps. Researchers argued about whether models “see” or “know” without mechanistic evidence.

The new view: conflict resolution is an asymmetric causal process with a sparse circuit. Visual grounding is the default; prior grounding is a specific override that can be disabled by ablating a few heads.

Before (mainstream approach):
[Probing/Viz] -> [Correlation] -> [Behavioral description]
                (no causality)

After (this paper):
[Patching] -> [Causal circuit] -> [Asymmetric mechanism]
              (sparse, localized)

One sentence: From correlational black-box to causal sparse-circuit understanding of perception-knowledge conflict.

Expert Assessment

Problem choice: Real gap. The behavioral layer had been well-explored (eg, GQA, VQA v2, etc), but the mechanistic cause was untouched. This sits at the intersection of interpretability and multimodal safety – important for debugging hallucinations.

Method maturity: Clever use of existing tools (activation patching, ablation) applied in a multi-granularity pipeline. Not a new technique, but the combination and the asymmetry finding are novel. Simpler approaches (e.g., attention pruning alone) would have missed the causal asymmetry – so this level of rigor is justified.

Experimental integrity: Fair baselines: they compare vision-default vs prior-knowledge prompts, and run patching across multiple models (LLaVA, InstructBLIP, etc). The asymmetric result is striking (68–96% vs 0.8–7.5%). Red flag: the paper does not report confidence intervals or variance across seeds; sample sizes are not clear. Also, ablation may disrupt other functions – the small flip in visual predictions (0.8–7.5%) could be due to collateral damage, not evidence of the circuit. This requires stronger control.

Writing quality: Well-organized, but the methods section is dense – it tries to cover three granularities in one go. A diagram with clearer notation would help. The “metaphor” section is missing; the paper itself uses no analogy, which makes the mechanism hard to internalize. Adding a few sentences of intuition would elevate it.

Verdict: weak accept – fills a real gap with rigorous causal methodology, but the contribution is incremental (identifying known structure in new context) and the experimental reporting could be tighter.

Takeaways

1. Activation patching toolkit: Practitioners can replicate this pipeline to find causal circuits for any multimodal behavior (eg, bias, object hallucination). The three-granularity approach (residual → head → MLP) is template-worthy.

2. Visual default assumption: When building VLMs for safety, treat visual grounding as the baseline and knowledge retrieval as a fragile, localized override. Fine-tuning to strengthen the visual default may be more robust than trying to police knowledge heads.

3. Sparse intervention strategy: If you need to suppress unwanted knowledge responses (eg, stereotypes), you can ablate a tiny fraction of attention heads rather than heavy retraining. The paper’s head ratios (2.5–4.8%) suggest targeted pruning is feasible.

4. Asymmetry as diagnostic: The asymmetry ratio (flip rate of prior vs vision) can be used as a metric to measure how “grounded” a model is – lower asymmetry means knowledge heads are less essential, which might be desirable for vision-centric tasks.