Investigating Relational Reasoning in VLMs: How Much Is Seeing and How Much Is Language?

Paper: 2608.23518 Authors: Adhithya Laxman Ravi Shankar Geetha, Aulia Kharis Rakhmasari, Haleema Ramzan, Xander Yap Categories: cs.CV

The Gap

Vision-Language Models score well on visual reasoning benchmarks, and that score is compatible with two very different stories. Either the model represents the visual relations in the image, or it exploits regularities that happen to correlate with the right answer — language cues in the question, priors about what usually goes with what. The benchmark number cannot distinguish them, because both paths produce the same output.

This is not a subtle methodological point. A model answering relational questions from language priors is not doing the task at all, and its accuracy will not degrade where you expect — it will degrade wherever the priors stop matching the images. That makes the failure mode hard to predict from aggregate scores, and it means accuracy improvements can come from better priors rather than better perception.

The gap is therefore one of instrumentation: what is needed is a setting where the visual evidence is artificially simple and fully controlled, so that language cues can be inserted or removed independently of the image, and where the causal weight of the visual evidence can be varied deliberately.

   A HIGH SCORE ADMITS TWO EXPLANATIONS
   VLM answers a relational question correctly
        |
        +-- [A] it REPRESENTS the visual relation
        |
        +-- [B] it exploits a SHORTCUT
                 - language cues in the question
                 - priors about usual co-occurrence
        |
        v
   both paths produce the SAME output -> same score
        |
        v
   [GAP] accuracy cannot separate them, and a
         prior-driven model fails wherever the priors
         stop matching the images
        |
        v
   NEEDED: controlled stimuli + queries that vary
           LANGUAGE CUES and VISUAL EVIDENCE independently

The Increment

One sentence: Before this paper, a VLM’s relational-reasoning accuracy could not be attributed to seeing or to language; after it, a synthetic geometric dataset with cue-controlled queries and a causal manipulation of visual evidence shows current VLMs doing both at once, with the shortcut component rooted primarily in language.

Core Mechanism

Three design choices carry the investigation.

A synthetic dataset of simple geometric shapes. Deliberately minimal stimuli. Simple shapes remove the confound that would come from rich natural images, where a model might succeed for reasons having nothing to do with relational structure. If a model can represent relations at all, simple shapes are where it should be easiest to demonstrate; conversely, failure here is informative because it cannot be blamed on visual complexity.

Queries crafted to test language cues precisely. The questions are built so that the linguistic form and the visual content can be pulled apart. This is the crux of the design: you cannot detect a model relying on language cues unless you can vary those cues while holding the image fixed, and vice versa.

A modified dataset that tests causal reliance on visual evidence. The visual evidence is manipulated so that it becomes decisive for the answer. If the model is genuinely reading the image, making the image causally determine the answer should change its behaviour in the corresponding way. If it is answering from priors, changing the image will matter much less than it should.

The analysis method adds a fourth element: the model’s internals are probed across depths to decode how visual information is encoded. Rather than only observing input-output behaviour, the study asks where in the network the visual content becomes decodable, which gives a second line of evidence alongside the behavioural manipulation.

The reported conclusion is a mixture, and the paper states it as such: current VLMs combine genuine visual reasoning with shortcut strategies primarily rooted in language cues. That is the honest characterisation, and it is more useful than a verdict either way. It means both mechanisms are active, that the language component is the dominant shortcut, and that any evaluation reporting a single accuracy figure is reporting a blend of two different competencies without saying in what proportion.

   DESIGN

   [1] SYNTHETIC STIMULI: simple geometric shapes
        -> removes visual-complexity confounds
        -> if relations are representable, easiest here
                    |
   [2] CUE-CONTROLLED QUERIES
        -> vary LANGUAGE form independently of the image
        -> vary IMAGE independently of language form
                    |
   [3] CAUSAL VISUAL MANIPULATION
        -> make visual evidence DECISIVE for the answer
        -> prior-driven models change too little
                    |
   [4] DEPTH-WISE INTERNAL PROBING
        -> where does visual content become decodable?
        -> second line of evidence beyond behaviour
                    |
                    v
   RESULT: genuine visual reasoning AND language-rooted
           shortcuts, both active at once

Think of it as testing whether someone can actually read a map, by handing them maps with the legend deliberately altered. If you only ever hand them real maps of real places, a high pass rate is ambiguous: they may be reading the map, or they may recognise the city and answer from memory. So you fabricate maps of invented places — simple, controlled, no prior to fall back on — and then you start changing what the legend means. Someone reading the map follows the change immediately. Someone answering from memory keeps giving yesterday’s answers, and now you can see exactly how much of their performance was memory all along.

Key Concepts

• Shortcut strategies rooted in language cues: answering from the linguistic form of the question rather than from the image. The paper identifies language as the dominant shortcut, which is actionable — it says which cue to control for first when building an evaluation.
• Causal reliance on visual evidence: the test of whether making the image decisive for the answer actually changes behaviour. It converts “does the model use the image” from an interpretive question into a manipulation with an expected signature.
• Depth-wise decoding of visual information: probing where visual content becomes recoverable at successive layers. It provides internal evidence that complements the behavioural results, and it is more robust than output inspection alone, which a model can satisfy for the wrong reasons.

Framework Shift

Before (accuracy on relational benchmarks):
  natural images + relational questions
  -> one accuracy number
  -> ambiguous between representation and shortcut
  -> failure mode unpredicted: models break where
     priors stop matching images

After (controlled stimuli + cue and causal manipulation):
  synthetic shapes
  + queries that separate language from image
  + manipulation making vision decisive
  + depth-wise internal probing
  -> "both mechanisms active; language is the
      dominant shortcut"
  -> evaluation can report the blend, not just the total

From reporting how often a model is right about relations, to separating how much of that correctness is seeing and how much is language, the core shift is that a single accuracy figure conceals two competencies and should not be reported alone.

Expert Assessment

Problem choice: Good and well-targeted. The shortcut question is the central open problem for VLMs, and the choice of synthetic stimuli is the right response: natural images make attribution impossible, and a controlled setting is what buys the ability to manipulate the causal weight of the evidence.

Method maturity: The combination of behavioural manipulation and depth-wise probing is the sound part — the two lines of evidence can corroborate each other, and probing where visual content becomes decodable is a genuine independent check rather than a restatement of accuracy. The investigation is, however, confined to a single model (Qwen3-VL-4B). That is a real limitation for a claim about “current VLMs”: the headline mixture finding may well be family-specific, and a conclusion about the class requires at least a second architecture.

Experimental integrity: The causal manipulation of visual evidence is the strongest element, because it has a predicted signature that a shortcut-driven model cannot fake — if the image becomes decisive and behaviour barely moves, that is evidence. Reporting a mixed conclusion rather than a clean “VLMs use shortcuts” result is also to the authors’ credit, since the cleaner claim would have been easier to publish.

Writing quality: The framing that high accuracy admits two explanations, and that these cannot be distinguished by output alone, is stated clearly and early, which is exactly right. The paper would be considerably more useful with a quantified split — an estimate of how much of the measured performance the language component accounts for. The qualitative “both, mainly language” is the right shape of answer but leaves the magnitude open.

Verdict: accept — a well-designed instrument and an honest mixed finding, with the caveats that it rests on one model and reports the direction of the effect rather than its size.

Takeaways

• Never report a single relational-reasoning accuracy without a cue-controlled comparison. The number blends representation and shortcut, and the blend is the thing you need to know.
• Use synthetic, minimal stimuli when you want to attribute capability. Rich natural images make success ambiguous, because priors can carry a model without any relational representation.
• Make the visual evidence causally decisive. A model that reads the image responds when the image starts determining the answer; a prior-driven model barely moves, and that gap is your measurement.
• Test more than one model family before generalising. The dominant-shortcut finding here comes from a single architecture and may not be a property of VLMs as a class.