The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

Paper: 2608.06270 Authors: Zhiheng Wang, Bo Peng, Lai Wei, Chaochao Lu Categories: cs.AI

The Gap

The “thinking-with-images” paradigm gives a multimodal LLM active visual operations — most commonly crop-and-zoom. The model looks at an image, decides a region matters, crops it, zooms in, and continues reasoning with the new view in context. On fine-grained perception benchmarks this reliably beats direct inference, and the field has largely accepted the obvious explanation: the model is looking closer, so it sees better.

The evidence against that explanation has been sitting in plain sight. Gains are often marginal or negative relative to the token cost. Models repeatedly crop irrelevant regions. Worst of all, models sometimes fail on questions that direct inference answers correctly — which is hard to square with “extra information helps.”

The unexamined step is a causal one. Accuracy going up when tool-use is enabled tells you the policy is better. It does not tell you that the returned pixels are what made the answer right. A model could just as easily be using the act of calling the tool as a behavioral cue — a signal to slow down, to switch into a more careful decoding mode, to condition on “this is a hard question” — with the returned crop contributing nothing. Aggregate accuracy cannot distinguish an observation-mediated path from an action-induced shortcut, because both raise the same number.

Observed: tool-use accuracy > direct-inference accuracy
         |
         v
Assumed: the returned crop causes the correct answer
         |
         v
[This paper] Ask instead: is there a causal path
             from observation to answer at all?
         |
         v
Formalize tool-use as a causal graph:
  observation-mediated path  vs  action-induced shortcut
         |
         v
Intervene at 3 levels: policy / trajectory / step
         |
         v
Evidence: gain concentrates in a "Calibrated" minority;
          many rollouts show zero observation effect
         |
         v
Conclusion: the illusion of visual tool-use

The Increment

One sentence: Before this paper, visual tool-use was evaluated by whether accuracy went up; after it, the field has a causal estimand that asks whether the returned image did any work at all — and the answer, across six models, is usually no.

Core Mechanism

The paper’s core move is to stop treating tool-use as a black box that either helps or does not, and instead write down the causal graph. In a rollout, the model emits an action (a crop request), the environment returns an observation (the cropped image), and the model emits an answer. Accuracy conflates two distinct routes from action to answer: the observation-mediated path, where the returned pixels carry information that changes the answer, and the action-induced shortcut, where merely having called the tool changes the model’s behavior regardless of what came back. These are separable only under intervention.

So the audit intervenes at three nested granularities. At the policy level, it compares tool-use against direct inference — this is the conventional comparison, and it recovers the conventional result. At the trajectory level, it corrupts all observations during a rollout: the model still calls tools, still gets responses, but the responses carry no valid visual content. If accuracy survives corruption, the gain was never coming from the pixels. At the step level, it counterfactually replaces one individual observation while holding the prefix fixed, which isolates the marginal contribution of that single returned view. This last one gives the paper’s estimand, Visual Evidence Gain — the causal contribution of one observation to the final answer.

Running this across six representative models and five fine-grained perception benchmarks turns up policy miscalibration in two distinct flavors. In Calling Without Looking, the model requests a crop and the returned observation has no causal effect on the answer — the tool call is theater. In Looking Without Planning, the observations genuinely are informative, but the model’s schedule for when to call is incoherent, so the information arrives at the wrong point in the reasoning. The trajectory-level diagnostic then decomposes the headline accuracy gain and shows it is concentrated in a Calibrated minority of rollouts. The aggregate number is real; the causal story behind it applies to only a slice of cases.

                  Rollout under audit

   [Question + Image]
          |
          v
   +-------------+
   |   Action    |  model requests crop-and-zoom
   +-------------+
          |
     +----+-----------------------------+
     |                                  |
     v                                  v
 +---------------+            action-induced shortcut
 | Observation   |            (calling changes behavior
 | (cropped view)|             regardless of content)
 +---------------+                      |
     |                                  |
     |  observation-mediated path       |
     +----+-----------------------------+
          |
          v
     +---------+
     | Answer  |
     +---------+

   Interventions that separate the two paths:

   policy      : tool-use  vs  direct inference     (coarse)
   trajectory  : corrupt ALL observations           (medium)
   step        : swap ONE observation, fix prefix   (fine)
                 -> Visual Evidence Gain

Here is a structural metaphor that carries the whole design. Imagine a student who does noticeably better on exams when allowed to use a pocket calculator. The obvious conclusion is that the arithmetic help is what raises the score. But there are two ways a calculator can raise a score. One: the student punches in the numbers and reads off an answer they could not have computed — the calculator-mediated path. Two: reaching for the calculator is a ritual that makes the student slow down, re-read the question, and take it seriously — the ritual shortcut. Both show up as a higher score.

To tell them apart you cannot just compare “with calculator” to “without calculator” — that is the policy-level test, and it is satisfied by both stories. You have to hand the student a calculator with a scrambled display and see if the score holds. If it does, you were paying for the ritual, not the arithmetic. That is the trajectory-level intervention. And if you want to know how much a particular calculation mattered, you swap that one display reading for a wrong number, keeping everything before it identical, and watch whether the final answer moves. That is the step-level intervention, and the size of the move is Visual Evidence Gain.

What the audit finds is a classroom where most students are performing the ritual, a few are genuinely computing, and the class average looks like everyone is computing.

Key Concepts

• Visual Evidence Gain: The causal contribution of one returned observation to the final answer, measured by counterfactually replacing that observation under a fixed prefix. The “fixed prefix” part is what makes it causal rather than correlational. If you simply compared rollouts that happened to crop region A against rollouts that cropped region B, you would be comparing models that already differed in their reasoning before the crop — the crop choice is downstream of the model’s state. Freezing everything up to the observation and swapping only the observation removes that confound. A concrete reading: Visual Evidence Gain near zero means that if you had handed the model a completely different crop at that step, it would have produced the same answer.

• Calling Without Looking: A failure mode where the tool call happens, an observation comes back, and the observation causally does nothing. This is the sharpest finding in the paper because it is invisible to every conventional metric — the trace looks like grounded visual reasoning, complete with plausible bounding boxes and a confident reference to “the zoomed region.” The behavior is well-formed; the causal link is absent. It is the multimodal analogue of a chain-of-thought that does not actually determine the answer it precedes.

• Looking Without Planning: The complementary failure. Here the returned observations are informative — there is real signal in the pixels — but the policy’s decision about when to call is incoherent, so evidence arrives detached from the reasoning step that needed it. This distinction matters practically: the two modes have opposite fixes. Calling Without Looking is a grounding problem (the model is not conditioning on what it retrieved). Looking Without Planning is a scheduling problem (the model is retrieving at the wrong time). Lumping them together as “tool-use is unreliable” would send you after the wrong repair.

Framework Shift

Before (mainstream evaluation):      After (this paper):

 enable tool-use                      write the causal graph
      |                                     |
      v                                     v
 measure accuracy                     intervene at 3 levels
      |                                     |
      v                                     v
 accuracy up?                         does observation ---> answer
      |                               edge carry any effect?
      v                                     |
 "the model sees better"                    v
                                      decompose the aggregate:
 (single number; cannot                 Calibrated minority
  separate observation                  Calling Without Looking
  from action)                          Looking Without Planning

From measuring whether tool-use raises the score to measuring whether the returned evidence causes the answer, the core shift is treating a benchmark gain as something that still needs a causal explanation rather than as the explanation itself.

Expert Assessment

Problem choice: This is a real gap, and a well-timed one. “Thinking with images” has moved fast enough that the evaluation methodology has not caught up, and the anomalies the paper opens with — negative gains at high token cost, repeated irrelevant crops, regressions on questions direct inference gets right — were already public knowledge that the field was absorbing as noise rather than as signal. Reframing them as symptoms of a missing causal link is the right diagnosis. The paper also sits in a productive lineage: the same move has been made for chain-of-thought faithfulness, and importing it into the multimodal setting is a natural and useful transfer rather than a novel invention.

Method maturity: The three-level intervention design is clean and the nesting is well-motivated — each level answers a question the level above cannot. Visual Evidence Gain is the real contribution, because a step-level estimand under a fixed prefix is the thing that lets you attribute credit to individual observations rather than to the policy as a whole. The obvious caution is that observation corruption is a blunt instrument. Corrupting an image and finding that accuracy holds is consistent with “the pixels did not matter,” but it is also consistent with the model detecting corruption and falling back to a prior it would have used anyway. The paper’s step-level counterfactual is the better tool precisely because a swapped-but-valid observation is harder to detect than a destroyed one, and I would want the headline claims resting on the step-level result more than on the trajectory-level one.

Experimental integrity: Six models and five benchmarks is a genuinely adequate spread for an audit paper — findings of this kind live or die on whether they generalize past one model family, and this is wide enough to take seriously. Releasing the code (CauAudit) matters more than usual here, since the entire contribution is a measurement procedure and an unreproducible measurement procedure is worth little. The claim I would most want to see stress-tested is the “Calibrated minority” decomposition: the practical weight of the paper depends on how small that minority is and whether its size is stable across models and benchmarks, and a result like this is easy to state and hard to pin down. I would also want to know whether the Calibrated rollouts are identifiable in advance, since that determines whether the finding is actionable or merely diagnostic.

Writing quality: The framing is strong and the terminology is unusually well chosen — “Calling Without Looking” and “Looking Without Planning” are the kind of names that survive because they compress a mechanism into a phrase. The abstract does the paper a small disservice by front-loading the causal machinery before the anomalies that motivate it; the empirical puzzle is more arresting than the graph. The section that would most elevate the paper is a serious treatment of the corruption-detection confound, which as stated is the largest threat to the trajectory-level conclusions.

Verdict: weak accept — The causal reframing is correct and the estimand is genuinely useful, but the strongest claims lean on the intervention most vulnerable to the model noticing it has been intervened upon.

Takeaways

Three things a practitioner can steal:

1. The fixed-prefix counterfactual as a general attribution tool. Any time your system retrieves something mid-reasoning — a document, a database row, an API response, a tool output — you can measure whether it mattered by freezing the prefix and swapping only the retrieved item. This is cheap, requires no retraining, and answers a question that end-to-end accuracy structurally cannot. If swapping the retrieved document does not move the answer, your RAG pipeline is decorative at that step.

2. Separate “did not use it” from “used it at the wrong time.” The Calling Without Looking / Looking Without Planning split is a reusable diagnostic frame for any agentic system, not just visual ones. Before you tune a retrieval policy, establish which of the two you have: a grounding failure needs the model to condition harder on what came back, while a scheduling failure needs the call to move to a different point in the trajectory. The symptoms are identical in aggregate metrics and the fixes are unrelated.

3. Treat a benchmark gain as a claim requiring explanation, not as evidence of the mechanism you assumed. The specific transferable habit: when a capability is enabled and the score rises, ask what else changed besides the capability. Enabling a tool also changes token budget, decoding length, and the model’s implicit read on task difficulty — all of which move accuracy on their own. The causal graph is worth drawing before the ablation table.