Reasoning with Image Generation

Paper: 2609.16409 Authors: Nishad Singhi, Hector Garcia Rodriguez, Aditya Arora, Marcus Rohrbach, Anna Rohrbach Categories: cs.CV

The Gap

Chain-of-thought let language models decompose problems into intermediate steps before answering, and that move reshaped NLP. The paper’s diagnosis is that confining reasoning to the textual domain presents limitations for tasks requiring direct manipulation of visual representations.

What exists today is a toolkit of external visual expert tools such as depth estimation or object detection modules. The paper names their limit precisely: they are fundamentally limited by their reliance on narrow, rigid operations that cannot flexibly generate or transform visual content. A depth estimator answers one kind of question. It cannot remove an occluder, invent a missing viewpoint, or assemble a floorplan from disjoint glimpses — which is exactly what multi-view spatial reasoning needs.

   WHY TEXT-ONLY AND FIXED TOOLS BOTH RUN OUT

   problem needs VISUAL MANIPULATION as a step
     not just "describe the image"
     not just "run detector once"
         |
         +-- text-only CoT
         |     decomposes in language
         |     CANNOT change the visual state
         |     fails when the missing info is spatial
         |
         +-- specialist vision tools
         |     depth / detection / segmentation
         |     operations are NARROW and RIGID
         |     cannot GENERATE or TRANSFORM content
         |
         v
   [GAP]
     need a visual step that accepts natural language
     and can do OPEN-ENDED visual operations
     (remove occlusion, synthesize missing views, ...)

The Increment

One sentence: Before this paper, multimodal reasoning either stayed in text or called fixed-function vision tools; after it, an image-generation model is used as an open-ended visual reasoning operator, with up to 25% gains on six visual reasoning tasks.

Core Mechanism

ReImaGin’s move is to stop treating the image model as a content producer and treat it as a reasoning operator. The pipeline is: the multimodal LLM decides that a visual step is needed, issues a natural-language command to a generation model, receives the transformed image, and continues reasoning on that result.

The difference from the prior toolkit is the command interface. A detector answers a fixed query type. A generator that accepts language can, in principle, perform open-ended visual operations, like removing an occlusion or generating a floorplan from multiple disjoint views of a room. Those two examples are the paper’s own illustrations of what “flexible” means: the first edits a scene; the second synthesizes a representation no single input view contains.

Across six diverse visual reasoning tasks including multi-view spatial reasoning and collision prediction, the method consistently outperforms both text-only reasoning and specialist vision-tool baselines, with gains of up to 25%. The comparison matters on both axes: beating text-only shows the visual step is not decoration; beating specialist tools shows the step must be generative, not merely perceptual.

   ReImaGin AS A REASONING LOOP

   MLLM sees the problem
     |
     v
   decides: need a visual step
     |
     v
   natural-language command --> image generation model
     "remove the occluder"
     "make a floorplan from these views"
     |
     v
   transformed image returns
     |
     v
   MLLM continues reasoning on the NEW visual state
     (repeat if needed)

Think of it as a draftsman who can be told what to redraw, rather than a clerk who only stamps forms. Fixed tools are the stamp: reliable, narrow, and unable to invent a missing drawing. The generative step is the draftsman: slower and less guaranteed, but able to produce the intermediate artifact the problem actually needs. The paper’s claim is that several spatial problems require drawings, not stamps — and that the loop that can request drawings beats the loop that cannot.

Key Concepts

• Visual reasoning as state change: the useful intermediate is not a better caption or a better detection box; it is a new visual state the next reasoning step can use.
• Open-ended visual operations: commands in language that the generator can realize (edit, complete, re-project), rather than a closed menu of tool APIs.
• Reasoning loop vs one-shot vision: the model can decide when to invoke the visual step, not just consume a precomputed feature.

Framework Shift

Before (tool-augmented MLLM):          After (ReImaGin):
  MLLM + depth/detector/seg            MLLM + generative visual step
  tools answer FIXED query types       steps accept LANGUAGE commands
  visual state is mostly READ-ONLY     visual state can be REWRITTEN
  cannot invent missing views          can synthesize intermediate images
  good for recognition-shaped tasks    good for manipulation-shaped tasks

From attaching perception modules to the language model, to letting the language model drive open-ended visual transformation as part of the reasoning trace, the core shift is that the intermediate representation becomes editable.

Expert Assessment

Problem choice: Strong. The text-only / rigid-tool dichotomy is a real fork in multimodal agent design, and naming “flexible generation as reasoning” is a cleaner research object than yet another multi-tool router.

Method maturity: Clever, not free. Image generators are stochastic and can hallucinate structure; the paper does not claim every generated intermediate is faithful. An unexamined assumption worth naming: that the LLM can reliably judge when a visual step is worth the cost and whether the returned image helped. If that gate is noisy, the loop can waste budget or, worse, reason confidently on a bad redraw.

Experimental integrity: Six tasks and two baseline families (text-only and specialist tools) is a fair comparison shape. The headline is “up to 25%,” which is a max, not a mean — readers should want the per-task table before treating 25% as typical. Collision prediction and multi-view spatial reasoning are the right stress tests for the claim.

Writing quality: The abstract is crisp about the limitation of prior tools. What would elevate the paper: a cost/latency budget story (generation is expensive) and a failure taxonomy of bad intermediate images.

Verdict: weak accept — a well-motivated reframing of visual tools as generative reasoning steps, with promising multi-task evidence; still needs a tighter account of cost and of when the generator misleads the reasoner.

Takeaways

• If your multimodal agent keeps failing on tasks that need a missing view or a cleaned scene, the missing piece may be a generative edit step, not another detector.
• Design the visual tool as a language-commanded operator, not a fixed API — the command surface is what makes the step reusable across tasks.
• Gate visual steps explicitly: measure whether the returned image changed the answer, or you will pay generation cost for decorations.