Imaginative Perception Tokens: Teaching Vision Models to Hallucinate Correctly

Paper: 2606.03988 Authors: Mahtab Bigverdi, Lindsey Li, Weikai Huang, Yiming Liu, Jaemin Cho, Jieyu Zhang, Tuhin Kundu, Chris Dangjoo Kim, Zelun Luo, Linda Shapiro Categories: cs.AI

The Gap

Vision language models (VLMs) like GPT-4V and Gemini handle direct perception well—if you show them an image and ask what’s there, they answer correctly. But throw spatial reasoning at them where critical information is occluded or requires imagining alternative viewpoints, and they crumble. Prior work tried textual chain-of-thought (CoT) reasoning, essentially forcing the model to verbalize spatial computations through language. The problem: spatial structure doesn’t compress well into linear text. You’re asking a model to describe a 3D rotation or trace an occluded path using words, which is like trying to explain how to tie shoelaces over the phone.

The boundary this paper addresses: VLMs lack intermediate representations for spatial imagination. They jump directly from image tokens to text output, with no way to represent “what I would see if I moved here” or “the occluded region probably looks like this.”

Problem: VLMs fail on spatial tasks requiring imagination
                    |
                    v
Assumption: Language is wrong modality for spatial reasoning
                    |
                    v
Method: Introduce Imaginative Perception Tokens (IPT)
        - Intermediate perceptual representations
        - Supervised by ground truth imagined views
                    |
                    v
Evidence: 3.4% accuracy gain on multiview counting
          Competitive with closed-source models on path tracing
          Outperforms textual CoT on spatial tasks
                    |
                    v
Conclusion: Spatial reasoning needs perceptual intermediate steps
            Forcing computation through language degrades performance

The Increment

One sentence: Before this paper, VLMs reasoned about unobserved spatial structure by verbalizing it awkwardly in text; after, they generate intermediate perceptual tokens representing imagined views, keeping spatial computation in the visual modality.

Core Mechanism

IPT adds a bottleneck layer between the vision encoder and the final answer. When the model encounters a spatial reasoning task, it first generates IPT—tokens that represent what the model would perceive under alternative spatial configurations (different viewpoint, occluded region filled in, multiple partial views integrated). These tokens are supervised using ground truth “imagination” data: for perspective-taking tasks, the ground truth is the actual image from the target viewpoint; for path tracing, it’s a visualization of the occluded path; for multiview counting, it’s integrated representations of multiple views.

The architecture uses BAGEL (a unified VLM) as the backbone. During training, IPT are generated as intermediate outputs, then fed back into the model alongside the original image tokens to produce the final answer. At inference, you can either generate actual images from IPT (using a decoder) to visualize what the model imagined, or skip image generation entirely and just use the tokens internally—the paper shows both work, but skipping generation is faster.

Input Image --> Vision Encoder --> Image Tokens
                                        |
                                        v
                      [Generate Imaginative Perception Tokens]
                                        |
                                        v
                                      IPT
                                        |
                                        v
                Image Tokens + IPT --> LLM Decoder --> Answer
                                        ^
                                        |
                        (supervised by ground truth imagination)

Think of IPT like architectural blueprints. When an architect designs a building, they don’t describe every angle and sightline in text—they draw elevations, cross-sections, and perspective views. Each drawing is an “imagination” of what you’d see from a specific vantage point or if you sliced through the structure. IPT are those cross-sections for spatial reasoning: instead of verbalizing “if I stood here, I would see X, which means Y,” the model draws the intermediate perceptual state. The original image is the building’s exterior photo; IPT are the internal views you’d need to reason about the structure. The final answer is the architectural conclusion (e.g., “this hallway connects to that room”). By generating these intermediate drawings, the model keeps spatial computation in the visual domain where distances, angles, and occlusions have natural representations, rather than forcing it through the lossy codec of language.

Key Concepts

• Imaginative Perception: VLMs typically reason over what’s directly observable in an image. Imaginative perception is the ability to infer what would be perceived under counterfactual spatial conditions—a different viewpoint, an occluded space made visible, or multiple fragmentary observations unified. It’s not hallucination (making things up ungrounded in input), but disciplined extrapolation: “Given what I see and the laws of 3D geometry, this is what I would see if X.” For example, if you show a model a photo of a street corner from the south, imaginative perception would generate what the same scene looks like from the north, constrained by the visible buildings, perspective geometry, and common-sense spatial layout. The key is that these imagined perceptions remain consistent with observed input—they’re hypotheses about unobserved structure, not random generation.

• Modality Mismatch: Spatial reasoning involves operations like rotation, translation, occlusion reasoning, and integrating multiple viewpoints. These operations have compact representations in visual or geometric modalities (rotation matrices, 3D coordinates, depth maps), but decompress poorly into language. When you force a VLM to reason spatially through text (as in textual chain-of-thought), you’re asking it to serialize parallel spatial structure into linear tokens. It’s like describing a sculpture by listing the coordinates of every point versus just showing a 3D rendering—the latter preserves structure. This paper shows that textual CoT degrades performance on spatial tasks because the modality can’t efficiently carry the computation. IPT sidestep this by keeping intermediate reasoning in the perceptual modality.

Framework Shift

Before (textual CoT approach):        After (IPT approach):

Image --> Tokens --> LLM              Image --> Tokens --> IPT Generator
            |                                      |              |
            v                                      v              v
      "If I rotate..."                         [visual         IPT
      "Then I would see..."                  imagination]        |
      "Which means..."                                           v
            |                                             Tokens + IPT
            v                                                     |
         Answer                                                   v
                                                               Answer

(spatial computation                  (spatial computation stays
 forced through text)                  in visual modality)

One sentence: From serializing spatial structure into text descriptions, to generating intermediate perceptual representations that externalize imagined views, the core shift is keeping spatial reasoning in the modality where it’s naturally structured.

Expert Assessment

Problem choice: Real gap. Spatial reasoning is a known weak spot for VLMs, and the limitation of textual CoT for spatial tasks is under-explored. The problem sits at the intersection of embodied AI (where spatial reasoning matters for robotics) and multimodal reasoning (where vision-language integration struggles). Not a manufactured problem—it’s well-motivated by actual failure modes.

Method maturity: Clever insight, but execution is straightforward. The core idea—intermediate perceptual representations supervised by ground truth imaginations—is sound. The simplicity is a feature: you’re just adding a supervised intermediate step. However, the reliance on ground truth imagination data limits generalizability. The paper uses rendered or captured ground truth views for supervision, which works for their three tasks but doesn’t scale to arbitrary spatial reasoning. A future direction would be self-supervised IPT or learning to imagine without ground truth.

Experimental integrity: Baselines are fair. They compare against textual CoT, label-only supervision, and closed-source models (GPT-4V, Gemini). The numbers are modest (3.4% gain on multiview counting) but consistent. One concern: the tasks are somewhat narrow (perspective taking, path tracing, multiview counting). It’s unclear whether IPT generalize to broader spatial reasoning like 3D reconstruction or spatial question answering on real-world images. The fact that textual CoT degrades performance is compelling evidence of modality mismatch, but I’d want to see ablations on how much supervision is needed and whether IPT help on tasks where ground truth imagination isn’t available.

Writing quality: The paper is clear but front-loads motivation over mechanism. Section 3 (method) buries the architectural details—how IPT are generated, how they’re integrated into the backbone, and what the supervision signal looks like. Rewriting Section 3 to lead with a concrete example (input image → IPT generation → final answer) before diving into generalities would elevate clarity. The related work section is thorough but reads like a laundry list. Cutting it by half and focusing on the three closest lines of work (spatial reasoning in VLMs, intermediate representations, CoT vs. modality-specific reasoning) would sharpen the contribution.

Verdict: weak accept — The core insight (spatial reasoning needs perceptual intermediate steps, not textual ones) is valuable, and the evidence supports it, but the method’s generalizability is limited by reliance on ground truth imagination data, and the experimental scope is narrow.

Takeaways

Supervision signal for imagination: The idea of supervising intermediate representations with “what you would perceive under alternative conditions” transfers beyond spatial reasoning. For temporal reasoning (predicting future frames), causal reasoning (imagining counterfactual outcomes), or even code generation (imagining intermediate execution states), you can apply the same pattern: generate intermediate perceptual or symbolic states supervised by ground truth alternatives, then use those states to improve final predictions.

Modality-specific reasoning: The evidence that textual CoT degrades spatial reasoning performance is a wake-up call. If your task has natural structure in a non-linguistic modality (visual, geometric, temporal), forcing intermediate reasoning through text may hurt rather than help. Consider whether your CoT should be visual (like IPT), symbolic (like execution traces), or stay in the domain where the computation is naturally expressed.

Externalize imagination for interpretability: Generating IPT at inference time produces human-interpretable intermediate views. This is useful for debugging model failures—if the model gets the wrong answer, you can inspect the imagined view to see where spatial reasoning broke down. This pattern (externalize intermediate reasoning as inspectable artifacts) applies to any domain where interpretability matters.