World2VLM: Teaching Vision Models to Imagine Motion by Distilling World Models

Paper: 2604.26934 Authors: Wanyue Zhang, Wenxiang Wu, Wang Xu, Jiaxin Luo, Helu Zhi, Yibin Huang, Shuo Ren, Zitao Liu, Jiajun Zhang Categories: cs.CV

The Gap

Vision-language models excel at describing static images but fail when asked “if I move forward 2 meters, what will I see?” Two camps tried to fix this: (1) synthetic data scaling approaches that generate more training examples but don’t explicitly model how motion transforms scenes, and (2) inference-time coupling where you attach a world model to generate future views on-the-fly, which works but burns compute at every query. The gap: VLMs lack internalized spatial imagination, and existing fixes either skip the motion physics or pay a heavy runtime tax.

Problem: VLMs can't imagine scene evolution under motion
   |
   v
Assumption: World models know motion physics, VLMs need that knowledge baked in
   |
   v
Method: Use world model as teacher during training, not crutch during inference
   |
   v
Evidence: Matches inference-time coupling performance at zero inference cost
   |
   v
Conclusion: Distillation transfers spatial imagination into VLM weights

The Increment

One sentence: Before, VLMs either guessed at spatial outcomes or needed a world model running alongside them at inference; after, they internalize motion-conditioned reasoning during training and answer spatial queries directly.

Core Mechanism

World2VLM has three stages. First, a view-consistent world model (trained on video data) takes an initial image and a camera trajectory, then generates geometrically aligned future frames showing what you’d see after that motion. Second, the framework extracts two types of supervision from these synthetic sequences: forward reasoning (given action, predict outcome) and inverse reasoning (given outcome, infer action). Third, the VLM undergoes two-stage post-training—first on forward tasks to learn action-to-outcome mappings, then on inverse tasks to learn outcome-to-action inference.

Stage 1: World Model Generation
   [Initial Image] + [Camera Trajectory] 
          |
          v
   [World Model] --> [Future View Sequence]
          |
          v
   Geometrically consistent frames

Stage 2: Supervision Extraction
   [Future Views] --> Forward: (action, initial) -> outcome
                  --> Inverse: (outcome, initial) -> action

Stage 3: VLM Post-Training
   Phase 1: Forward reasoning tasks
   Phase 2: Inverse reasoning tasks
          |
          v
   [VLM with internalized spatial imagination]

Think of it like teaching someone to navigate by mental simulation. The world model is an experienced guide who can vividly describe what’s around each corner. Instead of having the guide walk alongside the student forever (inference-time coupling), you have the guide create a training course with detailed “if you turn left here, you’ll see X” scenarios. The student practices these scenarios until they can mentally simulate turns themselves. The forward tasks are like “I’m turning left—what will I see?” and the inverse tasks are “I see a red door—what turn got me here?” After enough practice, the student internalizes the spatial logic and no longer needs the guide present.

Key Concepts

• View-consistent world model: A generative model trained on video that doesn’t just predict any plausible next frame, but specifically predicts frames that maintain geometric consistency with a specified camera motion. If you say “move forward 1 meter,” it generates a frame where objects scale and shift exactly as they would under that motion, not just a visually plausible but geometrically arbitrary scene. This is crucial because spatial reasoning requires geometric precision, not just visual coherence.

• Forward vs inverse spatial reasoning: Forward reasoning is “given this action, what outcome?” (predictive). Inverse reasoning is “given this outcome, what action caused it?” (diagnostic). Most prior work focuses only on forward, but inverse reasoning is essential for tasks like “which direction should I go to see the fountain?” The VLM needs both directions to build a complete mental model of space.

• Distillation as internalization: Traditional distillation compresses a large model into a small one. Here, distillation transfers a capability (spatial imagination) from one modality (generative world model) to another (vision-language model). The world model never runs at inference—it’s purely a teacher that creates training data encoding its spatial knowledge. The VLM learns to reproduce that knowledge through its own parameters.

Framework Shift

Before (mainstream approach):        After (this paper):

VLM alone:                           Training time:
[Image] -> [VLM] -> [Guess]          [Image] + [Trajectory]
   ^                                      |
   |                                      v
   No motion model                   [World Model Teacher]
                                          |
OR                                        v
                                     [Synthetic Views]
Inference-time coupling:                  |
[Image] + [Action]                        v
   |                                 [Forward + Inverse Tasks]
   v                                      |
[World Model] -> [Future View]            v
   |                                 [VLM Post-Training]
   v                                      |
[VLM] -> [Answer]                         v
   ^                                 Inference time:
   |                                 [Image] + [Query]
   Expensive at every query               |
                                          v
                                     [VLM] -> [Answer]
                                          ^
                                          |
                                     Knowledge baked in

From runtime dependency to training-time knowledge transfer, the core shift is moving the world model from inference partner to training teacher.

Expert Assessment

Problem choice: Real gap. VLMs demonstrably struggle with spatial reasoning (well-documented in SAT, VSI-Bench benchmarks), and the two existing solutions have clear tradeoffs. This isn’t manufactured—it’s a natural next step after observing that inference-time coupling works but doesn’t scale.

Method maturity: Clever insight with solid execution. The two-stage training (forward then inverse) shows thoughtful design—inverse reasoning is harder and benefits from forward priming. The use of a view-consistent world model (not just any video generator) is the right tool for the job. However, the paper doesn’t explore whether simpler motion models (e.g., depth-based warping) could provide similar supervision at lower cost. That’s a missed ablation.

Experimental integrity: Baselines are fair and comprehensive (base VLM, synthetic data scaling, inference-time coupling). The improvements are consistent across four benchmarks, which is strong evidence. One concern: the world model quality ceiling—if the teacher hallucinates geometry, does the student learn bad habits? The paper doesn’t deeply probe failure modes or show where distillation degrades the world model’s knowledge.

Writing quality: The method section is clear, but the related work undersells the novelty. The paper frames this as “distillation” but it’s really cross-modal capability transfer, which is conceptually richer. The ablation section (if expanded) could show which components matter most—is it the view consistency, the two-stage training, or the inverse tasks?

Verdict: weak accept — Solid contribution that advances VLM spatial reasoning with a practical training recipe, though the evaluation could probe failure modes more deeply and the method could be positioned more boldly.

Takeaways

The two-stage training recipe (forward then inverse) is immediately transferable to other domains where you want to teach a model bidirectional reasoning—train prediction first, then diagnosis. The broader idea: if you have a strong simulator (world model, physics engine, theorem prover) that’s too slow for inference, use it to generate structured training data that encodes its knowledge, then distill that into a faster model. This is more efficient than either ignoring the simulator or running it at inference. Specifically for robotics or embodied AI: this pattern of using generative models as training-time teachers rather than inference-time tools could unlock better sample efficiency.