WorldDrive: Teaching World Models to Speak the Planner's Language

Paper: 2603.14948 Authors: Xingtai Gui, Meijie Zhang, Tianyi Yan, Wencheng Han, Jiahao Gong, Feiyang Tan, Cheng-zhong Xu, Jianbing Shen Categories: cs.CV

The Gap

Driving world models have gotten good at predicting what a scene will look like in the future—think video generation conditioned on sensor input. But here’s the problem: these models optimize for visual fidelity (making pretty, realistic videos), while planners need something different—representations that encode actionable motion information. The world model learns “this is what the scene looks like if I turn left,” but the planner has to learn motion planning from scratch using a completely different representation space. It’s like having a weather forecaster who can show you beautiful animations of tomorrow’s storm, but can’t tell you whether to bring an umbrella.

Prior work (GAIA-1, DriveDreamer, etc.) focuses on controllable scene generation—you can condition on high-level actions, but the motion representation isn’t explicitly designed to be inherited by the planner. The world model and planner live in separate worlds, connected only by a thin bridge of high-level action labels.

Problem: World models optimize for visual fidelity
         |
         v
Assumption: Motion representation should be planner-compatible
         |
         v
Method: Trajectory-aware world model + shared encoders
         |
         v
Evidence: NAVSIM/nuScenes benchmarks show planning improvement
         |
         v
Conclusion: Unifying representations bridges generation and planning

The Increment

One sentence: Before, world models generated scenes and planners learned separately; now, they share a unified representation space where visual dynamics and motion intentions are explicitly coupled.

Core Mechanism

WorldDrive has three components working in concert. First, the Trajectory-aware Driving World Model takes sensor input and a trajectory from a learned vocabulary, then generates future scenes conditioned on that specific trajectory. The key is that trajectory tokens are embedded into the generation process, forcing the model to learn representations where visual dynamics (how the scene evolves) and motion intentions (where the car wants to go) are tightly coupled.

Second, the Multi-modal Planner doesn’t start from scratch. It inherits the vision and motion encoders from the world model—these are frozen, pre-trained representations that already understand the relationship between what you see and how you move. The planner adds a lightweight interaction module that combines motion representation, visual representation, and ego vehicle status to generate multiple trajectory candidates.

Third, the Future-aware Rewarder acts as a quality filter. It takes the frozen world model and uses it to evaluate trajectory candidates in real-time. For each candidate trajectory, it extracts future latent representations from the world model and scores them, selecting the trajectory that the world model thinks will lead to the most plausible future state.

Sensor Input ---> [Vision Encoder] ---> Visual Features
                         |                      |
Trajectory Vocab --------+                      |
                         |                      v
                         v              [World Model Decoder]
                  [Motion Encoder]              |
                         |                      v
                         |              Generated Future Scenes
                         |
                         +---> [Frozen for Planner]
                         |
                         v
              [Multi-modal Planner] ---> Trajectory Candidates
                         |
                         v
              [Future-aware Rewarder] ---> Selected Trajectory
                    (uses frozen world model)

Think of it like training a pilot. Traditional approaches are like teaching someone to fly using a flight simulator (world model) and then separately teaching them decision-making in a classroom (planner)—the simulator knowledge doesn’t transfer. WorldDrive is like a simulator that explicitly teaches both “what happens when I pull this stick” AND “how to decide which stick to pull,” using the same mental model. The instructor (world model) learns to explain maneuvers in terms the student (planner) will actually use. Then, when the student practices decision-making, they’re already fluent in the language the simulator taught them. Finally, before executing any maneuver, the student can mentally simulate it using their instructor’s frozen knowledge to check if it makes sense.

Key Concepts

• Trajectory Vocabulary: Instead of conditioning the world model on raw trajectory coordinates or high-level action labels, WorldDrive learns a discrete vocabulary of trajectory tokens through VQ-VAE. Think of it like learning a language of motion primitives—“sharp left turn,” “gradual lane change,” “emergency brake”—rather than describing every motion as a sequence of steering angles. This discretization serves two purposes: it makes the generation problem more tractable (finite vocabulary vs infinite continuous space), and it creates a shared “language” that both the world model and planner can speak. When the world model generates a scene conditioned on token #47, and the planner outputs token #47, they’re talking about the same motion intention.

• Representation Inheritance: Most end-to-end driving systems train the planner from scratch on raw sensor data. WorldDrive instead freezes the vision and motion encoders from the world model and transfers them to the planner. Why does this matter? The world model has already learned, through scene generation, which visual features matter for predicting motion and which motion patterns are physically plausible. The planner inherits this mature understanding rather than relearning it. It’s the difference between giving someone a pre-built vocabulary vs making them learn language from scratch—they can focus on composition (planning) rather than fundamentals (what features matter).

• Future-aware Reward: Traditional planners evaluate trajectories using hand-crafted cost functions (distance to lane center, collision risk, etc.). WorldDrive’s rewarder instead asks the frozen world model: “If I execute this trajectory, what future state will I reach, and how plausible is that state?” It extracts latent representations from the world model for each candidate trajectory and scores them based on the model’s internal notion of plausibility. This is like having a chess engine that doesn’t just evaluate positions using material count, but uses a neural network trained on millions of games to recognize “this position feels wrong” even if it can’t articulate why.

Framework Shift

Before (mainstream approach):        After (this paper):

[World Model]                        [Trajectory-aware World Model]
     |                                    |         ^
     | (visual scenes)                   |         |
     v                                   | (shared | (trajectory
[Separate Training]                      |  repr.) |  condition)
     |                                   v         |
     | (raw sensors)                [Multi-modal Planner]
     v                                    |
[Planner]                                | (candidates)
     |                                   v
     v                            [Future-aware Rewarder]
[Trajectory]                          (uses frozen WM)
                                         |
                                         v
                                    [Trajectory]

Gap: Different representations      Bridge: Unified representation

From parallel pipelines to nested inheritance, the core shift is making the world model’s knowledge directly usable by the planner through shared representation space.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The autonomous driving community has been excited about world models (see Tesla’s FSD, Wayve’s GAIA), but the connection to planning has been hand-wavy. Most papers show that world models can generate realistic videos, then separately show that planners can drive, but don’t rigorously connect the two. This paper tackles that schism head-on. It sits at a natural inflection point where world models are mature enough that we should be asking “how do we actually use them for decision-making?”

Method maturity: The approach is more clever than brute force. The trajectory vocabulary idea is elegant—it solves both the conditioning problem and the representation sharing problem with one mechanism. However, I’m skeptical about whether freezing the encoders is always optimal. The paper claims this ensures “mature representations,” but it also means the planner can’t adapt those representations to its specific needs. A more nuanced approach might allow fine-tuning with a small learning rate. The future-aware rewarder feels a bit bolted on—it’s not clear this is better than just training the planner end-to-end with the world model in the loop.

Experimental integrity: The benchmarks (NAVSIM, nuScenes) are standard and the baselines seem fair. The paper claims “leading performance among vision-only methods,” which is a careful hedge—they’re not claiming SOTA overall, just within the vision-only category. The ablations are reasonable, showing that each component contributes. One red flag: they don’t show failure cases or discuss when the approach breaks down. Also, the “high-fidelity action-controlled video generation” claim needs scrutiny—are they cherry-picking examples?

Writing quality: The abstract is dense and jargon-heavy (“planner-shared and inheritable” is awkward). The method section is clearer, but the paper would benefit from a more explicit discussion of design choices. Why VQ-VAE for trajectory vocabulary instead of alternatives? Why freeze encoders completely? The related work section does a decent job positioning the work, but could be more critical of prior approaches.

Verdict: Weak accept — solid contribution to a real problem, but the method feels like a first step rather than a definitive solution, and the experimental validation could be more thorough.

Takeaways

If you’re building any system where you have a generative model (world model, simulator, forecaster) and a decision-making component (planner, controller, agent), steal this idea: design your generative model’s conditioning mechanism to create representations that your decision-maker can inherit. Don’t just condition on high-level actions; condition on a structured representation (like the trajectory vocabulary here) that forces the generative model to learn decision-relevant features.

The trajectory vocabulary approach via VQ-VAE is directly transferable to other domains. If you’re doing robot manipulation, instead of conditioning your world model on raw joint angles, learn a vocabulary of motion primitives. If you’re doing dialogue systems, instead of conditioning on raw text, learn a vocabulary of conversational moves.

The “freeze and transfer” strategy is worth experimenting with, even if you ultimately fine-tune. Starting with frozen pre-trained representations from a related task often works better than random initialization, and it forces you to think about what knowledge should transfer vs what should be task-specific.