WorldDirector: When World Models Learn to Separate Directing from Filming

Paper: 2607.02517 Authors: Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang, Qingyan Bai, Ka Leong Cheng, Yue Yu, Yixuan Li, Yihao Meng, Zichen Liu Categories: cs.CV

The Gap

Here’s the problem: video world models like Sora, Genie, and GameNGen have gotten impressively good at generating plausible-looking video. But they share a fundamental design flaw — they treat physical dynamics and pixel rendering as one fused computation. The model that figures out where the ball should go is the same model that decides what the ball looks like. This entanglement creates two stubborn problems:

1. Identity drift: When a dynamic object leaves the frame (say, a red car drives behind a building), the model has no persistent memory of what that car looked like. When it re-emerges, it might be a slightly different shade of red, a different shape, or even a different car entirely. The model relies on continuous visual observation to sustain identity — no observation, no memory.

2. Limited controllability: Want to orbit the camera 180 degrees around a scene? Want to specify that an object follows an exact trajectory? Current models make this extremely difficult because the “what happens” logic is buried inside pixel-level computations that are hard to steer with explicit instructions.

WorldDirector asks: what if we stop asking one model to do everything, and instead separate what happens from how it looks?

Problem: World models fuse dynamics + rendering
    |
    | Consequence: objects lose identity when unobserved;
    | camera control is limited by pixel-level entanglement
    v
Gap: No persistent object memory; no explicit trajectory control
    |
    | Key question: Can we decouple "where things go"
    | from "how things look"?
    v
Assumption: An LLM can plan trajectories; a separate model renders
    |
    | Implementation: LLM orchestrates 3D paths as control signals
    v
Method: Semantic motion planning -> structured trajectories -> video gen
    |
    | Evidence: Objects retain identity across long occlusions;
    | complex multi-event scenes with free viewpoint control
    v
Conclusion: Explicit decoupling yields persistent memory + controllability

The Increment

One sentence: Before this paper, world models that could generate long video could not remember what objects looked like after they left the scene; after this paper, they can — because the job of planning motion and the job of rendering pixels are given to separate specialists.

Core Mechanism

WorldDirector has a three-stage pipeline, and the key architectural decision is that each stage operates at a different level of abstraction.

Stage 1 — The LLM Orchestrator. A language model receives the scene description, user intent, and current state. Its job is to produce a structured plan: for each dynamic object in the scene, specify a 3D trajectory (position over time), and for the camera, specify a corresponding camera path. The LLM doesn’t see pixels — it reasons about objects, positions, and motion in a symbolic space. This is where “the red ball rolls behind the couch at t=5s and re-emerges at t=8s on the right side” gets turned into explicit coordinates.

Stage 2 — Trajectory Coordination. The orchestrated trajectories are processed to ensure physical plausibility and spatial consistency. This stage also maintains the persistent dynamic memory — a registry of object identities (appearance features, semantic labels, geometric properties) that persists regardless of whether the object is currently visible. When the ball goes behind the couch, its identity doesn’t vanish; it’s stored in memory and retrieved when the ball comes back.

Stage 3 — Video Generation. A diffusion-based video generator receives the structured trajectories as control signals (not raw pixel inputs from previous frames). Because the control signal says “render the red ball at position (x, y, z) at this frame,” the generator can look up the object’s stored appearance from the persistent memory and render it consistently, even if the object was invisible for hundreds of frames.

User Intent / Scene Description
        |
        v
+------------------------+
| LLM Orchestrator       |
| - Parses scene objects  |
| - Plans 3D trajectories |
| - Plans camera paths    |
+------------------------+
        |
        | Structured trajectory plan
        | (object_id, position, time, camera_pose)
        v
+------------------------+
| Trajectory Coordinator |
| - Physics consistency   |
| - Object identity       |
|   registry (persistent) |
| - Collision / occlusion |
|   tracking              |
+------------------------+
        |
        | Control signals + object memory lookup
        v
+------------------------+
| Video Generator        |
| (Diffusion-based)      |
| - Renders per frame     |
| - Queries object memory |
|   for appearance        |
| - Follows trajectory    |
|   control signals       |
+------------------------+
        |
        v
Output: Extended video with
persistent object identity,
free viewpoint, controlled events

The movie production metaphor. Think of making a film. In traditional world models, one person is simultaneously the director, the script supervisor, the actor, and the camera operator — they decide what happens, track continuity, perform, and film all at once. When the actor walks off set, there’s no production bible to remember what they were wearing. When they walk back on, the costume might be different.

WorldDirector separates these roles clearly:

• The LLM is the director. It decides the blocking: “The red ball rolls from left to right at t=0, disappears behind the couch at t=5, re-emerges on the right at t=8.” It works from a script (the scene description), not from looking at the set.

• The trajectory coordinator is the script supervisor. It maintains the production bible — a running document of every object’s identity: “Red ball, 5cm diameter, matte texture, currently at position behind the couch.” Even when the camera can’t see the ball, the script supervisor still knows exactly what it looks like and where it is.

• The video generator is the camera crew. It doesn’t decide *what to film — it follows the director’s shot list and the script supervisor’s notes. When told to film the ball re-emerging, it looks up the production bible and renders the exact same ball, not a guess.

This separation is load-bearing: without the director planning movements first, the camera crew would just be reactively filming whatever appears, with no way to maintain continuity. The persistent memory only works because someone (the coordinator) is dedicated to tracking it, separate from the rendering process.

Key Concepts

• Motion-Retrieval Decoupling: Imagine you’re giving someone directions to a restaurant. There are two ways to do it: (1) describe every single visual landmark they’ll pass (“turn left at the red building, walk past the flower shop…”), or (2) give them a map with coordinates (“go to 5th and Main”). Traditional world models do option 1 — they plan motion by describing visual changes frame by frame. WorldDirector does option 2 — it plans motion as abstract spatial trajectories (coordinates over time), then hands the map to a separate renderer. The benefit: the plan survives even when visual landmarks are temporarily hidden. The trajectory “object at (3, 0, 5) at t=8” doesn’t depend on whether the camera could see that position at t=6.

• Persistent Dynamic Memory: In a video game, when an NPC walks into a building and you can’t see them anymore, the game engine still knows exactly what they look like — their texture, model, and properties are stored in memory. Traditional world models don’t have this. They’re more like a painter who, the moment something leaves the canvas, forgets what it looked like and has to guess when painting it back in. WorldDirector gives the video generator a “game engine-like” object registry: each dynamic entity has a stored identity that persists across the entire video, regardless of visibility. This is why the red car stays red even after being occluded for 30 seconds.

• LLM as Spatial Reasoner: LLMs were trained on language, not geometry. But because they’ve absorbed enormous amounts of text describing spatial relationships (“the ball rolled under the table,” “the camera panned left”), they can surprisingly plan plausible 3D trajectories when given the right prompting structure. WorldDirector exploits this by treating the LLM as a “director” that converts high-level intent (“a car chase through a city”) into structured spatial plans. It’s not doing precise physics simulation — it’s doing *narrative-level spatial planning that’s good enough to guide a video generator.

Framework Shift

Before (mainstream):                After (this paper):

+---Scene-input---+                +---Scene-input---+
|                 |                |                 |
+--------+--------+                +--------+--------+
         |                                  |
         v                                  v
+--------+--------+                +--------+--------+
| Single Model     |                | LLM Director    |
| dynamics+render  |                | (plans motion)  |
| fused together   |                +--------+--------+
+--------+--------+                         |
         |                                  | trajectories
         | pixel-level                      v
         | coupling                  +--------+--------+
         v                          | Object Memory   |
+--------+--------+                | (persistent)    |
| Video Output     |                +--------+--------+
| identity lost    |                         |
| after occlusion  |                         | control signals
+------------------+                         v
                                    +--------+--------+
                                    | Video Generator |
                                    | (renders only)  |
                                    +--------+--------+
                                             |
                                             v
                                    +--------+--------+
                                    | Video Output    |
                                    | identity kept   |
                                    +-----------------+

From entangled dynamics-rendering to separated orchestration-rendering, the core shift is that planning what happens and painting what it looks like are no longer the same computation.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Persistent object identity under occlusion has been a known pain point in video generation since the first long-video models appeared. The field has been quietly papering over it with short clips and limited camera movement. WorldDirector names the problem clearly and offers an architectural solution rather than a patch. It sits at the right trajectory point — world models are moving from “impressive demos” to “useful tools,” and controllability + consistency are exactly what’s blocking that transition.

Method maturity: The insight is genuinely clever: use an LLM’s emergent spatial reasoning to externalize motion planning, then let a specialized model handle rendering. However, calling the LLM a “spatial reasoner” is generous — it’s doing coarse narrative-level planning that works for demonstrations but may not scale to precision-intensive scenarios (surgery simulation, robotic manipulation). The approach also introduces a new bottleneck: the LLM’s planning quality becomes a ceiling for the whole system. There might be simpler approaches (e.g., explicit 3D scene graphs with learned renderers) that achieve similar persistence without relying on LLM spatial reasoning, which is famously brittle outside common scenarios.

Experimental integrity: The abstract claims “unprecedented controllability and persistent dynamic object memory,” which is strong language. I’d want to see: (1) quantitative metrics on identity preservation across varying occlusion durations (not just cherry-picked examples), (2) failure mode analysis — what happens when the LLM plans physically implausible trajectories?, (3) comparison against strong baselines that use explicit 3D tracking rather than LLM planning. Video world model papers have a history of impressive demos that don’t hold up under systematic evaluation. Without seeing the full experiments, I’m cautiously optimistic but not convinced.

Writing quality: The abstract is clean and well-structured. The framing of “entangled dynamics with pixel rendering” is sharp and memorable. However, the phrase “unprecedented controllability” is a red flag — it signals marketing language rather than measured claims. The related work section (if following typical structure) likely under-engages with the 3D Gaussian splatting and NeRF-based scene reconstruction literature, which has been solving related persistence problems from a different angle. A section explicitly comparing the “LLM-as-planner” paradigm against “explicit 3D reconstruction” would strengthen the paper considerably.

Verdict: weak accept — The decoupling insight is valuable and the persistent memory framing is a genuine contribution, but the reliance on LLM spatial reasoning introduces fragility that the paper needs to address more honestly.

Takeaways

Three concrete ideas worth stealing:

1. Separate planning from rendering in generative pipelines. This isn’t just for video. Any generative system that mixes “what to generate” with “how to render it” can benefit from decomposing into a symbolic planner + a neural renderer. Think text-to-3D, audio synthesis, even procedural content generation.

2. Object identity registries as architectural components. The idea of maintaining a persistent lookup table of entity appearances — independent of the generation process — transfers directly to multi-agent simulation, digital twins, and long-form content generation. If your system forgets what an entity looks like when it’s not being observed, you need this.

3. LLMs as coarse spatial planners, not fine-grained simulators. Don’t ask an LLM to do physics. Ask it to do *blocking — the narrative-level “who goes where, when.” Then hand off to a specialized system for precision. This principle applies anywhere you’re tempted to use an LLM for tasks that require more precision than language naturally provides.