WorldCam: Grounding Interactive Game Worlds in Camera Geometry

Paper: 2603.16871 Authors: Jisu Nam, Yicong Hong, Chun-Hao Paul Huang, Feng Liu, JoungBin Lee, Jiyoung Kim, Siyoon Jin, Yunsung Lee, Jaeyoon Jung, Suhwan Choi Categories: cs.CV

The Gap

Recent video diffusion models (GameNGen, Genie, DIAMOND) can generate interactive gaming worlds where users explore environments. But they treat user actions as abstract conditioning signals—discrete tokens like “move_forward” or “turn_left” fed into the model. This abstraction breaks the fundamental geometric coupling: in a 3D world, actions induce specific camera motions that accumulate into a global trajectory. Without this geometric grounding, models struggle with precise control (your “forward” might drift sideways) and long-horizon consistency (revisiting a location looks different because the model has no spatial memory).

Problem: Abstract actions lose geometric meaning
   |
   v
Assumption: Camera pose encodes both immediate motion and global position
   |
   v
Method: (1) Actions -> Lie algebra -> 6-DoF poses -> Camera embedder
        (2) Global poses -> Spatial index -> Retrieve past observations
   |
   v
Evidence: Outperforms SOTA on action accuracy, visual quality, 3D consistency
   |
   v
Conclusion: Geometric representation unifies control and memory

The Increment

One sentence: Before—actions as abstract tokens with implicit geometry; after—actions as explicit camera poses that ground both immediate control and spatial memory.

Core Mechanism

The method has two coupled components. First, a physics-based action space converts user inputs (keyboard/controller) into continuous 6-DoF camera poses using Lie algebra (se(3) representation). These poses are injected into the video diffusion transformer via a learned camera embedder, ensuring the generated frames align with the intended camera motion. Second, as the user navigates, the system maintains a global camera trajectory. When generating new frames, it uses the current global pose as a spatial index to retrieve relevant past observations from a memory bank, then conditions the diffusion model on these retrieved frames to maintain 3D consistency when revisiting locations.

User Input -> Lie Algebra -> 6-DoF Pose -> Camera Embedder
                                              |
                                              v
                                    Diffusion Transformer
                                              |
                                              v
                                      Generated Frame
                                              |
                                              v
Global Pose -> Spatial Index -> Memory Bank -> Retrieved Frames
                                                      |
                                                      v
                                              (feed back as condition)

Think of it like a GPS-guided tour bus with a photo archive. User inputs are steering and acceleration—the system translates these into precise GPS coordinates and heading (camera pose in Lie algebra). The bus driver (camera embedder) uses these coordinates to navigate exactly where you want to go. Meanwhile, the bus maintains a complete GPS log of everywhere it’s been. When you return to a familiar intersection, the system checks the GPS log, pulls up photos taken from that exact spot before (pose-indexed retrieval), and uses them to ensure the view looks consistent with your previous visit. Without GPS coordinates, the driver would just guess where “turn left” means, and you’d have no way to match current views with past photos.

Key Concepts

• Lie Algebra for Camera Poses: Imagine you’re describing how to move a camera in 3D space. You could say “move 2 meters forward, rotate 30 degrees left,” but these discrete steps don’t compose smoothly—two rotations don’t add like numbers. Lie algebra (specifically se(3)) is a mathematical framework that represents camera motions as vectors in a special space where you can add, scale, and interpolate them smoothly. It’s like switching from “turn-by-turn directions” to “velocity vectors”—you can now describe continuous motion paths. The paper uses this to convert discrete user inputs into smooth, physically plausible camera trajectories that the diffusion model can understand.

• Pose-Indexed Retrieval: When you generate a long video sequence, early frames fade from the model’s attention window—it “forgets” what distant locations looked like. The solution: store past frames in a memory bank indexed by their camera pose (position + orientation). When generating a new frame, compute its pose, search the memory bank for frames with similar poses, and feed those as additional context. It’s like a photo album organized by GPS coordinates—when you return to a location, you flip to that page to remember what it should look like. This prevents the model from hallucinating inconsistent views when revisiting areas.

Framework Shift

Before (mainstream approach):        After (this paper):

User Input                           User Input
    |                                    |
    v                                    v
[Action Token]                       [Lie Algebra]
    |                                    |
    v                                    v
Diffusion Model                      [6-DoF Pose] -----> Camera Embedder
    |                                    |                     |
    v                                    v                     v
Generated Frame                      Diffusion Model <--- [Pose Index]
(drifts over time)                       |                     |
                                         v                     v
                                   Generated Frame        Memory Bank
                                   (geometrically          (past frames)
                                    consistent)

From abstract action tokens to explicit geometric poses, the core shift is treating the 3D world’s geometry as a first-class citizen rather than an emergent property.

Expert Assessment

Problem choice: Real gap. Existing gaming world models do suffer from drift and inconsistency over long horizons—anyone who’s tried GameNGen knows the world gradually “melts.” The geometric framing is natural and well-motivated, not manufactured.

Method maturity: The insight about camera pose as a unifying representation is elegant, but the execution is fairly standard—Lie algebra for poses is textbook robotics, and retrieval-augmented generation is well-explored. The contribution is more in the problem formulation than novel techniques. I’d want to see ablations showing that both components (pose embedder + retrieval) are necessary, not just one doing the heavy lifting.

Experimental integrity: The paper introduces its own dataset (3,000 minutes of gameplay with camera annotations), which is good for reproducibility but raises questions about baseline fairness—did prior methods get to train on this data? The metrics (action controllability, 3D consistency) seem reasonable, but I’d scrutinize whether they’re measuring what matters to actual users versus what’s easy to quantify.

Writing quality: The abstract is clear and well-structured. The geometric motivation is compelling. However, I suspect the related work section undersells how much prior work exists on camera control in video generation (NeRF-based methods, camera-conditioned diffusion). The paper would be stronger if it explicitly positioned itself relative to those threads.

Verdict: Weak accept—solid contribution with clear improvements, but the novelty is more in problem framing than technical innovation. The dataset is valuable, and the results are convincing, but this feels like an incremental step rather than a paradigm shift.

Takeaways

Practitioners can steal the core principle: when your domain has inherent geometric structure, exploit it explicitly rather than hoping the model learns it implicitly. For any task involving spatial navigation or 3D consistency (robotics, AR/VR, autonomous driving), represent actions and states in their natural geometric form (poses, trajectories) rather than abstract tokens. The Lie algebra trick is particularly useful—it lets you interpolate and compose camera motions smoothly, which is critical for continuous control. The pose-indexed retrieval pattern generalizes: maintain a memory bank keyed by geometric coordinates, retrieve relevant past observations when revisiting locations. This is cheaper than trying to cram infinite context into attention windows.