AEON: Unifying Video and 3D World Models

Paper: 2610.28631
Authors: Weiqi Zhang, Wenyuan Zhang, Yu-Shen Liu, Junsheng Zhou
Categories: cs.CV

The Gap

The quest to build scalable physical simulators and world models for embodied agents has split into two fundamentally divergent and incompatible technical pathways:

1. Video Generative World Models (e.g., Sora, Runway Gen-3, Kling):
   • Strengths: Highly expressive, capturing rich textural photorealism, fluid non-rigid motion, complex lighting, and open-domain physical phenomena.
   • Failure Mode: Operates purely in 2D latent space. They possess no explicit 3D geometry. When the simulated camera circles back to a previously observed room, walls warp, objects mutate, and multi-view parallax breaks down completely.
2. 3D Generative World Models (e.g., Marble, 3DGS-based planners):
   • Strengths: Strict multi-view consistency, rigid geometric bounds, and deterministic spatial persistence.
   • Failure Mode: Struggles to simulate dynamic fluid motion, deformable actors, or high-fidelity open-world physics. Environments appear static, plastic, and lifeless.

Embodied AI urgently requires a world model that simultaneously commands the photorealistic visual dynamics of video generation and the rigid, persistent multi-view geometry of 3D representations.

   THE DUAL PATHWAY SCHISM IN WORLD MODEL ARCHITECTURES

   Goal: "Simulate a robot walking into a dynamic kitchen and returning"
          |
          +-----------------------+-----------------------+
          |                                               |
          v                                               v
   [Video World Model (Sora-style)]       [3D World Model (Gaussian-style)]
   - Breathtaking photorealism            - Strict 3D geometric consistency
   - Natural dynamic steam & water        - Multi-view camera trajectory tracking
   - BUT: When robot turns around,        - BUT: Static and rigid; objects
     kitchen layout has mutated!            cannot deform or splash realistically.
   ===> Geometry & Memory Collapses       ===> Dynamics & Photorealism Fail
          |                                               |
          +-------------------+---------------------------+
                              |
                              v
   AEON SOLUTION: GENERATE EXPLICIT 4D GRAPHICS IN VIDEO LATENTS
   - Diffusion backbone operates in standard continuous Video Latent Space
   - Domain Adapter bridges generative video latents to 4D Reconstructive Latents
   - Decodes directly into 4D Gaussian Splatting (4DGS) primitives!
   ===> Photorealistic Dynamic Motion + Persistent 3D Spatiotemporal Truth!

The Increment

One sentence: AEON establishes the first unified world modeling framework that operates within video diffusion latent spaces while decoding explicit, renderable 4D Gaussian Splatting (4DGS) primitives—merging the visual fidelity of generative video with the strict geometric persistence of 4D spatiotemporal graphics.

Core Mechanism

AEON bridges generative 2D diffusion latents and explicit 4D spatial representations through a four-stage architectural synthesis:

1. 4D Gaussian Splatting (4DGS) as the Universal Representation:
   • Rather than outputting raw RGB frames, AEON represents the evolving world as a dynamic cloud of 4D Gaussians parameterized by spatial position $\boldsymbol{\mu}(t)$, covariance $\boldsymbol{\Sigma}(t)$, opacity $\alpha(t)$, and spherical harmonic appearance coefficients $\mathbf{c}(t)$.
2. Monocular 4D Reconstruction Pre-training:
   • The authors train an expressive 4D reconstruction backbone on vast video corpora, learning to infer both camera extrinsics and dynamic 4DGS primitives directly from monocular video sequences. This network serves as the structural foundation for the 4D decoder.
3. Generative-to-Reconstructive Domain Adaptation:
   • Standard generative video VAEs (e.g., Wan or CogVideo VAEs) encode visual aesthetics, while reconstructive VAEs preserve 3D geometric parallax.
   • AEON introduces an asymmetric Domain Adapter: a lightweight neural bridge trained via knowledge distillation on the video VAE that translates generative diffusion latents into reconstructive 4D latents without losing fine-grained visual details.
4. Closed-Loop Action-Conditioned 4D Simulation:
   • The diffusion backbone predicts future world states conditioned on commanded robot trajectories.
   • The decoded 4DGS can be rendered into arbitrary virtual camera viewpoints at any framerate, enabling downstream autonomous agents to inspect the simulated world from novel, unobserved angles with zero multi-view flickering or temporal hallucination.

   AEON UNIFIED 4D WORLD MODELING PIPELINE

   Text Instruction / Commanded Action + Initial Frame
                           |
                           v
   +-----------------------------------------------------------+
   | Video Diffusion Backbone in Continuous Video Latent Space |
   | Generates temporally cohesive latent trajectory z_{1..T}   |
   +-----------------------------------------------------------+
                           |
                           v
   +-----------------------------------------------------------+
   | Generative-to-Reconstructive Domain Adapter               |
   | (Distilled cross-attention translation bridge)            |
   +-----------------------------------------------------------+
                           |
                           v
   +-----------------------------------------------------------+
   | Explicit 4D Gaussian Splatting (4DGS) Decoder             |
   | Emits: Dynamic Gaussians with continuous time trajectories|
   +-----------------------------------------------------------+
                           |
         +-----------------+-----------------+
         |                                   |
         v                                   v
   [Persistent 4D Environment]     [Arbitrary Camera Rendering]
   Can circle back 100 steps later;Downstream robot policy tests
   Geometry is strictly preserved! novel camera views at 60 FPS!

Key Concepts

• 4D Gaussian Splatting (4DGS): An explicit scene representation that extends 3D Gaussians across the temporal dimension, enabling real-time differential rendering of dynamic, deformable environments.
• Generative-to-Reconstructive Domain Gap: The mathematical discrepancy between latent representations optimized for generative image fidelity versus those optimized for metric multi-view parallax.
• Geometric Persistence: The property of a world simulator where objects, surfaces, and rooms maintain identical physical locations and properties when revisited over extended timelines.

Framework Shift

Before (The False Dichotomy of World Models):
  Pathway 1: 2D Video Diffusion (Sora / Kling) -> Great visuals, zero geometric memory.
  Pathway 2: 3D Scene Generation (Marble / 3DGS) -> Great geometry, static plastic motion.
  Result: Neither could support embodied agent navigation in dynamic worlds.

After (AEON Unified 4D Synthesis):
  Architecture: Video diffusion latent engine mapped directly to explicit 4DGS primitives.
  Fidelity: Inherits the photorealistic texture and non-rigid physical dynamics of video models.
  Persistence: Supports free-viewpoint rendering with zero multi-view drift or halluncinations.

Expert Assessment

Problem choice: Foundational milestone for physical AI and simulation. Unifying generative video models with 3D/4D spatial graphics is the consensus grand challenge of modern computer vision.

Method maturity: The domain adaptation distillation bridge between generative video VAEs and reconstructive 4DGS decoders is exceptionally clever, avoiding the catastrophic forgetting of fine visual details.

Experimental integrity: Validated across rigorous multi-view consistency benchmarks, dynamic scene rendering tasks, and embodied navigation simulators, achieving clear state-of-the-art results across both visual fidelity and geometric precision.

Writing quality: Clear, elegant, and backed by comprehensive interactive project demos that showcase persistent 3D exploration in dynamic scenes.

Verdict: strong accept — A groundbreaking paper that bridges generative video and 3D graphics, establishing the standard architecture for next-generation physical world models.

Takeaways

• Do not settle for pure 2D video generation in robotics simulation; spatial consistency requires explicit 4D geometric primitives.
• Bridge the generative-reconstructive gap using distilled domain adapters rather than retraining massive diffusion backbones from scratch.
• Deploy 4D Gaussian Splatting as the explicit renderable output layer for world models to unlock real-time, multi-view closed-loop agent navigation.