WorldSculpt: Carving Interactive 3D Worlds from Video Streams

Paper: 2609.05416 Authors: Muyao Niu, Jixuan He, Ruihan Yu, Lian Fu, Yonghao Yu, Zheng-Hui Huang, Yifan Zhan, Fengbo Lan, Yongtao Ge, Yinqiang Zheng, Kaipeng Zhang, Zhixiang Wang Categories: cs.CV

The Gap

Modern 3D computer vision has achieved photorealistic rendering via Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). However, these breakthrough representations remain fundamentally monolithic: they reconstruct an entire room or landscape as a single continuous field of splats or density voxels. In such a representation, a coffee cup is fused permanently to the desk, and the desk is fused to the floorboards. You can render dazzling walkthrough videos, but you cannot pick up the cup, calculate its physical collisions, or export its mesh into a game engine.

Downstream applications in gaming, spatial computing, virtual reality, and robotics demand compositional 3D representations: every object must be an independent, watertight 3D mesh placed in a shared world coordinate frame.

Achieving this in realistic, cluttered environments has been an intractable dilemma. Traditional geometry-based multi-view stereo or SLAM fails in cluttered rooms where hundreds of objects occlude one another, leaving behind jagged holes and incomplete back-faces. Meanwhile, modern 3D diffusion generative models generate plausible complete geometry for single objects in isolation, but training a model to generate hundreds of interacting objects simultaneously requires massive scene-level multi-object 3D datasets that simply do not exist.

[INPUT VIDEO] Video of Cluttered Real-World Room (Hundreds of overlapping items)
      |
      +-----------------------------+-----------------------------+
      v                                                           v
[MONOLITHIC 3DGS / NeRF]                                   [GEOMETRIC SLAM / MVS]
Continuous soup of splats                                  Fused depth meshes
- Stunning visual rendering                                 - Severe occlusion holes
- Zero physics or segmentation                              - Hidden sides completely missing
- Objects cannot be picked up or moved                      - Broken meshes unusable in games/sims
      |                                                           |
      +-----------------------------+-----------------------------+
                                    v
                 [THE COMPOSITIONAL 3D DILEMMA]
  Need: Hundreds of separate, watertight meshes in shared space
  Problem: No paired scene-level datasets exist to train multi-object 3D diffusion

The Increment

One sentence: Before this paper, converting video of a cluttered room into hundreds of interactive, watertight 3D meshes required manual 3D modeling or failed on occlusions; after it, WorldSculpt demonstrates that a single-object generative prior (Pixal3D) can be adapted via multi-view grounding to decompose cluttered scenes into compositional worlds without requiring any multi-object 3D training data.

Core Mechanism

WorldSculpt solves the multi-object bottleneck through an elegant paradigm shift: instead of attempting to learn scene-level spatial layout from scratch, it adapts a robust single-object 3D generative prior to multi-view posed observations.

The framework operates in three sequential phases:

1. Multi-View Grounding & Tracking: The input video is tracked across frames using 2D foundation models (bounding boxes, masks, and relative camera poses), associating multi-view image snippets for each detected object in the scene.
2. Multi-View Conditioned Single-Object Generation: The authors extend Pixal3D (a state-of-the-art single-object 3D native generative model) with a multi-view conditioning pathway. Even though the backbone was trained purely on isolated canonical objects, the conditioned pathway allows the model to synthesize the complete 3D geometry of heavily occluded objects by hallucinating invisible back-faces conditioned on whatever fragments are visible across camera perspectives.
3. Canonical-to-World Alignment & Composition: Each generated mesh is registered back into the global scene coordinates using spatial pose estimators. To rigorously validate the approach, the authors also introduce UE-MeshyScene, a photorealistic Unreal Engine benchmark containing densely cluttered rooms with ground-truth per-object meshes and extreme occlusion.

   WORLDSCULPT RECONSTRUCTION PIPELINE

   [Input: Video of Cluttered Scene]
                   |
                   v
   [2D Grounding & Multi-View Association]
   (Track 100+ object instances across camera trajectory)
                   |
                   v
   [Multi-View Posed Conditioning: Pixal3D]
   +---------------------------------------------------+
   | Feeds visible multi-view fragments                |
   | Generative prior hallucinates occluded back-faces |
   | Outputs watertight canonical single-object mesh   |
   +---------------------------------------------------+
                   |
                   v
   [Global Coordinate Registration & Placement]
                   |
                   v
   [Interactive Compositional 3D World]
   - Physics-ready meshes for every individual object
   - Converts monolithic 3DGS (Marble, HY-World) into interactive worlds

To explain this breakthrough, consider the structural metaphor of an expert forensic sculptor at an archaeological excavation. Amateurs take a single plaster cast of the entire dig site (the monolithic 3DGS approach), creating a solid rock block where artifacts are permanently fused together. The forensic sculptor carries a mental catalog of how every ancient pot, cup, and tool is shaped in three dimensions (the single-object generative prior). When examining a pile where thirty broken pots overlap, the sculptor glances at the exposed handles and rim curves from three angles, instantly imagines the complete form of each pot, sculpts thirty individual vessels, and places them cleanly on the museum display table.

Key Concepts

• Compositional 3D Representation: A spatial world representation structured as an assembly of independent, manipulable 3D meshes rather than a monolithic continuous density field.
• Zero-Shot Scene Composition: The capability to reconstruct complex scenes containing hundreds of occluded items by leveraging single-object priors without ever training on multi-object 3D ground truth.
• UE-MeshyScene Benchmark: A rigorous, photorealistic 3D evaluation dataset built in Unreal Engine featuring dense clutter, complex contact physics, and ground-truth CAD meshes for all individual components.

Framework Shift

Before (Monolithic / Single-Representation Paradigms):
Video / Photos ---> [3D Gaussian Splatting / NeRF] ---> Monolithic Splat Soup
(Visually pretty, but zero object agency: cannot simulate, edit, or interact)

After (WorldSculpt Compositional Paradigm):
Video ---> [Instance Grounding] ---> [Single-Object Generative Prior] ---> Compositional Meshes
(Full object modularity: every cup, chair, and laptop is an interactive game asset)

From treating a visual scene as a frozen decorative tapestry to decomposing it into an interactive lego set of physical assets, the core shift is realizing that single-object 3D diffusion models already possess the completion power to solve scene-level occlusion.

Expert Assessment

Problem choice: Exceptional. As embodied AI, robotics, and spatial game engines advance, monolithic representations are hitting a dead end. Robotics simulators require individual collision meshes, not just pretty radiance fields.

Method maturity: Avoiding scene-level 3D generative training by grounding a single-object foundation model is a masterclass in pragmatic machine learning design. Single-object 3D data is orders of magnitude more plentiful than full-scene mesh data.

Experimental integrity: The introduction of UE-MeshyScene provides a much-needed standardized benchmark for compositional 3D vision. The demonstrations converting monolithic worlds from Marble and HY-World 2.0 into interactive meshes show immediate practical utility.

Writing quality: Exemplary. The visualizations of extracted object meshes against ground truth under extreme clutter provide undeniable evidence of geometry completion.

Verdict: strong accept — A landmark paper that connects the dots between generative video, single-object 3D diffusion, and spatial simulation environments.

Takeaways

• Monolithic 3D representations (3DGS/NeRF) are insufficient for robotics and spatial computing; downstream interaction requires compositional mesh decomposition.
• You do not need paired scene-level 3D datasets to reconstruct complex scenes; single-object 3D generative priors can hallucinate occluded geometry when conditioned across video frames.
• WorldSculpt provides an immediate bridge to convert video-to-world generations into assets ready for physics engines and robotics simulators.