REST3D: From Floating Objects to Stable Scenes — Physics-Aware 3D Reconstruction

Paper: 2605.30338 Authors: Xiaoxuan Ma, Jiashun Wang, Nicolas Ugrinovic, Yehonathan Litman, Kris Kitani Categories: cs.CV

The Gap

Existing single-image 3D reconstruction methods (NeRF-based, image-to-3D models) optimize for visual plausibility — they make scenes that look right. But when you drop these reconstructed scenes into a physics simulator, objects float in mid-air or sink through tables. The geometry is plausible but physically inconsistent.

Image-conditioned generation methods (like those using diffusion priors) improve physical plausibility by learning what stable scenes look like, but they sacrifice accuracy — they give you a plausible arrangement, not the arrangement shown in the input image. A chair might be near the table instead of behind it.

The gap: no method reconstructs scenes that are both visually faithful to the input image and physically stable enough to run in simulation without objects flying apart or collapsing.

Problem: Visual reconstruction vs Physical stability
   |
   v
Observation: Existing methods pick one
   |
   +---> Image-to-3D: Looks right, physics broken (floating, penetration)
   |
   +---> Generation: Physics plausible, image match broken (wrong positions)
   |
   v
Assumption: Need explicit physical structure understanding
   |
   v
Method: Scene-tree (gravity+support) + Physics-constrained optimization
   |
   v
Evidence: Reduced penetration/floating, stable simulation, preserved visual match
   |
   v
Conclusion: Physical structure as prior enables both goals

The Increment

One sentence: Before REST3D, you chose between visually accurate but physically broken reconstructions or physically plausible but inaccurate arrangements; after REST3D, you get scenes that match the input image and survive physics simulation.

Core Mechanism

REST3D has three stages. First, it builds a scene-tree — a hierarchical representation where each object node stores its physical state (grounded, supported, floating) and relationships (what supports what). An agentic system analyzes the input image using vision-language models to construct this tree, reasoning about gravity and support from the ground up.

Second, it initializes geometry using off-the-shelf image-to-3D models for each object, then aligns them according to the scene-tree structure. Objects are positioned to respect support relationships — a book goes on the table, not floating beside it.

Third, it runs physics-constrained optimization. A differentiable physics simulator detects violations (penetration depth, unsupported objects). The optimizer adjusts object poses and shapes to minimize physical errors while keeping a visual consistency loss that penalizes deviation from the input image. The scene-tree guides which adjustments are valid — you can’t move a supported object without considering its supporter.

Input Image
    |
    v
[Scene-Tree Construction]  <--- VLM reasoning about gravity/support
    |
    | (tree structure: ground -> table -> book)
    v
[Geometry Initialization]  <--- Image-to-3D per object
    |
    | (rough 3D meshes, tree-guided placement)
    v
[Physics-Constrained Opt]  <--- Differentiable physics + visual loss
    |
    | (iterative: detect violations -> adjust -> re-simulate)
    v
Stable 3D Scene

Think of REST3D as building a house of cards with blueprints. Traditional methods are like stacking cards by eye — they look balanced in the photo, but touch them and they collapse. Generation methods are like having someone else build a stable card house for you, but they didn’t look at your photo, so the arrangement is different.

REST3D first draws blueprints (scene-tree): “This card rests on these two, which rest on the table.” Then it places rough cards (initialization) following the blueprint. Finally, it carefully adjusts each card’s position and angle (optimization) so the structure is stable and matches the photo. The blueprint prevents nonsensical adjustments — you can’t slide the top card without checking if its supports can still hold it.

Key Concepts

• Scene-tree from gravity-support perspective: Instead of treating a scene as a flat collection of objects, REST3D organizes it as a tree rooted at the ground. Each node is an object; edges represent “X supports Y” relationships. An object’s physical state (grounded, supported, floating) is determined by tracing its path to the root. This structure encodes physical constraints: if you move a table, everything it supports must move too. The tree is built by an agentic system that uses vision-language models to reason about the image: “Is the lamp on the table or the floor? What would happen if we removed the table?” This gives the reconstruction a structural prior that pure geometry-based methods lack.

• Physics-constrained optimization with visual consistency: The optimizer has two competing objectives. The physics loss penalizes penetration (objects overlapping), floating (objects with no support), and instability (high potential energy configurations). The visual loss penalizes deviation from the input image by comparing rendered views. The trick is balancing them: too much physics correction and the scene no longer matches the image; too much visual fidelity and objects sink through floors. The scene-tree helps by defining valid adjustment spaces — you can rotate a book on a table, but you can’t move it far from the table without breaking support. The optimization runs in a differentiable physics simulator, so gradients flow from physical violations back to object poses and shapes.

Framework Shift

Before (mainstream approach):        After (REST3D):

Image --> [Image-to-3D] --> Meshes   Image --> [Scene-Tree] --> Structure
              |                                      |
              v                                      v
         Render Loss                    [Init Geometry] <-- Tree-guided
              |                                      |
              v                                      v
         Visual Match                   [Physics + Visual Opt]
         (but unstable)                             |
                                                    v
                                         Stable + Accurate Scene

Key difference:                          Key difference:
Optimize geometry for                    Optimize geometry for
appearance only                          appearance AND physics
                                        with structural prior

From appearance-driven reconstruction to structure-aware reconstruction, the core shift is making physical relationships explicit before optimizing geometry.

Expert Assessment

Problem choice: Real gap. The simulation-readiness problem is not manufactured — VR, robotics, and content creation all need scenes that don’t explode when you add physics. The gap between visual plausibility and physical stability has been noted but not systematically addressed. This sits at the intersection of reconstruction and simulation, a growing area as digital twins and embodied AI gain traction.

Method maturity: The scene-tree idea is clever and well-motivated. Using VLMs for physical reasoning is a natural fit given their spatial understanding capabilities. The physics-constrained optimization is not novel in isolation (differentiable physics has been used before), but the integration with scene-tree guidance is thoughtful. One concern: the agentic scene-tree construction could be brittle — VLMs hallucinate, and errors in the tree propagate. The paper would benefit from ablating tree construction quality vs final stability.

Experimental integrity: Baselines are fair (NeRF methods, image-to-3D models, generation methods). Metrics cover both visual quality (PSNR, LPIPS) and physical stability (penetration volume, floating ratio, simulation success rate). The synthetic dataset (Structured3D) provides ground truth for quantitative evaluation; real-world images (ScanNet, casual photos) test generalization. One red flag: no comparison with methods that do post-hoc physics correction (e.g., running optimization without the scene-tree prior). This would isolate the contribution of the structural prior vs just adding physics loss.

Writing quality: The paper is well-structured, but the scene-tree construction section is dense and could use a worked example. The optimization section glosses over hyperparameter choices (how to weight physics vs visual loss?). The VR demo is compelling but feels like an afterthought — integrating it into the main narrative would strengthen the motivation. The related work section is thorough but could better position the work relative to recent neural simulation methods.

Verdict: weak accept — Addresses a real problem with a principled approach, but the contribution feels incremental (combining existing pieces) rather than introducing a fundamentally new technique. The scene-tree prior is the main novelty, and its robustness needs more scrutiny.

Takeaways

Structural priors beat post-hoc fixes: If you’re building a system that needs to satisfy multiple constraints (visual, physical, semantic), encode the constraints as structure *before optimization rather than adding penalty terms afterward. The scene-tree makes certain violations impossible by construction (you can’t place an object without a supporter), which is more robust than penalizing violations during optimization.

Agentic reasoning for structured outputs: Using VLMs to construct intermediate structured representations (like the scene-tree) is a pattern worth stealing. Instead of end-to-end learning, break the problem into “understand structure” (VLM) and “optimize geometry” (differentiable rendering + physics). This makes the system more interpretable and easier to debug.

Differentiable physics as a regularizer: If your reconstruction needs to be simulation-ready, add a differentiable physics simulator to the optimization loop. The key is balancing physics loss with task-specific loss (visual, semantic, etc.) — too much physics and you lose fidelity, too little and you get instability. The scene-tree provides a middle ground by defining valid adjustment spaces.