SimuScene: Physics-in-the-Loop 3D Scene Reconstruction from Single Images

Paper: 2606.03994 Authors: Inhee Lee, Sangwon Baik, Sungjoo Kim, Hyeonwoo Kim, Hyunsoo Cha, Hanbyul Joo Categories: cs.CV, cs.RO

The Gap

Single-image 3D reconstruction has made strides in recovering individual object shapes. When you compose those objects into a scene, though, everything falls apart — literally. Objects interpenetrate, hover mid-air, or sink through tables when you drop them into a physics simulator. The root cause: existing methods treat physics as an afterthought, a post-processing step to nudge layouts into place after shapes are already frozen.

Prior work splits cleanly into two camps. Geometry-first methods (like those using neural radiance fields or single-view shape predictors) prioritize visual plausibility but ignore physical constraints during reconstruction. Physics-aware methods apply gravity settling or collision resolution after the fact, but by then the underlying geometry errors — a table that’s too thin, a mug bottom that’s concave — are baked in. You can shuffle objects around, but you can’t fix shapes that were never physically grounded to begin with.

Problem: Single-image 3D scenes collapse in simulation
   |
   v
Existing split:
   Geometry-first ----------> Beautiful but unstable
   Physics post-hoc ---------> Layout fixes, geometry errors remain
   |
   v
SimuScene assumption: Physics is a measurement tool, not a corrector
   |
   v
Method: Diagnostic simulation during reconstruction
   - Drop objects under gravity
   - Measure penetration/support failures
   - Feed back as correction signals
   |
   v
Evidence: Stability benchmarks + manipulation tasks
   |
   v
Conclusion: Physics-in-the-loop produces simulation-ready scenes

The Increment

One sentence: Before this paper, physics cleaned up bad geometry; after it, physics guides geometry creation.

Core Mechanism

SimuScene is a feedback loop wrapped around a compositional 3D reconstruction pipeline. Start with a single RGB image. Use off-the-shelf detectors to segment objects and predict initial depth. Lift each object into 3D using a neural shape predictor, then compose them into a scene. Here’s where the departure happens: instead of declaring victory, SimuScene drops the reconstructed objects into a physics simulator under gravity and watches what breaks.

The simulator becomes a diagnostic instrument. Objects interpenetrating? The contact forces tell you where geometry overlaps. Object hovering? The gap distance quantifies the support failure. Object sinking? The penetration depth reveals where the supporting surface is too weak or misaligned. These aren’t just error flags — they’re structured correction signals with magnitudes and directions.

SimuScene closes the loop by converting these signals into shape and layout adjustments. For gravity-axis errors (sinking, hovering), it stretches or compresses object geometry along the vertical direction. For occlusion-related shape errors, it resamples the amodal (hidden) portions of objects using the shape prior but conditioned on the measured collision feedback. Iterate this diagnose-correct cycle until the scene stabilizes under simulation.

Single Image
    |
    v
[Segment + Depth] --> Object masks + initial 3D layout
    |
    v
[Lift to 3D] --> Per-object meshes (neural shape predictor)
    |
    v
[Compose Scene] --> Initial 3D scene
    |
    v
[Diagnostic Simulation] <--+
    |                       |
    | (drop under gravity)  |
    v                       |
[Measure Failures]          |
  - Penetration depth       |
  - Support gaps            |
  - Contact forces          |
    |                       |
    v                       |
[Generate Corrections] -----+
  - Gravity-axis stretch
  - Amodal shape resample
    |
    v
Stable, simulation-ready scene

Think of SimuScene as a sculptor working with clay under a heat lamp. Traditional methods sculpt the entire piece, let it dry, then try to glue the hardened parts together — cracks and gaps are inevitable. SimuScene keeps the clay soft. After each sculpting pass, the heat lamp (physics simulator) reveals where stress concentrates. The sculptor responds by adding material where gaps form and smoothing where pieces press too hard. The lamp isn’t just checking the final result; it’s part of the sculpting process itself, guiding each adjustment before anything hardens. The lamp’s feedback — heat distribution, shadow patterns — directly informs where the next sculpt happens.

Key Concepts

• Diagnostic Simulation: Most systems use physics engines to check if a scene works (pass/fail). SimuScene uses the engine as a measuring device. When you drop objects under gravity, the simulator computes contact forces, penetration depths, and support gaps at every collision point. These aren’t binary flags; they’re continuous measurements with spatial structure. A table leg sinking 2cm into the floor at coordinates (x, y, z) with normal vector n is actionable feedback. SimuScene reads these measurements like a doctor reads lab results — each number suggests a specific intervention. This shifts physics from validator to instrument.

• Gravity-Axis Stretching: When an object hovers 3cm above a table, you could move it down. But if the table’s geometric thickness is wrong, the object might then penetrate. SimuScene stretches the supporting object along the gravity axis (vertical) instead. If the gap is 3cm, extend the table upward by 3cm. If penetration is 2cm, compress downward by 2cm. Why does this work? Most household support structures (tables, shelves, floors) have their primary geometric variation in the horizontal plane; vertical thickness is relatively constant and less visually critical. Stretching along gravity preserves the visible horizontal structure while correcting the invisible vertical error.

• Amodal Shape Resampling: When objects occlude each other in the input image, the shape predictor guesses the hidden parts. These guesses are often wrong in ways that cause collisions. Rather than editing the visible geometry (which must match the image), SimuScene resamples only the amodal (occluded) regions. It conditions the shape prior on the measured collision feedback: “regenerate the back half of this mug, but ensure it doesn’t penetrate the table at these contact points.” This preserves what the camera saw while fixing what the camera couldn’t see.

Framework Shift

Before (geometry-first + physics post-hoc):

Image --> [3D Lift] --> Objects --> [Compose] --> Scene
                                        |
                                        v
                                  [Physics Check]
                                        | (fails)
                                        v
                                  [Nudge Layouts]
                                        |
                                        v
                                  Stable but geometrically wrong


After (SimuScene, physics-in-the-loop):

Image --> [3D Lift] --> Objects --> [Compose] --> Scene
                            ^            |            |
                            |            v            |
                            |      [Diagnostic        |
                            |       Simulation] <-----+
                            |            |
                            +---- [Corrections]
                                  (shape + layout)
                                        |
                                        v
                                  Stable + geometrically grounded

From post-hoc validation to in-process guidance, the core shift is using physics as a generative constraint rather than a corrective filter.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Robotics labs genuinely struggle with this: you can’t test grasping or manipulation policies in simulation if the reconstructed scenes collapse when you touch them. The timing is right — single-image 3D reconstruction has matured enough that the bottleneck has shifted from “can we get any 3D?” to “can we get physically valid 3D?”

Method maturity: The insight is elegant: physics simulation is differentiable and structured, so use it as a sensor. The execution, though, leans toward engineering integration rather than algorithmic novelty. Gravity-axis stretching is a heuristic (works for tables, less clear for curved surfaces). Amodal resampling depends heavily on the quality of the underlying shape prior. There’s a simpler baseline they don’t explore: direct optimization with physics loss terms from the start, rather than iterative correction. That said, the iterative approach is more modular and easier to debug.

Experimental integrity: The baselines are fair — they compare against recent single-image reconstructors plus physics post-processing. The stability metrics (penetration volume, support ratio) are well-defined. Red flag: the manipulation task results are compelling but lack ablations. How much of the success comes from physics-in-the-loop vs. just having better amodal completion? The paper would be stronger with a controlled ablation isolating each feedback mechanism’s contribution.

Writing quality: Section 3 (Method) front-loads implementation details before building intuition. If they moved the diagnostic simulation intuition forward and the algorithm pseudocode to an appendix, the core idea would land faster. The related work section undersells how much prior physics-aware work exists in robotics — they frame this as more novel than it is.

Verdict: Weak accept — solves a real problem with a clean idea, but the method is more integration than innovation, and the ablations don’t fully isolate the contributions.

Takeaways

Iterative measurement for generative tasks: When your output must satisfy constraints (physics, geometry, feasibility), don’t just check at the end. Build a measurement loop that quantifies violations and feeds them back as generation signals. This pattern transfers: use compiler errors to guide code generation, use runtime traces to guide program synthesis, use user interaction logs to guide UI generation.

Decompose corrections by visibility: SimuScene only edits amodal (hidden) geometry, preserving what the input image shows. Generalize this: when refining generated content, lock down the observable parts and iterate only on the unobservable. In video generation, fix the first frame and refine temporal consistency. In code refactoring, lock the API surface and optimize internals.

Heuristics with geometric priors: Gravity-axis stretching works because vertical thickness matters less than horizontal structure for furniture. Know your domain’s geometric priors. In typography, vertical spacing is more flexible than horizontal. In architecture, floor heights are more negotiable than room widths. Use these asymmetries to allocate your error budget.