NewtPhys: Testing Whether Vision Models Actually Understand Physics

Paper: 2606.03986 Authors: Sebastian Cavada, Soumava Paul, Tuan-Hung Vu, Andrei Bursuc, Raoul de Charette Categories: cs.CV

The Gap

Existing physics reasoning benchmarks test vision-language models (VLMs) with high-level questions about synthetic scenes: “Will the tower fall?” or “Which ball reaches first?” These tests measure conceptual understanding but sidestep the hard part—does the model actually perceive forces, momentum, or acceleration from raw pixels? Previous work like CLEVRER uses simplified 2D shapes, while IntPhys employs controlled scenarios that don’t reflect real visual complexity. The gap: we have no way to evaluate whether models understand low-level Newtonian mechanics—3D forces, per-pixel motion, occluded dynamics—when faced with photorealistic scenes.

Problem: VLMs claim physics understanding, but tests are high-level
   |
   v
Assumption: Real understanding requires perceiving fine-grained quantities
   |
   +---> Method: Build 4D dataset with real scenes + simulated physics
   |              (multiview images → 3D reconstruction → physics sim)
   |
   +---> Evidence: Test 56 VLMs + 10 VFMs on dense annotations
   |               (forces, velocities, occlusions, contacts)
   |
   v
Conclusion: Models fail at low-level physics despite high-level competence

The Increment

One sentence: Before NewtPhys, we thought vision models understood physics because they passed synthetic VQA tests; after NewtPhys, we know they lack fine-grained Newtonian reasoning on real-world visuals.

Core Mechanism

NewtPhys constructs physically-grounded ground truth by fusing real-world capture with simulation. First, 50 real scenes are filmed with synchronized multi-camera rigs, capturing objects in motion (falling, rolling, colliding). These multiview images feed into 3D reconstruction pipelines (NeRF-based) to recover geometry and object shapes. Then, a physics simulator (Isaac Sim) re-enacts each scene using the reconstructed 3D assets and material properties, producing per-timestep annotations: 3D forces on every object, amodal per-pixel depth and segmentation, contact points, velocities, accelerations. The dataset spans 4D (3D space + time) with 1000+ timesteps per scene, providing dense supervision that bridges perception and physics.

Real Scene (50 scenarios)
    |
    v
Multiview Capture (6 cameras, sync'd)
    |
    v
3D Reconstruction (NeRF → mesh + materials)
    |
    v
Physics Simulation (Isaac Sim, replays motion)
    |
    +---> Per-frame outputs:
          - 3D forces (vectors per object)
          - Amodal depth/segmentation (pixel-level)
          - Contacts, velocities, accelerations
    |
    v
NewtPhys Dataset (4D annotations)

Think of NewtPhys as a forensic reconstruction lab. The multiview cameras are crime scene photographers capturing evidence from every angle. The 3D reconstruction is the forensic team building a physical model of the scene—measuring dimensions, identifying objects, estimating materials. The physics simulator is the investigator replaying the event in slow motion, calculating forces and momentum at every instant. The final report isn’t “did the vase break?”—it’s a frame-by-frame analysis showing exactly which force vectors acted on each object, how occlusions changed, where contact occurred. When you test a vision model, you’re asking: can it read this forensic report directly from the raw footage?

Key Concepts

• Amodal annotation: Standard annotations label only visible pixels. Amodal annotations include occluded regions—the back of a ball hidden behind a cube still gets labeled with its geometry and velocity. This matters for physics: understanding what’s behind an object is necessary to predict forces and contacts. NewtPhys generates amodal data by rendering from the simulator’s omniscient viewpoint, not the camera’s limited one. Example: A rolling ball half-hidden by a box. Visual models see only the exposed arc, but the physics annotation includes the ball’s full shape, spin, and forces acting on the occluded half.

• 4D dense supervision: Most datasets annotate sparsely—bounding boxes every 10 frames, or single-point answers. NewtPhys annotates densely across both space (every pixel) and time (every timestep). This reveals whether models track continuous dynamics or just recognize static patterns. Dense supervision means if an object accelerates between frame 50 and 51, the annotation captures that acceleration vector. If a contact happens for 0.1 seconds, it’s marked exactly when and where, not averaged out.

• Real-to-sim ground truth: The insight is that real scenes provide visual realism (textures, lighting, clutter) but lack ground-truth physics, while simulators provide perfect physics but look synthetic. NewtPhys fuses both: capture reality, reconstruct it, then simulate physics on the reconstructed assets. The simulator “knows” all forces because it computes them, but the visual input stays photorealistic. This sidesteps the usual tradeoff between realism and annotation accuracy.

Framework Shift

Before (mainstream approach):        After (NewtPhys):

Synthetic Scenes                     Real Multiview Capture
    |                                     |
    v                                     v
Simple Shapes (cubes, balls)         3D Reconstruction (meshes)
    |                                     |
    v                                     v
High-Level VQA                       Physics Simulation (forces)
("Will it fall?")                        |
    |                                     v
    v                                Dense 4D Annotations
Binary/Categorical Answers           (per-pixel, per-timestep)
                                         |
                                         v
                                    Low-Level Physics Eval
                                    (force vectors, contacts)

From testing conceptual physics reasoning to measuring Newtonian quantity perception—the core shift is ground truth depth.

Expert Assessment

Problem choice: Real gap. Vision models dominate high-level tasks but their low-level physics understanding is untested. The field has sidestepped this because annotating real-world physics is hard. This paper solves the annotation problem cleanly, making the evaluation feasible.

Method maturity: Smart hybrid approach. Using reconstruction to bridge real and sim is elegant—you get photorealism without sacrificing annotation accuracy. The weak link: reconstruction quality affects downstream physics. If the mesh is noisy, the simulated forces won’t match reality. Authors acknowledge this but don’t quantify reconstruction error’s impact on annotations.

Experimental integrity: Thorough model sweep (56 VLMs, 10 VFMs) with diverse architectures. Baselines are fair—they test both open and closed models, small and large. The metrics (force prediction error, contact detection F1) directly measure physics understanding. One concern: no human baseline. How well can humans estimate 3D forces from 2D video? Without that anchor, we can’t tell if the task is too hard or models are too weak.

Writing quality: Strong intro and method, but the results section drowns in tables. The key finding—models fail at low-level physics—gets buried in ablations. Section 4.2 should lead with failure modes (e.g., models confuse contact with proximity) before showing numbers. The related work undersells how different this is from CLEVRER/IntPhys; a comparison table would clarify.

Verdict: strong accept — Fills a real evaluation gap with a technically sound dataset and exposes genuine model weaknesses.

Takeaways

Reconstruction-to-simulation pipeline: If you need ground-truth annotations for real-world phenomena that are hard to measure (occlusions, forces, internal states), consider reconstructing the scene and simulating the process. This pattern works beyond physics—imagine reconstructing surgical scenes to simulate tissue deformation, or reconstructing traffic to simulate collision dynamics.

Dense temporal supervision reveals brittleness: Models that pass frame-level tests often fail when asked to track quantities across time. If your domain involves continuous processes (fluid flow, growth, deformation), sparse sampling hides failure modes. NewtPhys’s per-timestep annotations show where models lose the thread.

Amodal reasoning is a sharp filter: Testing whether models understand occluded regions separates true scene understanding from texture matching. If your model needs to generalize to partial observations (robotics, medical imaging), add amodal annotations to your eval set even if training data lacks them.