World Models' Last Exam in Physics

Paper: 2610.08791 Authors: Mingju Gao, Qingle Liu, Yuzhao Peng, Xinjie Lin, Ziming Qin, Zheng Jiang, Wenyi Li, Calvin Xiao, Youjie Zheng, Kaisen Yang, Qinhuai Na Categories: cs.CV

The Gap

Generative video models (such as Sora, Kling, Gen-3, and Wan) are increasingly celebrated as emergent “world simulators” capable of serving as engines for robotics and embodied policy planning. Their outputs feature photorealistic lighting, coherent geometry, and plausible object textures.

However, visual plausibility is not physical consistency:

1. Superficial evaluations: Existing benchmarks rely on subjective human aesthetic scoring or prompt-based Vision-Language Model (VLM) judges. These evaluators are easily fooled by high-frequency texture fidelity and cinematic composition.
2. Narrow physical domain: Past physics tests focused almost exclusively on basic rigid-body mechanics (e.g., dropping balls or sliding blocks on inclined planes), completely ignoring optics, thermodynamics, fluids, and electromagnetism.
3. Reference dependence: Traditional metrics (FVD, PSNR) require a ground-truth reference video, making it impossible to evaluate open-ended forward simulations under novel initial conditions.

   PROBLEM: PHOTOREALISM MASKS CATASTROPHIC PHYSICAL HALLUCINATIONS

   Video Generation Prompt: "Laser enters prism" / "Ice melts in hot oil"
                            |
                            v
   State-of-the-Art Video World Model (Sora / Kling / Gen-3)
                            |
                            v
   Visually Gorgeous Output (Cinematic lighting, smooth pixels)
                            |
         +------------------+------------------+
         |                                     |
         v                                     v
   Subjective Human / VLM Judge           Rigorous Quantitative Physics
   "Looks real! Grade: 95/100"            Snell's Law: Refraction angle violated!
                                          Thermodynamics: Heat flows backwards!
                                          Fluid Mechanics: Volume not conserved!
                            |
                            v
   METHOD: WORLD MODELS' LAST EXAM IN PHYSICS (40 Controlled Tasks)
   Optics, Fluids, Thermal/Phase Change, Electromagnetism, Surface Tension, Mechanics
   Zero reference videos required; quantitative measurement modules
                            |
                            v
   EVIDENCE: Top frontier model scores only 57.76 / 100 across 1,280 rollouts!
                            |
                            v
   CONCLUSION: Current video models simulate movie sets, not physical laws

The Increment

One sentence: Establishing a measurement-grounded physical consistency benchmark of 40 controlled tasks spanning six fundamental branches of physics, this paper reveals that leading video generative models achieve an overall accuracy of only 57.76 out of 100, proving that current systems excel at pixel texture interpolation while failing to encode governing conservation laws.

Core Mechanism

The authors develop an automated, reference-free evaluation suite grounded in actual physical observables:

1. Six Physical Domains (40 Tasks):
   • Optics: Snell’s law refraction through prisms, concave mirror ray convergence, shadow casting from multiple point sources.
   • Fluids: Viscous fluid pouring, Archimedes’ buoyant displacement, vortex shedding.
   • Thermal & Phase Transitions: Melting latent heat, thermal expansion, convection current formation.
   • Electromagnetism & Magnetostatics: Magnetic pole attraction/repulsion, Lenz’s law eddy current braking.
   • Surface Tension: Capillary rise in narrow tubes, droplet coalescence on hydrophobic surfaces.
   • Classical Mechanics: Inelastic collisions, rolling without slipping, pendulum harmonic oscillation.
2. Measurement-Grounded Evaluator:
   • Each task pairs a standardized initial condition frame with a physical trigger prompt.
   • Rather than asking an LLM “does this look realistic?”, the benchmark runs computer vision measurement routines: tracking ray angle vectors, measuring fluid meniscus elevation over time, and computing conservation of mass and momentum.
   • Includes a task-observability screening module to verify that the generated video actually captures the region of interest before taking measurements.
3. Experimental Disclosures:
   • Tested across eight frontier video generation models over 1,280 generations.
   • The top-performing commercial system reached an overall score of only 57.76 / 100.
   • Performance drops drastically outside rigid-body mechanics: thermal phase change and electromagnetism scored below 35/100 across all models, with models regularly producing fluids that vanish into thin air or light beams that bend in random directions.

   MEASUREMENT PIPELINE FOR PHYSICAL BENCHMARKING

   Initial State Image + Controlled Physical Scenario Prompt
                            |
                            v
             Video Generation Under Test
                            |
                            v
   +------------------------------------------------+
   | Task-Observability Verification Screen         |
   | (Confirms relevant physical interface in view) |
   +------------------------------------------------+
                            |
                            v
   +------------------------------------------------+
   | Quantitative Domain-Specific Measurement       |
   | - Ray tracing angle tracking (Optics)          |
   | - Mass & volume conservation contouring (Fluid)|
   | - Dynamic meniscus height measurement (Surface)|
   +------------------------------------------------+
                            |
                            v
   Physical Consistency Vector -> Score: 57.76 / 100

The load-bearing structural metaphor is a Hollywood visual effects artist versus a licensed structural engineering inspector.

• Current video world models are Hollywood VFX artists: given a scene of a suspension bridge during a storm, they render photorealistic churning ocean waves, cinematic lightning, and dramatic swinging cables. The audience in the cinema is awestruck.
• But when the structural engineer attaches laser strain gauges and accelerometers to the bridge’s suspension cables (the measurement module in this benchmark), they find that the steel cables stretch like rubber bands, tension is negative, and the bridge towers violate Newton’s third law. The film looks stunning on a movie poster, but if a robot steps onto that digital bridge to deliver a payload, the bridge evaporates underneath it.

Key Concepts

• Measurement-Based Grounding: Benchmarking generative models by extracting quantitative physical coordinates and state variables (angles, velocities, temperatures) rather than prompting neural network judges with natural language adjectives.
• Reference-Free Evaluation: Assessing whether a generated trajectory respects conservation laws ($E, p, m$) without needing a pre-recorded ground-truth physical recording for comparison.
• Physical Hallucination: The generation of visual textures that mimic familiar materials while violating the governing partial differential equations of the underlying system.

Framework Shift

Before (Visual & VLM-Judged World Model Evaluations):
  Input: "Pour water into cup" -> Model: Generates liquid-like pixels.
  Judge: GPT-4V looks at 4 frames -> "High visual quality, 9/10."
  Reality: Total water volume doubled in mid-air; gravity was ignored.

After (World Models' Last Exam in Physics):
  Input: Standardized initial geometry + physical scenario.
  Evaluator: Extracts physical state vectors (fluid volume, refraction angle, acceleration).
  Score: Best model scores 57.76 / 100.

From “admiring how pretty video world models look,” the core shift is that a model cannot be trusted as an embodied world simulator until its generative rollouts survive quantitative physical measurement across non-mechanical domains.

Expert Assessment

Problem choice: Crucial and timely. Generative video labs frequently market their models as “physical simulators” for training autonomous robots and autonomous vehicles. This paper puts an empirical stop to unjustified marketing claims.

Method maturity: The decision to build specialized computer vision measurement modules (measuring ray angles, meniscus height, conservation metrics) rather than relying on LLM/VLM subjective vibes is the correct methodological path.

Experimental integrity: Tested across 8 state-of-the-art models on 1,280 controlled videos. The validation on synthetic physical engine videos proves that the measurement module itself does not introduce false rejections.

Writing quality: Exemplary organization, extensive appendices detailing physical instrumentation, and unambiguous experimental findings.

Verdict: strong accept — A defining benchmark that establishes an objective, measurable baseline for physical world modeling.

Takeaways

• Do not use raw generative video models as physical simulators or training environments for robotics without rigorous verification; current models do not understand physics beyond surface textures.
• Replace VLM judges with deterministic computer vision measurement probes when testing physics, geometry, or kinematics in video rollouts.
• When selecting video world models for embodied applications, test beyond rigid mechanics: thermal changes, fluids, and optical reflections reveal where neural rendering breaks down.