PhyCo: Teaching Video Models to Respect Physics Through Controllable Material Properties

Paper: 2604.28169 Authors: Sriram Narayanan, Ziyu Jiang, Srinivasa Narasimhan, Manmohan Chandraker Categories: cs.CV, cs.AI, cs.LG

The Gap

Video diffusion models like Sora and Gen-2 can generate visually stunning footage, but they’re physically illiterate. A rubber ball bounces like a rock. Objects pass through each other. Friction seems to be a random number generator. The models learned appearance correlations from internet videos, but never learned the causal structure underneath—that material properties determine motion.

Prior work tried two paths: (1) post-hoc physics simulation to correct generated videos (expensive, requires geometry reconstruction), or (2) training on real videos with weak physics annotations (noisy labels, limited diversity). Neither gives you controllable physics at generation time. You can’t say “make this ball bouncier” and have the model understand what that means in terms of motion trajectories.

Problem: Video models ignore physics
    |
    v
Assumption: Physical properties (friction, elasticity) 
            can be learned as continuous control signals
    |
    v
Method: Train on synthetic data with ground-truth 
        physics labels + VLM reward feedback
    |
    v
Evidence: Physics-IQ benchmark + human studies show
          improved realism and controllability
    |
    v
Conclusion: Scalable path to physically consistent
            video generation without runtime simulation

The Increment

One sentence: Before PhyCo, you could generate videos that look right but move wrong; after PhyCo, you can control how objects move by specifying their material properties, and the motion will be physically plausible.

Core Mechanism

PhyCo has three components working in sequence. First, a synthetic data engine generates 100K+ videos using physics simulators (MuJoCo, PyBullet), systematically varying friction (0.1 to 0.9), restitution (0.0 to 1.0), deformation stiffness, and applied forces across diverse scenarios—balls rolling, blocks sliding, cloth draping. Each video comes with pixel-aligned property maps showing which regions have which physical attributes.

Second, a ControlNet architecture extends a pretrained video diffusion model (CogVideoX). The ControlNet takes these property maps as input and learns to condition the denoising process on physical attributes. During training, the model sees both the visual frames and the property maps, learning the correlation between material parameters and motion patterns. The key insight: treat physics as a spatial control signal, like depth or pose, rather than a global scene descriptor.

Third, a vision-language model (InternVL2) is fine-tuned to evaluate physical realism. Given a generated video and a physics query (“Does the ball’s bounce height match its restitution?”), the VLM outputs a score. This score becomes a differentiable reward signal for further optimization via Direct Preference Optimization (DPO). The VLM acts as a physics critic, pushing the model toward outputs that satisfy physical constraints even when the property maps are imperfect or missing.

Synthetic Data Engine          ControlNet Training         VLM Reward Optimization
                                                          
[Physics Simulator]            [Pretrained Diffusion]     [Fine-tuned VLM]
       |                              |                          |
       v                              v                          v
[100K videos with]  -------->  [+ Property Map]  ------>  [Physics Query]
[property labels]              [  Conditioning  ]         [   Scoring   ]
       |                              |                          |
       v                              v                          v
[Friction: 0.5  ]              [Denoising with]           [Reward Signal]
[Restitution: 0.8]             [physics control]          [for DPO fine-tuning]

Think of PhyCo as a cooking school for video models. The synthetic data is the recipe book—thousands of dishes (videos) with exact ingredient lists (physics parameters). The ControlNet is the student chef learning to associate ingredients with outcomes: more salt (higher friction) means slower sliding, more baking powder (higher restitution) means higher bounce. The VLM is the head chef tasting the dishes and giving feedback: “This bounce is too weak for that restitution value, try again.” Over time, the student internalizes the rules and can cook new dishes (generate new videos) that taste right (move correctly) even without the recipe in hand.

Key Concepts

• Property Maps as Spatial Control: Instead of describing physics globally (“this scene has high friction”), PhyCo uses pixel-aligned maps where each spatial location has its own friction, restitution, and stiffness values. This is like giving the model a material texture map—rubber here, ice there—so it knows different parts of the scene should move differently. The model learns to read these maps during denoising, adjusting motion predictions region by region. At inference, you can paint properties onto objects like Photoshop layers, and the model generates motion accordingly.

• VLM as Differentiable Physics Critic: Traditional physics evaluation requires simulators or hand-crafted metrics. PhyCo fine-tunes a vision-language model to answer targeted physics questions: “Is the collision response realistic?” “Does deformation match stiffness?” The VLM’s text output is converted to a numerical score, which flows back as a gradient signal during DPO training. This closes the loop: the generative model gets continuous feedback on whether its outputs satisfy physical constraints, without needing explicit simulation. The VLM essentially learns to be a soft physics engine that grades plausibility rather than computing exact trajectories.

• Sim-to-Real via Diverse Synthetic Data: The 100K training videos span varied scenarios—different object shapes, materials, lighting, camera angles—all rendered photorealistically. This diversity is crucial: the model learns physics principles (friction slows motion, restitution controls bounce) rather than memorizing specific synthetic scenes. At test time, it generalizes to real-world footage because the underlying physics is invariant, even if the visual domain shifts. The synthetic data acts as a physics curriculum, teaching concepts that transfer beyond the training distribution.

Framework Shift

Before (mainstream approach):        After (PhyCo):

[Internet Videos]                    [Synthetic Physics Data]
       |                                     |
       v                                     v
[Diffusion Model]                    [Diffusion + ControlNet]
       |                                     |
       v                                     v
[Generate Video]                     [Property Maps] ---> [Generate Video]
       |                                     |
       v                                     v
[Looks good,                         [Looks good,
 moves wrong]                         moves right]
                                            |
                                            v
                                     [VLM Physics Critic]
                                            |
                                            v
                                     [Reward Feedback Loop]

From learning appearance correlations to learning causal physics, the core shift is treating material properties as first-class control inputs rather than implicit scene attributes.

Expert Assessment

Problem choice: Real gap. Video generation has hit a wall where visual quality is high but physical plausibility is a coin flip. This matters for applications like robotics simulation, VFX previsualization, and scientific visualization where physics correctness is non-negotiable. The problem sits at the intersection of graphics and learning—a natural next frontier after appearance synthesis.

Method maturity: Clever synthesis of existing techniques (ControlNet, DPO, VLM fine-tuning) rather than a novel algorithmic insight. The innovation is in the data pipeline and the VLM-as-critic idea. One concern: the approach is data-hungry and compute-intensive (100K synthetic videos, multiple training stages). Could a smaller, more targeted dataset with better physics coverage work? The paper doesn’t explore the data efficiency frontier. Also, the VLM reward optimization feels like a patch for imperfect ControlNet training—ideally, the property maps alone should suffice.

Experimental integrity: Baselines are fair (CogVideoX, Sora, Gen-2), and the Physics-IQ benchmark is a solid contribution—finally, a standardized eval for physical realism. Human studies confirm the quantitative gains. However, the paper leans heavily on synthetic test cases. Real-world generalization is shown qualitatively but not rigorously measured. How does it handle complex real scenes with unknown materials? The ablations are thorough (each component contributes), but I’d like to see failure cases and boundary conditions more explicitly discussed.

Writing quality: The method section is dense and could use a clearer narrative arc. The three components (data, ControlNet, VLM) are presented sequentially, but their interdependencies aren’t immediately obvious. A unified system diagram upfront would help. The related work section is comprehensive but reads like a literature dump—tighter framing around what specifically failed in prior work would strengthen the motivation. The results section is strong, with good visual comparisons and ablations.

Verdict: weak accept — Solid execution on an important problem with a practical path forward, but the method feels like an engineering solution rather than a conceptual breakthrough. The VLM reward loop is interesting but underexplored. Would benefit from deeper analysis of what the model actually learns about physics.

Takeaways

Steal the VLM-as-critic pattern: Fine-tuning a vision-language model to evaluate domain-specific properties (physics, aesthetics, safety) and using its scores as differentiable rewards is broadly applicable. This sidesteps the need for hand-crafted metrics or expensive oracles. You could apply this to any generative task where quality is hard to specify but easy to judge—architectural design, UI layouts, molecular structures.

Spatial control via property maps: Treating control signals as pixel-aligned maps rather than global descriptors gives you fine-grained manipulation. This generalizes beyond physics—imagine emotion maps for character animation, style maps for image generation, or attention maps for text layout. The key is making the control signal spatially explicit so the model can learn local dependencies.

Synthetic data as a physics curriculum: When real data lacks ground-truth labels, synthetic data with systematic parameter sweeps can teach causal relationships. The diversity of scenarios matters more than photorealism—the model needs to see the same physical principle (e.g., friction) play out across varied contexts to extract the invariant. This applies to any domain where you can simulate the underlying process: fluid dynamics, material science, economic models.