LaMo: Learning Motion Priors from Unlabeled Video for Physically Realistic Generation

Paper: 2605.23878 Authors: Bo Jiang, Depu Meng, Yihan Hu, Yichen Xie, Tianshuo Xu, Wei Zhan Categories: cs.CV

The Gap

Video diffusion models generate visually impressive clips but fail at physical plausibility—objects float, momentum vanishes, collisions look wrong. Prior fixes bolt on external physics engines (PhysDreamer), distill from teacher models (VideoPoet), or curate physics-specific datasets (Physion). These approaches work but require heavy external machinery. The gap: can we extract motion supervision from the same unlabeled videos already used for training, without adding new data sources or simulators?

Problem: Video generators lack physical consistency
         |
         v
Observation: Training videos contain implicit motion patterns
         |
         v
Assumption: Frame-to-frame latent changes encode motion priors
         |
         v
Method: Model P(latent_change | current_latent, prompt)
        + Use as training loss (macro drift)
        + Use as sampling guidance (micro field)
         |
         v
Evidence: Improves physics benchmarks (VideoPhy/VideoPhy2)
          Preserves general quality (VBench)
         |
         v
Conclusion: Unlabeled video contains usable motion supervision

The Increment

One sentence: Before LaMo, improving video physics meant adding external supervision; after LaMo, you can extract motion priors from the training data itself.

Core Mechanism

LaMo models the distribution of frame-to-frame changes in latent space, conditioned on the current latent and text prompt. This “latent motion prior” has two readouts. First, a macro motion drift predictor outputs a single vector representing expected overall motion direction. During training, this drift vector acts as a regularization target—the model’s actual latent change should align with the predicted drift. Second, a micro motion field predictor outputs a dense spatial field of motion vectors. During sampling, this field guides the denoising process toward physically plausible trajectories.

Both components are lightweight networks trained on the same video data used for the base diffusion model. The drift predictor sees consecutive latent frames and learns to predict their difference. The field predictor learns a spatially-resolved version of the same task. Crucially, neither requires physics labels—they learn purely from observing how real video latents evolve over time.

Training:
  Video frames --> Encoder --> Latent sequence
                                    |
                    +---------------+---------------+
                    |                               |
                    v                               v
            Drift Predictor                 Diffusion Model
            (macro motion)                  (generates video)
                    |                               |
                    v                               v
            Expected drift <---compare---> Actual latent change
                                |
                                v
                        Motion Drift Loss
                        (regularization)

Sampling:
  Noisy latent + prompt --> Diffusion Model --> Predicted noise
                                    ^
                                    |
                            Motion Prior Guidance
                                    |
                                    v
                            Field Predictor
                            (micro motion field)
                            nudges toward physical trajectories

Think of LaMo as a driving instructor sitting beside a student driver (the diffusion model). The instructor doesn’t control the wheel but has two ways to help. First, before each trip (training), they point out the general direction traffic should flow—“cars on this road usually move northeast.” This is the macro drift, a compass heading. Second, during the drive (sampling), they tap the dashboard when the student drifts into the wrong lane—“that truck ahead is moving left, you should account for it.” This is the micro field, a real-time correction based on local motion patterns. The instructor learned both skills not from a driving manual (external physics) but from years of watching traffic (unlabeled video). The student driver improves because the instructor’s corrections are grounded in how real traffic behaves, not abstract rules.

Key Concepts

• Latent Motion Prior: In diffusion models, videos are compressed into latent space before generation. A motion prior is a learned distribution over how these latents should change from frame to frame. Instead of modeling raw pixel motion (expensive, noisy), LaMo models latent-space motion—the compressed representation of motion. This is like learning how chess positions evolve rather than tracking individual piece pixels. The prior is “self-supervised” because it’s trained on frame pairs from real videos without needing labels like “this is a falling object” or “this obeys Newton’s laws.” The model simply learns: given latent A and prompt P, latent B tends to follow in direction D with spatial variations F.

• Macro vs Micro Motion: Macro motion is a single vector summarizing the overall direction of change—think “camera pans right” or “object moves down.” Micro motion is a dense field where each spatial location has its own motion vector—think “top-left corner moves up, bottom-right moves down, center stays still.” LaMo uses macro drift during training as a soft constraint (the model’s generated motion should roughly align with typical motion patterns) and micro field during sampling as guidance (steer the denoising process toward locally consistent motion). The split lets LaMo operate at two scales: coarse regularization during learning, fine-grained correction during generation.

Framework Shift

Before (mainstream approach):        After (LaMo):

External Physics Engine              Training Video Corpus
        |                                    |
        v                                    v
  Simulate dynamics              Extract latent motion patterns
        |                                    |
        v                                    v
  Supervise video model            Self-supervise motion prior
        |                                    |
        +-----> Video Generator              +-----> Video Generator
                                                     (with motion guidance)

Key difference:                      Key difference:
- Requires physics simulator        - Uses existing training data
- Explicit physical rules            - Implicit motion statistics
- External dependency                - Self-contained

One sentence: From explicit physics simulation to implicit motion statistics learned from the training corpus itself.

Expert Assessment

Problem choice: Real gap. Video generators genuinely struggle with physics, and existing fixes are heavyweight. The self-supervised angle is underexplored and practically valuable—if you’re already training on millions of videos, why not extract motion cues from them? Sits at the intersection of video generation and self-supervised learning, both hot areas.

Method maturity: Clever but not revolutionary. The core idea—model latent changes as a prior—is straightforward. The dual readout (drift + field) is sensible engineering. However, the paper doesn’t deeply explore why this works or when it fails. Are certain motion types (rotation, deformation) harder to capture? Does the prior collapse to dataset biases (e.g., most videos have static cameras)? The method feels like a solid first step, not a final answer.

Experimental integrity: Baselines are fair (CogVideoX, recent physics-aware methods). VideoPhy and VideoPhy2 are appropriate benchmarks. VBench results show LaMo doesn’t hurt general quality, which is important. However, the paper lacks failure case analysis. What happens on out-of-distribution physics (e.g., zero gravity, underwater)? Ablations are present but shallow—would be stronger with analysis of when drift vs field matters most.

Writing quality: Clear structure, good motivation. The related work section is thorough. However, the method section is dense—Figure 2 does heavy lifting but isn’t self-explanatory. The paper would benefit from a “LaMo in 3 sentences” box upfront and more intuitive explanation of why latent-space motion is the right level of abstraction. The results section lists numbers but doesn’t tell a story—why does LaMo win on some metrics but not others?

Verdict: weak accept — Solid contribution with practical value, but lacks depth in understanding when and why the method works. Good engineering, needs more science.

Takeaways

Steal the two-scale supervision pattern: Training with coarse-grained regularization (drift loss) and sampling with fine-grained guidance (motion field) is a transferable idea. If you’re building any generative model where output should follow implicit patterns (e.g., code generation following API usage patterns, molecule generation following chemical stability), consider learning a prior over transitions and using it at both training and inference time.

Latent-space priors are underutilized: Most work on improving generation quality focuses on architecture or data. LaMo shows you can extract useful supervision from the latent dynamics of your training data. If your model compresses inputs into latents, ask: what structure exists in how those latents evolve? Can you model and exploit it?

Self-supervision from temporal structure: Video has natural supervision in its temporal axis—frame N+1 follows frame N. LaMo exploits this without labels. The same principle applies to any sequential data: audio (waveform evolution), time-series (sensor dynamics), even text (token transition patterns). If your data has a time dimension, you have free supervision.