PDI-Bench: Quantifying Geometric Coherence in Video Generation Models

Paper: 2605.15185 Authors: Jiaxin Wu, Yihao Pi, Yinling Zhang, Yuheng Li, Xueyan Zou Categories: cs.CV, cs.AI

The Gap

Video generation models are increasingly framed as “world models” — systems that implicitly learn physics and 3D structure. But existing evaluation methods rely on perceptual metrics (FVD, IS) or human judgment, which don’t directly measure geometric plausibility. A video can look sharp and realistic while violating basic projective geometry: objects shrinking as they approach the camera, rigid structures bending mid-motion, or depth relationships flipping arbitrarily.

Prior work either uses learned discriminators (which inherit dataset biases) or manual inspection (which doesn’t scale). No one has built a quantitative pipeline that directly audits whether generated videos respect the geometric constraints of a 3D world projected onto 2D frames.

Problem: Video models claim to learn "world models"
   |
   v
Gap: No way to measure geometric coherence quantitatively
   |
   v
Assumption: Projective geometry violations are detectable
            from 2D observations (segmentation + tracking)
   |
   v
Method: Lift 2D tracks to 3D, compute geometry residuals
   |
   v
Evidence: PDI catches failures invisible to perceptual metrics
   |
   v
Conclusion: Diagnostic signal for physically grounded video generation

The Increment

One sentence: Before this paper, we evaluated video quality through perceptual similarity; after, we can quantify whether a video obeys the geometric rules of 3D-to-2D projection.

Core Mechanism

PDI-Bench operates in three stages. First, it extracts object-centric observations from generated video using off-the-shelf tools: SAM 2 or MegaSaM for segmentation, CoTracker3 for dense point tracking across frames. This gives you per-object 2D trajectories.

Second, it lifts these 2D tracks into 3D world coordinates using monocular depth estimation (e.g., Depth Anything). Now each tracked point has an estimated 3D position at each frame.

Third, it computes three families of geometric residuals. Scale-depth alignment checks whether objects shrink/grow consistently with their depth changes (perspective projection law). 3D motion consistency verifies that an object’s velocity in 3D space remains smooth, not teleporting or jittering. 3D structural rigidity measures whether rigid objects (like buildings or furniture) maintain fixed inter-point distances across frames, or whether they warp and bend.

Input Video
    |
    v
[Segmentation]  -->  Object masks per frame
    |
    v
[Point Tracking]  -->  2D trajectories (x,y) over time
    |
    v
[Monocular Depth]  -->  3D coordinates (X,Y,Z) per point
    |
    v
[Geometry Residuals]
    |
    +---> Scale-depth alignment error
    +---> 3D motion consistency error
    +---> 3D structural rigidity error
    |
    v
PDI Score (lower = more coherent)

Structural metaphor: Think of PDI-Bench as a forensic accountant auditing a company’s books. The video generator is the company claiming its finances (3D world) are sound. PDI doesn’t trust the surface presentation (perceptual quality). Instead, it reconstructs the underlying ledger (3D coordinates) from the public statements (2D frames), then cross-checks three types of inconsistencies: (1) Do the reported asset sizes (object scales) match their stated distances (depths)? (2) Do the cash flows (object motions) follow smooth trajectories, or are there unexplained jumps? (3) Do the fixed assets (rigid structures) maintain their book values (inter-point distances), or are they being quietly revalued frame-by-frame? If the numbers don’t reconcile, the books are cooked — the video isn’t geometrically coherent.

Key Concepts

• Projective Geometry Residuals: In a real 3D world captured by a camera, certain mathematical relationships must hold. For example, if an object moves twice as far from the camera, its apparent size should halve (perspective projection). If you track two points on a rigid object, the 3D distance between them should stay constant across frames. A “residual” is the difference between what these geometric laws predict and what you actually observe in the video. Large residuals mean the video is violating 3D geometry — it’s not a plausible projection of any consistent 3D scene. PDI computes residuals for scale-depth coupling, motion smoothness, and structural rigidity, then aggregates them into a single diagnostic score.

• Monocular Reconstruction: You can estimate 3D structure from a single 2D image using learned depth predictors. These models (like Depth Anything or MiDaS) output a depth map: for each pixel, how far is that point from the camera? Given depth and 2D pixel coordinates, you can “unproject” back to 3D world coordinates using the camera’s intrinsic parameters (focal length, principal point). This isn’t perfect — monocular depth is ambiguous and scale-uncertain — but it’s good enough to detect gross geometric violations. PDI uses monocular reconstruction as a probe: if the video is geometrically coherent, the reconstructed 3D should obey physical laws; if not, the residuals will spike.

• Object-Centric Evaluation: Instead of analyzing the entire frame as a blob, PDI segments the video into individual objects (people, cars, buildings) and tracks each one separately. This matters because geometric failures often manifest per-object: a person’s limbs might stretch unnaturally, or a car might shrink as it approaches. By isolating objects, PDI can pinpoint which entities violate geometry and how. It also makes the evaluation more interpretable — you can say “this model struggles with rigid structures” rather than “this video has high error.”

Framework Shift

Before (mainstream approach):        After (this paper):

Video Generator                      Video Generator
    |                                    |
    v                                    v
Generated Video                      Generated Video
    |                                    |
    v                                    v
[Perceptual Metrics]                 [Segmentation + Tracking]
 - FVD, IS, CLIP-score                   |
 - Learned discriminators                v
    |                                [Monocular 3D Reconstruction]
    v                                    |
"Looks realistic"                        v
(no geometric diagnosis)             [Projective Geometry Residuals]
                                         |
                                         v
                                     "Obeys 3D laws"
                                     (quantitative diagnosis)

One sentence: From judging videos by perceptual similarity to real data, to auditing whether they respect the mathematical constraints of 3D-to-2D projection.

Expert Assessment

Problem choice: This is a real gap. The field has been treating video generation as a perceptual problem (make it look good) while claiming these models learn physics. PDI asks the uncomfortable question: do they actually? The timing is right — video models are being deployed as simulators and world models, so geometric coherence matters beyond aesthetics.

Method maturity: The approach is pragmatic rather than elegant. It stitches together existing tools (SAM 2, CoTracker3, Depth Anything) into a diagnostic pipeline. The residuals are straightforward geometric checks, not novel theory. This is a strength — the method is immediately usable and interpretable. The weakness is that it inherits the errors of its components: if the depth estimator is wrong, the residuals are noisy. The paper doesn’t deeply analyze this error propagation, which would strengthen the claims.

Experimental integrity: The baselines are fair — they test state-of-the-art video generators (Sora-like models, diffusion-based methods) on a custom dataset (PDI-Dataset) designed to stress geometric constraints. The results show PDI catches failures that perceptual metrics miss, which is the core claim. However, the paper doesn’t validate PDI against ground-truth 3D data (e.g., synthetic videos with known geometry), which would establish its accuracy. The correlation with human judgment is mentioned but not quantified. These are fixable gaps, not fatal flaws.

Writing quality: The paper is clear in motivation and method, but the results section is dense with tables and lacks visual examples of failure modes. Showing side-by-side comparisons (high FVD + low PDI vs. low FVD + high PDI) would make the diagnostic value visceral. The related work section undersells the novelty — it reads like a laundry list rather than positioning PDI as a paradigm shift from perceptual to geometric evaluation.

Verdict: weak accept — Addresses a real problem with a practical solution, but needs stronger validation and clearer presentation of diagnostic insights.

Takeaways

For practitioners building video models: PDI gives you a debugging tool. If your model scores well on FVD but poorly on PDI, you’re generating perceptually plausible but geometrically incoherent videos. This suggests your model is memorizing texture patterns without learning 3D structure. You can use PDI to diagnose specific failure modes (scale-depth coupling, motion jitter, structural warping) and target them in training.

For researchers evaluating world models: PDI demonstrates that perceptual metrics are insufficient for physical reasoning tasks. If you’re using video generation as a testbed for world models (e.g., for robotics planning), you need geometric evaluation. The three-residual framework (scale-depth, motion, rigidity) is a template you can adapt to other domains (e.g., auditing 3D scene generation, evaluating neural radiance fields).

For dataset builders: PDI-Dataset shows how to design stress tests for geometric coherence. The scenarios (camera motion, object interaction, occlusion) are chosen to expose specific failure modes. This is more diagnostic than random sampling from a large dataset. The lesson: evaluation datasets should be adversarial, not representative.

Transferable technique: The “lift 2D to 3D, compute residuals” pattern works beyond video. You can apply it to any generative model that claims to produce 3D-consistent outputs from 2D observations (e.g., image-to-3D, novel view synthesis). The key is identifying the geometric invariants your domain should respect, then checking whether the model violates them.