GeoT2V-Bench: Diagnosing 3D Consistency of Camera-Prompted Text-to-Video via Reconstruction

Paper: 2606.24829 Authors: Chenrui Fan, Paolo Favaro Categories: cs.CV

The Gap

Mainstream T2V evaluation (FVD, CLIP score, aesthetic scores) treats each frame independently or checks temporal smoothness.
These metrics cannot tell whether a video that looks like a camera orbit actually corresponds to a single rigid 3D scene:
the chair might morph between frames, or the background might drift without the viewer noticing.
Prior 3D-aware benchmarks (e.g., DreamBooth-style multi-view datasets) either require ground-truth 3D or limit to synthetic data.

GeoT2V-Bench bridges this: it diagnoses, without ground-truth 3D, whether a camera-prompted T2V clip can support explicit rigid reconstruction.

[Problem: T2V videos look good but fail 3D consistency]
    |
    v
[Assumption: Visual plausibility != static 3D coherence]
    |
    v
[Method: Reconstruction pipeline with flexible + static GS]
    |
    v
[Evidence: 5-metric profile exposes orthogonal failure modes]
    |
    v
[Conclusion: Need holistic 3D consistency, not single score]

The Increment

One sentence: Before, no tool could tell you *why a camera-prompted video was wrong in 3D; now GeoT2V-Bench gives a continuous, multi-axis diagnosis.

Core Mechanism

The pipeline runs on a single T2V clip (given a camera prompt) and outputs five continuous curves.
Step 1: Per-frame camera intrinsics & poses are estimated with VGGT-style geometry (a 3D-aware network).
Step 2: A Deformable Gaussian Splatting (DeformableGS) model is fitted to the whole clip – it can move each Gaussian per frame to explain motion.
Step 3: The temporal median of all Gaussians’ positions yields a *MedianGS proxy – a rigid static reconstruction under the assumption that non-rigid motion is noise.
Step 4: New frames are rendered from both models along the estimated camera path.
Step 5: Five metrics are computed per frame: apparent motion magnitude, trajectory deviation, MedianGS reconstruction error, optical flow agreement between MedianGS renders and the original video, and the gap between DeformableGS (flexible) and MedianGS (static) renderings.

The key insight: the five metrics rarely agree, so each catches a different failure type – e.g., wobbling trajectory, non-rigid object deformation, or static reconstruction collapse.

[Input: video frames + camera prompt]
    |
    v
[VGGT] -> intrinsics + poses per frame
    |
    v
[DeformableGS] -> per-frame deformable field (flexible)
    |
    v
[MedianGS] -> static proxy (median over time)
    |
    +--------------------+--------------------+
    |                    |                    |
[Render along path] [Render along path]  [Compare renders]
    |                    |                    |
[v] Apparent motion [v] Trajectory     [v] 5 metrics

Structural metaphor: Think of a film set’s “blueprint check.”

• VGGT is a laser rangefinder that measures where the camera was for each take.
• DeformableGS is a sculptor who can *push any patch of clay between frames — this captures if objects actually moved (e.g., an actor turning).
• MedianGS is a plaster cast that freezes the clay into one shape assuming the set is rigid.
• The five metrics are different inspectors: one checks if the camera path looks smooth (trajectory), another whether the plaster cast matches each raw video frame (static reconstruction error), a third whether the optical flow of the plaster video agrees with the original video, etc.
  If the sculptor’s pushes are large but the plaster cast looks terrible, the video probably has dynamic deformation rather than a static scene. If the plaster cast looks fine but the trajectory is erratic, the camera prompt was ill-posed. The metaphor holds because each role has a distinct failure signature.

Key Concepts

• DeformableGS: A 3D Gaussian representation where each Gaussian has a per-frame offset. It can model moving objects or drifting backgrounds. In this pipeline, its render quality acts as an *upper bound — if even the flexible model fails, the video is irrecoverably bad.

• MedianGS: A static scene proxy created by taking the temporal median of each Gaussian’s position/scale from DeformableGS. It represents what the scene *should look like if all motion were removed. Its reconstruction error reveals how much non-rigid deformation exists.

• Reconstruction Profile: Instead of a single pass/fail score, the benchmark outputs five curves over time (one per metric). The profile exposes *which kind of inconsistency plagues a video: e.g., high static-reconstruction error + low flow agreement = deep 3D inconsistency; high trajectory deviation = camera prompt misalignment.

Framework Shift

The old approach evaluates videos as sequences of images; the new approach evaluates them as candidate 3D captures.

Before (mainstream):              After (this paper):
[T2V clip]                        [T2V clip]
   |                                  |
[Frame-level metrics]             [3D reconstruction]
   |                                  |
[Single score]                    [5-axis profile]
   |                                  |
[Answer: "good" or "bad"]        [Answer: "which failure mode?"]

One sentence: From judging *visual appearance to diagnosing 3D consistency, the core shift is treating video as a multi-view capture rather than an animated flat image.

Expert Assessment

Problem choice: Real gap. As T2V with camera control matures, creators need to know if the output can be treated as a real scene. The field was lacking a diagnostic benchmark; this fills it logically. Sits exactly at the boundary between generation quality and 3D geometric correctness.

Method maturity: Clever combination of off-the-shelf components (VGGT, 3DGS) with a novel aggregation trick (MedianGS). The multi-metric profile is the main innovation, not the individual parts. There’s some brute force (fitting DeformableGS is expensive), but the authors are aware and provide completed reconstructions for reproducibility.

Experimental integrity: Fair – uses 4 seeds, 12 model configs (including open-weight like ZeroScope, ModelScope, etc.), 80 prompts from GeCo-Eval. The finding that metrics disagree is well-supported. One red flag: VGGT might be biased toward videos that resemble training data, but the authors use a fixed estimator, so comparisons across models remain valid.

Writing quality: Abstract and introduction are clear. The methodology section is dense but logically ordered. The experiments section could be improved by showing a single failure-mode visualization per metric (they provide some, but a dedicated “diagnosis card” per video would elevate the paper).

Verdict: weak accept — solid, needed benchmark, but the method is more diagnostic than generative; impact depends on adoption by the T2V community.

Takeaways

1. MedianGS as cheap 3D consistency proxy: Anyone working with T2V can plug this into their own pipeline (after estimating cameras) to get a per-frame static reconstruction error without training a new model.
2. Multi-metric profiles beat single scores: A pass/fail mask hides the specific failure. For debugging a T2V model, knowing it suffers from *static reconstruction collapse (high MedianGS error) vs camera drift (trajectory deviation) saves hours.
3. Four-seed evaluation is a fair practice: Many benchmarks use one seed; the authors show that model ranking changes across seeds, so future benchmarks should adopt multi-seed evaluation.