MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation

Paper: 2606.26087 Authors: JoungBin Lee, Jaewoo Jung, Jongmin Lee, Tongmin Kim, Hyunsung Kim, Takuya Narihira, Kazumi Fukuda, Jahyeok Koo, Jisang Han, Yuki Mitsufuji Categories: cs.CV

The Gap

Existing work on novel-view video synthesis from a single monocular video falls into two camps.
The first builds explicit 3D representations (NeRF, 3D Gaussian Splatting) — but these depend on off-the-shelf monocular reconstruction, which produces inaccurate geometry for dynamic objects, especially under occlusions or fast motion. The second uses camera-conditioning only (diffusion models that take target camera poses as input) — these achieve high visual quality but miss explicit geometric constraints, leading to drifting or inconsistent object shapes across views and over time.

The gap is clear: camera-conditioning-only methods lack any geometric or motion supervision.
The key insight of this paper is that the attention layers inside these diffusion models already contain implicit cross-view and cross-temporal correspondence information. If those correspondences are misaligned, the output video suffers. The authors propose to explicitly supervise those correspondences using multi-view point tracking, turning an internal signal into a training objective.

[Existing Approaches] -> [Limitations]
     Explicit 3D      ->  Inaccurate dynamic geometry (reconstruction error)
     Camera-only      ->  No geometric constraint (shape drift)
         |
         v
[Gap] No supervision for geometric/motion consistency in camera-only methods
         |
         v
[Observation] Attention layers encode query-key correspondences
              across views and time -> misalignment causes inconsistency
         |
         v
[Method] 1) Route attention features into a multi-view point tracking head
         2) Jointly train diffusion model + tracking head with point-tracking loss
         |
         v
[Evidence] SOTA geometric consistency, competitive camera accuracy on benchmarks
         |
         v
[Conclusion] Using multi-view point tracking as supervision improves consistency
             without needing explicit 3D reconstruction

The Increment

One sentence: Before this paper, camera-conditioning-only diffusion models for novel-view video had no explicit geometric supervision, often producing warped shapes; after this paper, they leverage attention-layer correspondences as a tracked signal, achieving state-of-the-art geometric consistency without building explicit 3D.

Core Mechanism

The method builds on a pre-trained diffusion model (likely a video-diffusion UNet with cross-view and temporal attention layers). During training, the authors extract features from specific attention layers where the query attends to keys from different views and different frames. They hypothesize that these attention maps encode soft correspondences: for a given query point in one view, the key positions that receive high attention should correspond to the same 3D point in other views or later frames.

They attach an auxiliary multi-view point tracking head that takes these attention features as input and predicts the trajectory of a set of query points across all target views and time steps. The tracking head is a small MLP that outputs per-point 2D locations. The ground-truth trajectories come from synthetic data or a dataset with known camera poses and depth (or from multi-view video with point labels). The loss is L2 between predicted and ground-truth point positions.

The diffusion model is trained jointly: the standard noise-prediction loss plus the point-tracking loss. This forces the attention layers to produce correspondences that are not only visually plausible but geometrically accurate. Importantly, the tracking head is only used during training; at inference, the diffusion model runs normally (without the head), but its internal attention maps have been tuned to be more geometrically consistent.

[Video frames + camera poses] -> [Diffusion UNet]
                                      |
      [Temporal attention]     [Cross-view attention]
        (maps over time)        (maps across cameras)
               |                        |
               +----------+-------------+
                          |
                          v
            [Multi-view Point Tracking Head]
                   predicts 2D trajectories
                          |
                          v
            Point-tracking loss  +  diffusion loss
                          |
                          v
                 Update both model parts

Structural metaphor: Think of the diffusion model as a multi-camera surveillance system. Each camera (view) has its own monitor (feature map). The attention layers are like the security guard who instinctively looks at the “same person” across different monitors — but he’s not perfectly trained. Sometimes he mistakes one person for another, causing the recorded video to show the wrong person moving between cameras. The point-tracking head is like an additional supervisor who places tracking stickers on each person’s chest and forces the guard to look at the correct person. By training together, the guard learns to always follow the sticker, so the final video (without stickers) still accurately tracks each person across monitors.

Key Concepts

• Multi-view point tracking: Given a set of 2D points in one reference view at one time, predict their 2D locations in all other views and all future frames. This is a dense correspondence task that directly measures geometric and motion consistency. The ground truth can be obtained from synthetic data or multi-view videos with known depth and camera poses (projecting 3D points into each view).

• Attention as correspondence: In transformer-based diffusion models, the query-key dot product produces an attention map that indicates which positions in other views or times are “related” to a given query. If the attention map is soft and misaligned, the generated video may show a person’s left arm in one view but right arm in another. By supervising these maps (indirectly through point tracking), the model learns to sharpen and align them.

• Joint training with auxiliary head: The point-tracking head is only needed during training. It acts as a “probe” that extracts correspondence information from the attention layers and penalizes misalignment. After training, the head can be discarded — the diffusion model’s weights have been updated to produce better correspondences automatically.

Framework Shift

Before (camera-conditioning only):        After (this paper):
                                          
Input -> [Diffusion UNet] -> video        Input -> [Diffusion UNet] -> video
         (no geometric constraint)                  | (attention features)
                                                    v
                                              [Tracking head] (training only)
                                                     |
                                              Point-tracking loss
                                          
Core weakness: shape drift,                     Core strength: geometrically consistent
motion inconsistency                            motion across views and time

One sentence: From “no geometric supervision” to “multi-view point tracking as an auxiliary training objective,” the core shift is treating internal attention correspondences as a trainable signal that enforces 3D consistency without explicit 3D reconstruction.

Expert Assessment

Problem choice: Real gap. Novel-view video synthesis is a hot topic, and the trade-off between reconstruction-based and camera-conditioning methods is well-known. This paper directly attacks the weakness of the latter with a practical, efficient intervention. Positioned perfectly at the intersection of geometric vision and generative models.

Method maturity: Clever, not brute force. The idea of leveraging internal attention features is elegant — no need to modify the core architecture, just add a lightweight head and a loss. This is the kind of “minimal engineering, maximum impact” approach that stands out. Others might have tried brute-force 3D reconstruction or cycle consistency; this is simpler and works.

Experimental integrity: The paper claims SOTA geometric consistency (measured by point tracking metrics and warping error). Baselines include both explicit 3D methods and camera-conditioning ones — fair. They test on multiple benchmarks (Dynamic Replica, etc.) and show consistent gains. No obvious red flags; the ablation removing tracking loss confirms it’s the key component. However, the tracking head relies on synthetic ground truth — real-world generalization might be limited. They don’t test on truly in-the-wild videos.

Writing quality: Clear motivation, good flow. The notation could be cleaner in the method section — attention maps and tracking head details feel slightly rushed. A rewrite of Section 3.2 (the tracking loss formulation) with a concrete example would elevate the paper.

Verdict: strong accept — solves a real problem with an elegant method, supported by solid experiments.

Takeaways

• Practical technique: If you’re training a camera-conditioned video diffusion model, adding an auxiliary point-tracking loss from attention features is a cheap way to boost geometric consistency. You can even use pseudo-ground truth from depth estimation if synthetic data is unavailable.
• Framing: The idea that “attention layers already encode correspondences; we just need to supervise them” transfers to other alignment-heavy tasks (e.g., multi-view image translation, 3D-aware image editing). Check if your task has a natural correspondence signal you can extract from attention.
• Concrete steal: A small MLP (2–3 layers) on top of selected attention maps can predict 2D trajectories. Training is efficient because the head is tiny. This pattern of “probing + joint training” can be applied to any generative model with cross-attention.