DVD: Turning Video Diffusion Models into Deterministic Depth Estimators

Paper: 2603.12250 Authors: Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Jing He, Zixin Zhang, Haodong Li, Yihao Liang, Kanghao Chen, Bin Ren, Xu Zheng Categories: cs.CV

The Gap

Video depth estimation has two camps, and both are stuck.

Generative models (think Marigold, DepthCrafter) borrow the rich visual priors baked into diffusion models. The problem: diffusion is inherently stochastic. Run it twice, get two different depth maps. Across video frames, this produces “geometric hallucinations” — surfaces that flicker, scale that drifts, geometry that contradicts itself temporally. You can average over multiple samples to stabilize, but that’s expensive and still doesn’t fully fix the consistency problem.

Discriminative models (think UniDepth, Depth Anything) are deterministic and fast. But they’re hungry — they need massive labeled datasets to learn the semantic cues that resolve depth ambiguity (is that a dark patch a shadow or a hole?). Collecting metric depth labels at scale is genuinely hard, so these models hit a ceiling defined by annotation cost.

The gap: nobody has successfully converted a generative video model into a deterministic regressor while preserving the geometric priors that made the generative model good in the first place.

Problem: Generative = stochastic + hallucinations
         Discriminative = deterministic + data-hungry
         |
         v
Assumption: Diffusion models encode deep geometric priors
            that can be unlocked without the stochastic sampling loop
         |
         v
Method: Freeze backbone, repurpose timestep as anchor,
        add LMR for sharpness, global affine coherence for long video
         |
         v
Evidence: SOTA zero-shot on multiple benchmarks,
          163x less training data than leading baselines
         |
         v
Conclusion: Generative priors + deterministic inference is achievable
            and data-efficient

The Increment

One sentence: Before DVD, you had to choose between geometric richness (generative, stochastic) and reliable determinism (discriminative, data-hungry) — DVD shows you can have both by treating diffusion as a feature extractor rather than a sampler.

Core Mechanism

DVD takes a pre-trained video diffusion model and surgically converts it into a depth regressor. The key insight is that the diffusion model’s internal representations already encode geometry — the denoising process has learned what surfaces, edges, and spatial relationships look like. You don’t need to run the full stochastic sampling chain to access those representations.

The first design is the timestep anchor. In normal diffusion, the timestep tells the network “how noisy is this input right now.” DVD fixes the timestep at a specific value during inference — not running the denoising chain, just using that fixed timestep as a conditioning signal. This turns out to balance two things: low timesteps give you fine-grained detail sensitivity, high timesteps give you global structural stability. Picking the right fixed value lets you tune this trade-off once and bake it in.

The second design is Latent Manifold Rectification (LMR). When you convert a generative model to a regressor, the training objective changes from “reconstruct the data distribution” to “minimize regression loss.” This tends to produce over-smoothed outputs — the model hedges its bets and blurs boundaries. LMR adds a differential constraint during training that penalizes smoothing at edges, essentially telling the model “if there’s a sharp boundary in the input, preserve it in the depth map.” The third design is global affine coherence — a property they derive showing that across temporal windows, the predicted depth maps are bounded in how much their global scale and shift can diverge. This means you can process long videos in chunks and stitch them without complex post-hoc alignment.

Input Video Frames
       |
       v
  +--------------------+
  | Video Diffusion    |
  | Backbone (frozen)  |  <-- fixed timestep t* as structural anchor
  +--------------------+
       |
       | latent features (rich geometric priors)
       v
  +--------------------+
  | Depth Regression   |
  | Head               |  <-- trained on task-specific data (163x less)
  +--------------------+
       |
       | raw depth (over-smoothed)
       v
  +--------------------+
  | LMR                |
  | (differential      |  <-- restores sharp boundaries + motion edges
  |  constraints)      |
  +--------------------+
       |
       v
  Depth Map (sharp, temporally coherent)
       |
  [long video: global affine coherence bounds window-to-window drift]

Think of it like a master chef’s knife being repurposed as a scalpel. The knife (diffusion model) was forged with incredible metallurgy — the steel knows how to hold an edge, the balance is perfect, the geometry is refined over years. A scalpel needs different things: precision over power, determinism over versatility. You don’t reforge the steel (retrain the backbone). Instead, you reshape the handle (add a regression head), add a depth stop (timestep anchor so you always cut to the same depth), and add a guide rail (LMR to keep the cut clean at boundaries). The knife’s metallurgy — its geometric priors — transfers completely. The scalpel’s precision comes from the new fixtures, not new steel.

Key Concepts

• Stochastic geometric hallucination: In diffusion-based depth estimation, the model samples from a probability distribution over possible depth maps. Two runs of the same model on the same frame can produce geometrically inconsistent results — a wall might be at depth 2m in one sample and 2.3m in another. Across video frames, this compounds into flickering and drift. It’s not noise in the signal-processing sense; it’s the model genuinely being uncertain and expressing that uncertainty as variation. The problem is that for downstream tasks (3D reconstruction, robotics), you need one answer, not a distribution.

• Latent Manifold Rectification (LMR): When you train a model with L1 or L2 regression loss, it learns to minimize average error. At ambiguous boundaries — where depth changes sharply — the safest bet for minimizing average error is to predict a smooth transition rather than a sharp edge. LMR adds a term that explicitly penalizes this smoothing by comparing spatial gradients in the prediction against gradients in the input. If the input has a strong edge, the depth map should too. “Manifold” here refers to the idea that valid depth maps live on a lower-dimensional surface in the space of all possible depth maps — LMR pushes predictions back onto that surface.

• Global affine coherence: Depth maps are often predicted up to an unknown scale and shift (affine ambiguity). For long videos processed in windows, each window might drift in its global scale. This paper derives a bound showing that their architecture’s outputs across windows can only differ by a bounded affine transformation — meaning the drift is controlled and predictable, not unbounded. This is what lets them skip the complex temporal alignment post-processing that other methods need.

Framework Shift

Before (mainstream approach):        After (this paper):

  Generative path:                     Single deterministic path:
  Input -> Diffusion -> Sample         Input -> Frozen Diffusion
        -> Sample                             Backbone (fixed t*)
        -> Sample                             -> Regression Head
        -> Average -> Depth                   -> LMR
  (slow, stochastic, drifts)                  -> Depth
                                       (fast, deterministic, coherent)
  Discriminative path:
  Input -> Trained-from-scratch        Data requirement:
         Encoder                       163x LESS labeled data
         (needs massive labels)        (priors already in backbone)
         -> Depth
  (fast but data-hungry, misses
   subtle geometry)

  Long video:
  Window1 -> align -> Window2          Window1 -> Window2 -> Window3
          -> align -> Window3          (global affine coherence bounds
  (complex post-hoc alignment)          drift automatically)

From stochastic sampling or data-hungry training, the core shift is: treat the diffusion backbone as a frozen geometric prior extractor, not as a generative engine.

Expert Assessment

Problem choice: This is a real gap. The generative-vs-discriminative tension in depth estimation is well-documented, and the specific failure mode of temporal inconsistency in video diffusion depth is a genuine pain point for practitioners. The paper is well-positioned in the field’s trajectory — diffusion models have matured enough that “repurpose rather than retrain” is a credible research direction. Not manufactured.

Method maturity: The timestep-as-anchor idea is clever and underexplored. LMR is a reasonable engineering fix for a known regression pathology, though differential constraints on depth maps aren’t new in isolation — the novelty is applying them in this specific conversion context. The global affine coherence claim is the most interesting theoretical contribution, but the paper would benefit from a cleaner derivation. Overall: more clever insight than brute force, but the components are individually incremental; the value is in the combination.

Experimental integrity: The 163x data efficiency claim is striking and, if the baselines are fair, genuinely impressive. The concern is whether the comparison is apples-to-apples — leading baselines like DepthCrafter were trained on different data distributions with different objectives, so “163x less data” needs careful reading. Zero-shot benchmarks are the right evaluation choice. I’d want to see ablations on the timestep anchor value sensitivity and failure cases on scenes with unusual lighting or transparent surfaces, which are the classic stress tests for geometric priors.

Writing quality: The global affine coherence section feels rushed. It’s the most theoretically novel claim in the paper, but it’s presented more as an empirical observation than a derived property. If the authors tightened the mathematical argument there and showed exactly what assumptions it rests on, the paper would be significantly stronger. The LMR motivation is clear; the coherence derivation is not.

Verdict: weak accept — solid engineering contribution with a genuinely clever core idea, but the theoretical claims need more rigorous treatment and the experimental comparisons need clearer framing around data distribution differences.

Takeaways

The timestep-as-anchor trick is directly stealable for any task where you want to repurpose a diffusion model as a deterministic predictor. The insight — that a fixed timestep acts as a structural conditioning signal rather than a noise level — generalizes to optical flow, surface normals, or any dense prediction task where diffusion models have been applied generatively.

The LMR approach (adding differential constraints to fight regression-induced smoothing) is a clean, low-overhead fix that could be dropped into any dense regression pipeline where boundary sharpness matters. It’s not specific to depth or diffusion.

The broader framing — “frozen backbone as prior extractor, lightweight head as task adapter, minimal task-specific data” — is a template worth keeping in mind as foundation models get larger and labeled data stays expensive.