PDD: Skip Steps, Not Quality — Trajectory Distillation for Fast Video Generation

Paper: 2607.26004 Authors: Neta Shaul, Chao Liu, Arash Vahdat, Julius Berner Categories: cs.CV, cs.LG

The Gap

Current state-of-the-art diffusion acceleration relies on Variational Score Distillation (VSD) and adversarial losses to compress multi-step generators into few-step models. These approaches work — they produce stunning video — but they come with serious baggage: training instability, mode collapse, loss of video diversity, and motion that feels frozen. The field has been stuck in a local optimum: you can have speed or quality, but getting both without headaches has remained elusive.

The logical path from problem to solution:

Problem:
Diffusion/flow models need 20-50 denoising steps
for good generation -- too slow for video.

        |
        v

Prior solutions:
VSD + adversarial losses compress steps
but cause mode collapse and training pain.

        |
        v

Key insight:
What if we learn to skip steps directly
from teacher trajectories instead of
optimizing complex objectives?

        |
        v

PDD method:
Train student to predict multi-step
velocity from teacher demonstrations.
No adversarial losses. No VSD.

        |
        v

Evidence:
SOTA on LTX-2.3, Wan 14B, Qwen-Image
at 4-8 NFE with improved diversity.

        |
        v

Conclusion:
Simpler objective, better results.
The field's complexity was unnecessary.

The Increment

One sentence: Before this paper, fast video generation required wrestling with unstable adversarial training; after this paper, you can distill any diffusion model into a fast generator using straightforward trajectory supervision — and get better diversity as a bonus.

Core Mechanism

PDD works by training a student model to mimic a pretrained teacher’s denoising behavior, but with a crucial twist: the student learns to predict where the teacher would be after multiple steps, not just the next step. The teacher runs its full denoising process (say, 50 steps) on various inputs, recording the trajectory of intermediate states. The student then learns to jump from a noisy state directly to a state several teacher-steps ahead.

The architecture mirrors the teacher model, which means PDD is plug-and-play with any existing pretrained diffusion or flow matching model. During training, for each noisy input, the teacher computes its velocity field across the trajectory. The student receives the same noisy input plus timestep information and learns to output a velocity that corresponds to the mean trajectory over multiple steps — not the instantaneous derivative. This is the key simplification: you don’t need JVPs (Jacobian-vector products) or finite-difference approximations to approximate derivatives.

The student’s output can be evaluated with varying NFE at inference time. With 4 steps, you get fast generation; with 8 steps, you get better quality. This flexibility is built into the training procedure, not bolted on afterward.

Teacher (frozen):
  x_noisy --> [Teacher net] --> v_t (velocity at step t)
                    |
                    v
              trajectory recording
              v_t, v_{t+1}, ..., v_{t+k}

Student (training):
  x_noisy + t --> [Student net] --> v_predicted
                                     |
                                     v
                              loss against mean velocity
                              over [t ... t+k] from teacher

At inference:
  x_noisy --> [Student net] --> v_predicted --> x_less_noisy
       repeat 4-8 times --> clean output

Structural metaphor: Learning to drive with a GPS

Imagine teaching someone to navigate a city. The old way (adversarial/VSD) is like giving them a driving instructor who constantly critiques every micro-movement — “too far left, now brake harder, why did you turn there?” — and the student gets so anxious they only memorize one route and freeze on new streets. That’s mode collapse.

PDD is different. You record an expert driver taking a full journey (the teacher trajectory), then teach the student: “From this intersection, after 5 minutes of expert driving, you’ll end up here.” The student doesn’t need to learn the physics of steering or the derivative of acceleration. They learn the mapping: “current position + 5 minutes of expert driving = this destination.” During practice, they make big jumps — one confident turn instead of five hesitant corrections. At test time, they can choose to take 4 big jumps or 8 smaller ones depending on how much time they have. The expert driver’s GPS data is the trajectory; the student’s confident multi-block jumps are the multi-step predictions. No adversarial instructor needed.

Key Concepts

• Trajectory-based distillation: Instead of optimizing complex objectives that try to match distributions (like VSD does with its variational approach), PDD simply watches the teacher solve the problem repeatedly and learns the patterns. Think of it like learning math: the VSD approach is like trying to derive the formula from first principles (hard, many ways to fail), while trajectory distillation is like studying worked examples until you see the pattern (straightforward, scales with data). The teacher’s trajectory is a sequence of “here’s what happened at each step,” and the student learns to predict multiple steps at once from that data.

• Mean velocity prediction: In diffusion models, at each denoising step, the model predicts a “velocity” — the direction and magnitude to push the noisy image toward cleanliness. PDD doesn’t predict the velocity at just one timestep. Instead, it predicts the average velocity across several timesteps. Why? Because averaging smooths out the noise (pun intended). If the teacher’s velocity fluctuates between steps 10-15, the mean velocity captures the overall trajectory direction. This makes the student’s job easier and more robust — it’s the difference between following a GPS that recalculates every second versus one that gives you the overall direction.

• NFE flexibility (Number of Function Evaluations): This is how many times you run the neural network to generate one output. Traditional distilled models are fixed at one NFE. PDD supports varying NFE — you can use 4 steps for speed or 8 for quality, with the same trained model. This matters in practice because real deployments have varying latency budgets. Think of it as a zoom lens: want a quick sketch? Use 4 NFE. Want a polished result? Use 8. Same model, same weights.

Framework Shift

Before (mainstream):                    After (this paper):

  Teacher                                Teacher
    |                                      |
    v                                      v
  [Complex objectives]                   [Record trajectories]
  VSD + adversarial                       |
  losses                                  v
    |                                   [Simple velocity loss]
    v                                      |
  GAN-like training dynamics              v
  Mode collapse risk                    Student (multi-step)
  Hard to tune                             |
    |                                      v
    v                                   [Plug and play]
  Student (few-step)                     Works with any model
  Fragile, limited diversity              |
                                           v
                                         Better diversity
                                         + SOTA quality

From complex adversarial distillation to simple trajectory learning, the core shift is recognizing that you don’t need to fight the training dynamics — just watch the expert and learn to skip ahead.

Expert Assessment

Problem choice: This is a real gap. The field has been aware for a while that VSD-based distillation is painful to work with, but few alternatives have matched its quality. The problem is genuine and practically important — every team building video generation products struggles with inference latency.

Method maturity: The approach is elegant in its simplicity. Learning from trajectories rather than optimizing adversarial objectives is a cleaner path. That said, the “predict mean velocity” idea isn’t entirely novel conceptually — it echoes ideas from consistency models and progressive distillation. The contribution is in showing it works at scale with modern architectures and that the NFE flexibility emerges naturally. It’s more of a “right framing at the right time” than a fundamental theoretical breakthrough.

Experimental integrity: The baselines are reasonable and the models tested (LTX-2.3, Wan 14B, Qwen-Image) are genuinely state-of-the-art, not toy setups. The diversity improvements are demonstrated, though I’d want to see more rigorous diversity metrics — FVD and human evaluations can miss subtle collapses. The 4-8 NFE results are compelling. One concern: how does PDD degrade gracefully with very few NFE (1-2)? The paper focuses on 4-8, which might be the sweet spot but limits understanding of the method’s boundaries.

Writing quality: The paper reads clearly and the method is well-motivated. However, the ablation study could be stronger — I want to see what happens when you remove individual components (mean velocity vs. instantaneous, trajectory length, etc.). Section 4 could benefit from a failure case analysis: when does PDD struggle? Every method has limitations, and transparency builds trust.

Verdict: weak accept — The simplicity is the selling point, and the results on modern large-scale models are convincing, but the paper needs deeper ablations to fully earn confidence in the method’s robustness.

Takeaways

Three concrete things to steal:

1. Trajectory supervision beats adversarial objectives for distillation. If you’re distilling any generative model, consider recording the teacher’s full trajectories and training the student on multi-step targets. This is simpler, more stable, and often works just as well. The insight transfers beyond video to any iterative generation process (audio, 3D, molecules).

2. NFE flexibility as a first-class design goal. When building deployed generation systems, train with variable computation budgets baked in. This is underappreciated — most distilled models are fixed-step. PDD shows you can have a single model that adapts to latency constraints. Design your loss to support this from the start.

3. Mean over instantaneous. When the target signal is noisy or oscillatory, predict the average over a window rather than the point value. This is a general principle: smoothing the learning target often makes training easier without sacrificing final quality. If you’re working with noisy labels or unstable targets, try averaging before predicting.