RAVEN: Bridging Training-Inference Distribution Gap in Autoregressive Video Generation

Paper: 2605.15190 Authors: Yanzuo Lu, Ronglai Zuo, Jiankang Deng Categories: cs.CV

The Gap

Autoregressive video diffusion models generate video in chunks, using previously generated frames as context for the next chunk. The standard approach distills a high-quality bidirectional teacher into a fast causal student. The problem: during training, the model sees clean ground-truth history, but at inference it must extrapolate from its own noisy predictions. This distribution mismatch compounds over long sequences—errors accumulate because the model never learned to handle its own mistakes.

Prior work (PAB, AVDC) trains on clean history and hopes the model generalizes. This paper argues that hope is not a strategy.

Problem: Training sees clean history, inference sees noisy self-generated history
    |
    v
Assumption: Models must train on the distribution they'll encounter at test time
    |
    v
Method: RAVEN interleaves clean endpoints + noisy rollout states during training
        CM-GRPO applies RL directly to consistency model's Gaussian kernel
    |
    v
Evidence: Outperforms PAB/AVDC on UCF-101, MSR-VTT across quality/semantic/motion
    |
    v
Conclusion: Explicit distribution alignment > implicit generalization hope

The Increment

One sentence: Before RAVEN, autoregressive video models trained on clean history and hoped to generalize to noisy self-predictions; after RAVEN, they train directly on interleaved clean-noisy sequences that mirror inference rollouts.

Core Mechanism

RAVEN’s training loop works like this: generate a video autoregressively (self-rollout), then repack that rollout into a training sequence where clean ground-truth chunks alternate with noisy intermediate denoising states. Each chunk’s loss now supervises not just the current prediction, but also the history representation that future chunks will depend on. The model learns to denoise while simultaneously learning to use partially-denoised history as context.

The data flow: start with a video, split into chunks. For each chunk position, run the model’s denoising process (multiple steps from noise to clean). Collect both the final clean output and intermediate noisy states. Repack these into a sequence: [clean chunk 1, noisy state from chunk 2 denoising, clean chunk 2, noisy state from chunk 3 denoising, …]. Train the model to predict each next chunk given this mixed-fidelity history.

CM-GRPO adds reinforcement learning on top. Standard flow-model RL (like in prior work) requires simulating an Euler-Maruyama discretization of the flow ODE, which is expensive. RAVEN observes that a consistency model’s sampling step is already a conditional Gaussian transition—you can write down the mean and variance analytically. So CM-GRPO treats this Gaussian kernel as the policy, computes advantages using a value network, and applies policy gradient directly. No auxiliary ODE simulation needed.

Structural metaphor: Think of training an autoregressive video model like teaching someone to navigate by giving turn-by-turn directions. The old way (PAB/AVDC) is like practicing navigation with a perfect GPS signal during training, then expecting the navigator to handle a glitchy GPS at test time. RAVEN is like training with a GPS that occasionally drops to low accuracy mid-route—the navigator learns to course-correct when the signal degrades. The interleaved clean-noisy sequence is the simulated signal dropout. CM-GRPO is like having a coach who watches the navigator’s decisions in real-time and adjusts their strategy based on whether they’re heading toward or away from the destination, without needing to simulate every possible wrong turn in advance.

Key Concepts

• Distribution mismatch in autoregressive generation: When a model generates sequences step-by-step, each step’s output becomes the next step’s input. If the model only trains on ground-truth inputs (clean data), it never sees the kinds of errors it will make at test time. Imagine learning to play piano by only practicing with a metronome, then performing without one—you’ve never practiced recovering from timing mistakes. In video generation, this means the model trains on perfect history frames but must generate from its own imperfect predictions, and small errors compound into large drift over long sequences.

• Consistency models: A consistency model learns to map any noisy version of data directly to the clean version in one step, rather than iteratively denoising like standard diffusion. Think of it as learning a “shortcut function” that jumps from any point on the noisy-to-clean path straight to the destination. The key property: if you apply the model to a slightly-less-noisy version of the data, you should get the same clean output (consistency). This makes sampling fast (one step instead of many), but the training objective is different from standard diffusion—you’re enforcing that all points along the denoising trajectory map to the same endpoint.

• Policy gradient on Gaussian kernels: In RL, a policy is a probability distribution over actions. For continuous actions, this is often a Gaussian (normal distribution). The policy gradient theorem says you can improve the policy by moving probability mass toward actions that got higher rewards. When RAVEN treats the consistency model’s sampling step as a Gaussian policy, it means: the model outputs a mean and variance, you sample from that Gaussian to get the next video chunk, then you adjust the mean/variance to make high-reward samples more likely. The math works out cleanly because Gaussians have simple derivatives, so you can compute gradients without simulating the full sampling process.

Framework Shift

Before (PAB/AVDC):                    After (RAVEN):
                                      
Training:                             Training:
[GT] -> [GT] -> [GT] -> [GT]          [GT] -> [noisy] -> [GT] -> [noisy]
 |       |       |       |              |        |         |        |
 v       v       v       v              v        v         v        v
Model sees only clean history         Model sees mixed clean/noisy
                                      
Inference:                            Inference:
[GT] -> [pred] -> [pred] -> [pred]    [GT] -> [pred] -> [pred] -> [pred]
 |        |         |         |         |        |         |         |
 v        v         v         v         v        v         v         v
Distribution shift!                   Aligned distribution
Errors compound                       Trained to handle errors

[One sentence: From training on idealized clean sequences to training on realistic noisy rollouts, the core shift is explicit distribution alignment.]

Expert Assessment

Problem choice: Real gap. The training-inference mismatch in autoregressive generation is well-documented (exposure bias in NLP, compounding errors in RL). Video generation inherits this problem but prior work mostly ignored it, hoping distillation quality would compensate. RAVEN directly addresses the elephant in the room.

Method maturity: The interleaved clean-noisy training is conceptually simple—almost obvious in hindsight—but the execution details matter (how to sample noisy states, how to weight losses across the sequence). CM-GRPO is clever: recognizing that consistency models already define a Gaussian policy avoids the Euler-Maruyama overhead of prior flow-model RL. However, the paper doesn’t deeply explore failure modes—what happens when the value network is miscalibrated, or when the RL objective conflicts with the distillation objective?

Experimental integrity: Baselines are recent and relevant (PAB, AVDC). Metrics span quality (FVD), semantics (CLIP), and motion (dynamic degree). The ablations isolate RAVEN’s contribution from CM-GRPO’s. One concern: all experiments are on relatively short videos (UCF-101 is 16 frames, MSR-VTT clips are short). The compounding error problem should be most visible on longer horizons—would be stronger to show 64+ frame results. Also, no user study, which matters for video quality perception.

Writing quality: The abstract and intro are clear. The method section gets dense—Figure 2’s diagram helps but the interleaved sequence construction could use a step-by-step algorithm box. The CM-GRPO derivation in Section 3.2 assumes familiarity with policy gradients; a reader without RL background will struggle. The related work section is thorough but could better position RAVEN relative to scheduled sampling and other exposure bias solutions from NLP.

Verdict: weak accept — Addresses a real problem with a principled solution, shows consistent gains, but experiments could be more comprehensive (longer videos, user studies) and the writing could be more accessible.

Takeaways

• Interleaved training for autoregressive models: If your model generates sequences and uses its own outputs as future inputs, train it on sequences that mix ground-truth and model-generated content. This applies beyond video—text generation, audio synthesis, any autoregressive domain. The key is sampling the noisy intermediate states in a way that reflects inference-time behavior.

• Exploit structure in your sampling process: RAVEN’s CM-GRPO works because consistency models have a closed-form Gaussian transition. If your generative model’s sampling step has exploitable structure (analytic gradients, known distributions), you can often apply RL more efficiently than treating it as a black-box policy. Look for shortcuts in your model’s math.

• Value networks for generation quality: Using a learned value function to estimate long-term quality (rather than just immediate reward) is underexplored in generative models. RAVEN’s value network predicts cumulative reward over the remaining sequence. This could transfer to other multi-step generation tasks where you want to optimize for end-to-end quality, not just per-step losses.