Mode Seeking meets Mean Seeking for Fast Long Video Generation

Paper: 2602.24289 Authors: Shengqu Cai, Weili Nie, Chao Liu, Julius Berner, Lvmin Zhang, Nanye Ma, Hansheng Chen, Maneesh Agrawala, Leonidas Guibas, Gordon Wetzstein Categories: cs.CV, cs.LG

The Gap

Current video generation models excel at producing high-fidelity clips lasting a few seconds, trained on abundant short-form content from platforms like YouTube and TikTok. But extending these models to generate coherent minute-long videos hits a data wall: long-form video datasets are scarce, domain-specific (mostly driving scenes or surveillance footage), and lack the visual diversity needed for general-purpose generation. Existing approaches either train exclusively on short clips (losing narrative coherence beyond a few seconds) or attempt to scale up on limited long-video data (sacrificing local visual quality). The state-of-the-art thus faces a fidelity-horizon tradeoff: you can have sharp 5-second clips or blurry 60-second sequences, but not both.

The Increment

Before: Train one model on either abundant short videos (high fidelity, no coherence) or scarce long videos (coherent structure, poor quality). After: Train one model with two heads that simultaneously learn global narrative flow from limited long videos and local visual realism from abundant short videos.

The core mechanism resembles a film production with two directors working in parallel. The global Flow Matching head acts like a narrative director who watches entire long videos and learns the arc of a story—how scenes transition, how camera movements evolve, how objects persist across time. This head uses supervised flow matching on the scarce long-video data to capture the “mean” trajectory of how minute-scale videos should unfold. Meanwhile, the local Distribution Matching head acts like a cinematography director who ensures every individual frame and short segment looks photorealistic. This head takes sliding windows (short clips) from the student model and aligns them to a frozen short-video teacher model using reverse-KL divergence—a “mode-seeking” objective that forces each window to match the sharp, high-fidelity distribution the teacher learned from abundant data.

The architecture is a Decoupled Diffusion Transformer where both heads share the same latent representation but optimize different objectives. During training, the global head pulls the model toward long-range coherence (mean-seeking: follow the average narrative structure), while the local head pulls every short segment toward high visual quality (mode-seeking: match the sharp modes of the short-video distribution). At inference time, the model generates minute-long videos in just a few denoising steps because the flow matching formulation provides a direct path from noise to video, and the local alignment ensures each step produces realistic frames.

Key Concepts

Flow Matching vs. Diffusion: Traditional diffusion models learn to reverse a gradual noising process—think of it like learning to unscramble an egg by watching thousands of eggs being scrambled in reverse. Flow matching takes a different approach: it learns a direct velocity field that transports noise to data, like learning the wind patterns that blow leaves from a pile into their original positions on a tree. Mathematically, diffusion models learn $p(x_{t-1}|x_t)$ iteratively, while flow matching learns a vector field $v_t(x)$ such that following $\frac{dx}{dt} = v_t(x)$ takes you from noise to data in one continuous path. The advantage? Flow matching often requires fewer steps (hence “few-step fast generation”) because you’re following an optimal transport path rather than a stochastic random walk.

Mode-Seeking (Reverse-KL) vs. Mean-Seeking (Forward-KL): Imagine you’re trying to learn a distribution of where people stand in a crowded room. Forward-KL (mean-seeking) penalizes you heavily if you miss any cluster of people—it forces you to spread your probability mass to cover all modes, even if that means putting some mass in empty spaces between clusters. Reverse-KL (mode-seeking) does the opposite: it penalizes you for putting mass where there are no people, so you focus on matching the densest clusters even if you miss some smaller groups. In this paper, the global head uses forward-KL (via supervised flow matching) to capture the full diversity of long-video narratives, while the local head uses reverse-KL to ensure each short segment locks onto the high-quality modes of the short-video teacher, avoiding blurry “average” frames.

Sliding Window Alignment: The local head doesn’t just check if the entire minute-long video looks good—it slides a short window (say, 2 seconds) across the generated video and aligns each window independently to the short-video teacher. Think of it like quality control on an assembly line: instead of inspecting the final product once, you inspect every subsection as it’s being built. This ensures that no matter where you pause the minute-long video, that local segment has the same visual fidelity as a standalone short clip. The frozen teacher model acts as a reference standard trained on millions of high-quality short videos, and the reverse-KL objective forces the student’s sliding windows to match that standard.

Expert Assessment

Problem significance: This is a high-impact problem. The video generation community is racing toward longer horizons, and the data scarcity bottleneck is real—companies like Runway and Pika are hitting the same wall. The affected community includes not just researchers but also content creators, filmmakers, and anyone building video synthesis applications. Solving this could unlock minute-long generation without requiring massive new long-video datasets.

Method maturity: This is closer to proof-of-concept than deployment-ready. The paper demonstrates the approach works, but several limitations loom. First, the method still requires *some long-video data for the global head—it’s not a zero-shot solution. Second, the sliding window alignment adds computational overhead during training (you’re effectively training two objectives simultaneously). Third, the paper doesn’t deeply explore failure modes: what happens when the global narrative structure conflicts with local realism? For example, if the flow matching head wants a camera pan but the local head resists because panning creates motion blur that the short-video teacher never saw, which objective wins? The authors acknowledge the need for balancing the two losses but don’t provide principled guidelines.

Experimental rigor: The baselines are reasonable but not exhaustive. The paper compares against training on short videos only, long videos only, and naive combinations, but doesn’t compare against recent autoregressive video models or hierarchical generation approaches that also tackle long-horizon synthesis. The datasets used (likely UCF-101 for short videos and something like Kinetics-600 or driving datasets for long videos) are standard but not disclosed in the abstract, making it hard to assess generalization. One red flag: the paper claims “few-step” generation but doesn’t specify how many steps or compare inference speed against baselines quantitatively.

Verdict: Weak accept — the core idea of decoupling objectives is clever and addresses a real bottleneck, but the execution feels incomplete without deeper analysis of failure modes and more comprehensive baselines.

Takeaways

Decouple objectives when data distributions mismatch: The key transferable insight is that when you have abundant low-level data (short videos) and scarce high-level data (long videos), you don’t need to choose one or the other—you can train dual objectives that exploit both. This applies beyond video: in robotics, you might have lots of teleoperation data (high-quality local actions) and sparse autonomous navigation data (long-horizon plans). Train one head to match local action quality and another to learn global navigation structure.

Mode-seeking for quality, mean-seeking for coverage: Use reverse-KL when you want to lock onto high-quality modes (e.g., aligning to a strong teacher model) and forward-KL when you need to cover diverse scenarios (e.g., learning from varied long-form data). This is a general principle in distribution matching that extends to GANs, VAEs, and any generative model where you’re balancing fidelity vs. diversity.

Sliding window supervision scales local quality: The sliding window trick is underutilized in sequence modeling. Instead of supervising only the final output, supervise every contiguous subsequence. This is applicable to text generation (align every sentence to a style model), audio synthesis (align every 1-second chunk to a quality reference), or even time-series forecasting (ensure every local window matches historical patterns).