HiAR: Reversing the Order of Video Generation to Fix Error Accumulation

Paper: 2603.08703 Authors: Kai Zou, Dian Zheng, Hongbo Liu, Tiankai Hang, Bin Liu, Nenghai Yu Categories: cs.CV

The Gap

Autoregressive diffusion models can theoretically generate infinite-length videos by producing one block at a time, conditioning on previous blocks. The problem: existing methods fully denoise each block before moving to the next. This creates a vicious cycle—clean context carries prediction errors with high certainty, which compounds across blocks, causing progressive quality degradation (temporal drift). Prior work (VideoPoet, Tora, etc.) tries to maintain continuity by conditioning on highly denoised contexts, but this actually makes error propagation worse because mistakes are baked in with full confidence.

The core tension: you need temporal continuity (requires conditioning on previous blocks) but you also need to prevent error accumulation (which conditioning on clean blocks exacerbates).

Problem: AR video generation with clean context
    |
    v
Observation: Bidirectional diffusion maintains coherence 
             at shared noise levels
    |
    v
Hypothesis: Same-noise-level conditioning provides 
            sufficient signal without error propagation
    |
    v
Method: Hierarchical denoising across all blocks 
        simultaneously at each step
    |
    v
Evidence: Best VBench score, lowest temporal drift,
          1.8x speedup with pipeline parallelism
    |
    v
Conclusion: Reversing generation order (denoise-then-generate
            → generate-then-denoise) solves AR degradation

The Increment

One sentence: Before HiAR, autoregressive video generation meant completing each block fully before starting the next; after HiAR, all blocks are generated simultaneously at each denoising level, conditioning on equally-noisy context.

Core Mechanism

HiAR inverts the conventional generation order. Instead of: denoise block 1 completely → denoise block 2 completely → denoise block 3 completely, it does: denoise all blocks one step → denoise all blocks another step → repeat. At each denoising timestep t, every block conditions on the previous block at the same noise level t, not at a fully clean state.

The architecture has three key components: (1) a causal attention mechanism that lets each block attend to previous blocks at the current noise level, (2) a hierarchical denoising schedule that processes all blocks in parallel at each step, and (3) a forward-KL regularizer during distillation that prevents the model from collapsing into low-motion outputs.

Data flows like this: at timestep t, block i receives noisy input at level t and attends to block i-1 also at noise level t. The model predicts the denoising direction for all blocks simultaneously. As t decreases (noise reduces), all blocks gradually become cleaner together, maintaining temporal coherence because they’re always conditioned on context at matching noise levels.

Conventional AR:
Block 1: [noise=100] -> [noise=0] DONE
                                    |
Block 2:              [noise=100] -+-> [noise=0] DONE
                                                   |
Block 3:                            [noise=100] --+-> [noise=0]

HiAR (Hierarchical):
Step t=100: [Block1:100] -> [Block2:100] -> [Block3:100]
              |               |               |
Step t=75:  [Block1:75]  -> [Block2:75]  -> [Block3:75]
              |               |               |
Step t=50:  [Block1:50]  -> [Block2:50]  -> [Block3:50]
              |               |               |
Step t=0:   [Block1:0]   -> [Block2:0]   -> [Block3:0]
            (arrows show causal conditioning at same noise level)

Think of it like developing photographs in a darkroom. The old way: you fully develop photo 1, then use it as reference to develop photo 2, then use photo 2 to develop photo 3. Any mistake in photo 1 gets copied with full confidence into photo 2, then amplified into photo 3. HiAR’s way: you develop all photos simultaneously, gradually increasing exposure. At each exposure level, photo 2 looks at the partially-developed photo 1 (at the same exposure level). Mistakes in photo 1 are still fuzzy and uncertain, so they don’t propagate with full force. By the time everything is fully developed, the errors have been smoothed out across the gradual process rather than locked in early.

Key Concepts

• Same-noise-level conditioning: In diffusion models, noise level represents uncertainty. When block 2 conditions on block 1 at noise level 100, it’s saying “here’s a rough sketch of what came before.” When it conditions on block 1 at noise level 0 (fully clean), it’s saying “this is exactly what came before, trust it completely.” The insight: rough sketches provide enough signal for temporal continuity but don’t force the model to trust and propagate errors. Imagine writing a story chapter-by-chapter. If you edit chapter 1 to perfection before writing chapter 2, any plot holes in chapter 1 become canon. If you draft all chapters roughly first, then refine them together, you can fix inconsistencies across the whole story.

• Hierarchical denoising: Instead of a flat sequence (denoise A, denoise B, denoise C), you have nested loops: outer loop over noise levels, inner loop over blocks. This creates a hierarchy where the coarse structure (high noise) is established across all blocks first, then fine details (low noise) are filled in. It’s like sculpting: you rough out the entire figure before adding facial details, rather than completing the head perfectly before touching the torso.

• Forward-KL regularization: During distillation (compressing many denoising steps into fewer steps), the standard reverse-KL objective has a mode-seeking behavior—it prefers to generate one safe, low-motion output rather than risk diverse motions. Forward-KL is mode-covering—it penalizes the model for missing any mode of the true distribution. The paper uses forward-KL in bidirectional attention mode (where future context is visible) to teach the model about motion diversity, then switches to causal mode (no future context) for actual inference. It’s like learning to drive: you practice with an instructor who can see ahead (bidirectional), but you drive alone (causal) after training.

Framework Shift

Before (mainstream approach):        After (this paper):

Autoregressive Generation:           Hierarchical Generation:

t=T  [###] clean                     t=T  [~~~] [~~~] [~~~]
      |                                    \  |  /
t=0  [~~~] noisy                     t=T/2 [::] [::] [::] 
      |                                     \  |  /
      v                                t=0  [###][###][###]
t=T  [###] clean                           (all blocks denoise
      |                                     together at each
t=0  [~~~] noisy                            noise level)
      |
      v
t=T  [###] clean
      |
t=0  [~~~] noisy

(sequential: finish                  (parallel: all blocks
 each block before                    progress together,
 starting next)                       condition on same-
                                      noise-level context)

From sequential completion to parallel progression, the core shift is conditioning on uncertainty rather than certainty.

Expert Assessment

Problem choice: This is a real gap. Error accumulation in autoregressive generation is well-documented (not just in video—language models face it too). The paper correctly identifies that the standard solution (condition on clean context) actually worsens the problem. The insight about same-noise-level conditioning is borrowed from bidirectional diffusion but applied cleverly to the AR setting. It sits at the intersection of two active areas: long-form generation and diffusion model efficiency.

Method maturity: The core idea is elegant—reverse the loop order. But the execution has some brute force elements. The forward-KL regularizer feels like a patch for a symptom (low-motion collapse during distillation) rather than addressing the root cause. Why does distillation amplify this shortcut? The paper doesn’t dig deep. Also, the method requires training from scratch with the hierarchical structure; it’s not clear if existing AR diffusion models could be adapted. The pipeline parallelism speedup is nice but not groundbreaking (1.8x for 4 steps).

Experimental integrity: Baselines are reasonable (Tora, Pyramid Flow, etc.). VBench is a comprehensive benchmark. The temporal drift metric directly measures the problem they claim to solve, which is good. However, the paper doesn’t show failure cases or discuss when HiAR might struggle. The distillation experiments are somewhat isolated—would be stronger to show the forward-KL trick works across different distillation methods. The ablations are thorough but mostly confirm expected results.

Writing quality: The abstract and introduction are clear. The method section gets dense quickly—Figure 2 tries to show everything at once and ends up confusing. The bidirectional diffusion motivation (Section 3.1) could be tightened; it takes too long to get to the punchline. The distillation section (3.3) feels tacked on, like a separate paper. If I were rewriting, I’d restructure Section 3 to lead with the hierarchical denoising insight, show the architecture, then discuss distillation as an extension rather than a core component.

Verdict: weak accept — Solid idea with clear motivation and good results, but the execution has rough edges and the distillation component feels orthogonal to the main contribution.

Takeaways

The transferable insight: when doing autoregressive generation with iterative refinement (diffusion, iterative decoding, etc.), condition on context at the same refinement level as the current output, not on fully refined context. This prevents error propagation while maintaining coherence. You could apply this to: (1) autoregressive image generation with diffusion, (2) iterative text refinement models, (3) any sequential generation task where you have a notion of “partial completion.”

The pipeline parallelism trick: if your generation process has nested loops (outer: refinement level, inner: sequence position), you can parallelize the inner loop and pipeline the outer loop. This is architecture-specific but worth considering for inference optimization.

The forward-KL regularization during distillation: if your distillation collapses to low-variance outputs, try training with bidirectional context (where you can compute forward-KL) then deploying causally. This is a bit hacky but might work for other mode-collapse issues in distillation.