AdaState: Self-Evolving Anchors for Streaming Video Generation

Paper: 2605.30349 Authors: Yusuf Dalva, Pinar Yanardag Categories: cs.CV

The Gap

Autoregressive video diffusion models generate video chunk by chunk, conditioning each new segment on previously generated frames stored in an attention cache. The standard approach anchors the entire generation process to the first frame: its key-value representation sits permanently in the cache as the primary scene reference. Because the first frame is the cleanest (never corrupted by generation errors), the attention mechanism gravitates toward it disproportionately. This creates a structural bias: the model keeps referencing a frozen initial viewpoint even as the scene should naturally evolve. The result is temporally shallow videos where motion is dampened, camera movement is restricted, and scene composition remains locked to the starting configuration.

Prior work has focused on improving temporal consistency or extending generation length, but the fundamental architectural bias toward the first frame remains unaddressed. The gap is structural: the attention cache treats the first frame as privileged, creating an asymmetry that conflicts with the goal of generating videos with natural temporal dynamics.

Problem: Static first-frame anchor
   |
   v
Observation: Attention gravitates to cleanest cache position
   |
   v
Hypothesis: Anchor should evolve with content
   |
   v
Method: Replace first frame with adaptive hidden state
   |
   v
Evidence: Improved motion dynamics, natural scene progression
   |
   v
Conclusion: Recurrent state enables temporal depth

The Increment

One sentence: Before this paper, autoregressive video models were structurally anchored to a static first frame; after, the anchor becomes a hidden state that evolves alongside generated content, enabling richer temporal dynamics.

Core Mechanism

AdaState replaces the first frame in the attention cache with an adaptive state—a hidden latent that the model denoises at every generation step but never renders into pixels. At each chunk, the model attends to both the previous adaptive state and the current content being generated. Instead of denoising only the visible frames, the model jointly denoises the adaptive state and the new content frames. The adaptive state transitions forward through the same diffusion process that generates visible content, creating a recurrence where the state at step t becomes the reference for step t+1.

The key architectural change is treating time as relative rather than absolute. In standard autoregressive generation, each chunk has a different positional encoding reflecting its absolute position in the sequence. AdaState uses the same positional structure at every step: the adaptive state always occupies position 0, and the current chunk always occupies positions 1 through N. This positional invariance means the model sees the same structural pattern regardless of how far generation has progressed. The denoising operation itself becomes the state transition function, and the KV cache becomes the carrier of recurrent information—no external RNN or memory module required.

Data flows like this: at step t, the model receives the adaptive state from step t-1 and generates chunk t. During denoising, attention operates over the adaptive state and the current chunk. The model outputs both the denoised chunk (which gets rendered) and the updated adaptive state (which gets stored for step t+1). The adaptive state never becomes pixels—it exists purely as a latent reference that evolves to track scene progression.

Step t-1:                    Step t:
                                |
Adaptive State (t-1) -----> Attention <----- Current Chunk (t)
     |                          |                    |
     |                          v                    |
     |                    Joint Denoising            |
     |                          |                    |
     |                    +-----+-----+              |
     |                    |           |              |
     v                    v           v              v
Stored in Cache --> Adaptive State (t)  Rendered Frames (t)
                         |
                         +---> (feeds into step t+1)

Positional encoding (same at every step):
[State: 0] [Chunk: 1, 2, ..., N]

Think of it like a tour guide leading a group through a museum. A traditional autoregressive model is like a guide who stands at the entrance and shouts directions to the group as they move through rooms—the reference point never moves, so instructions become increasingly disconnected from where the group actually is. AdaState is like a guide who walks with the group: at each room, the guide observes both where they just were and where they are now, updates their mental model of the tour’s progression, and uses that updated understanding to lead into the next room. The guide’s “mental model” is the adaptive state—it’s not a physical location (not rendered) but an evolving representation of the tour’s trajectory. The guide sees each transition as structurally identical (room-to-room) even though the absolute position in the museum keeps changing.

Key Concepts

• Adaptive State: A hidden latent vector that occupies the anchor position in the attention cache but is never rendered into pixels. Unlike the first frame in standard models (which is generated once and frozen), the adaptive state is denoised at every generation step alongside visible content. It evolves through the same diffusion process that generates frames, allowing it to track scene progression rather than locking to an initial configuration. Think of it as the model’s “working memory” of where the scene has been and where it’s going—it’s not a snapshot but a continuously updated summary.

• Relative Time Encoding: Standard autoregressive models use absolute positional encodings where chunk 1 is at positions 1-N, chunk 2 at positions N+1-2N, and so on. AdaState uses the same positional structure at every step: the adaptive state is always position 0, the current chunk is always positions 1-N. This makes every generation step look structurally identical to the model, regardless of how many chunks have been generated. It’s like resetting the coordinate system at each step so the model always sees “here and now” rather than “frame 347 of an infinite sequence.” This positional invariance is what enables the recurrence—the state transition is the same operation repeated at every step.

• Recurrence Through Denoising: Traditional recurrent models (RNNs, LSTMs) have an explicit state transition function separate from their main computation. AdaState achieves recurrence by making the denoising operation itself serve as the transition function. At each step, the model denoises both the adaptive state and the current content, producing an updated state that feeds into the next step. The KV cache acts as the carrier of recurrent information—no separate memory module needed. It’s like using the same tool for two jobs: the diffusion process generates visible frames and simultaneously updates the hidden state that guides future generation.

Framework Shift

Before (static anchor):              After (adaptive state):

First Frame (frozen)                 Adaptive State (t-1)
     |                                      |
     +---> Chunk 1                          +---> Denoising
     +---> Chunk 2                          |         |
     +---> Chunk 3                          v         v
     +---> ...                         State (t)  Chunk (t)
                                            |
Attention always references              (feeds forward)
the same frozen anchor
                                      Attention references
Absolute positions:                   evolving anchor
[0] [1..N] [N+1..2N] ...
                                      Relative positions (repeated):
                                      [0] [1..N] [0] [1..N] ...

One sentence: From a fixed reference point that suppresses dynamics to an evolving reference that moves with the content, the core shift is from spatial anchoring to temporal recurrence.

Expert Assessment

Problem choice: This is a real gap. The first-frame bias in autoregressive video models is a known issue but has been treated as an inevitable tradeoff rather than a solvable architectural problem. The paper correctly identifies that the issue is structural—the attention mechanism’s preference for clean cache positions—rather than a tuning problem. The timing is good: as video generation scales up, temporal depth becomes the bottleneck.

Method maturity: The core insight is elegant: use the diffusion process itself as the state transition function rather than adding external modules. The relative time encoding is clever and well-motivated. However, the paper doesn’t deeply explore failure modes—what happens when the adaptive state drifts or accumulates errors over long sequences? The method feels like a strong v1 that will need refinement for production use. The simplicity is a feature, not a bug, but it also means there’s less room for tuning when things go wrong.

Experimental integrity: The baselines are fair and the improvements are substantial. The paper shows both quantitative metrics (motion scores, scene progression) and qualitative examples. One concern: the experiments focus on relatively short sequences (up to 10 seconds). The recurrent nature of the method raises questions about long-term stability that aren’t fully addressed. The ablations are solid—showing that both the adaptive state and relative time encoding are necessary—but I’d like to see stress tests on longer sequences and more diverse scene types.

Writing quality: The abstract and introduction are strong—they clearly articulate the problem and the solution. The method section is dense and could benefit from more visual aids to show the data flow. The related work section is adequate but doesn’t deeply engage with prior work on temporal consistency or recurrent video models. The conclusion is too brief and doesn’t discuss limitations or failure cases. Rewriting the method section with clearer diagrams and a step-by-step walkthrough would elevate the paper significantly.

Verdict: weak accept — The core idea is sound and the results are promising, but the paper needs more thorough evaluation on long sequences and failure modes before it’s a strong accept. The method is simple enough to implement and test, which is a plus for reproducibility.

Takeaways

The big transferable idea: when your model has a structural bias toward a fixed reference point, consider making that reference point recurrent. This applies beyond video generation—any autoregressive model that anchors to an initial state (text generation with a fixed prompt embedding, audio generation with a static conditioning vector) could benefit from letting that anchor evolve. The specific technique of using the same positional structure at every step to enable recurrence is worth stealing for other sequential generation tasks.

A smaller but concrete takeaway: the KV cache can serve as a recurrent memory without adding external modules. If you’re working with transformer-based generation and need temporal coherence, look at what’s already in the cache before adding new components.

Finally, a framing insight: treating time as relative rather than absolute can simplify recurrent architectures. Instead of tracking absolute position in a potentially infinite sequence, reset the coordinate system at each step. This makes the model’s job easier (same structure every time) and enables weight sharing across steps.