Memento: Reconstruct to Remember for Consistent Long Video Generation

Paper: 2606.14667
Authors: Xuan Wei, Longbin Ji, Guan Wang, Xiangrui Liu, Zhenyu Zhang, Shuohuan Wang, Yu Sun, Qingqi Hong
Categories: cs.CV

The Gap

Long video generation today works by decomposing the video into shots and generating them sequentially, with each shot conditioned on the previous one. This “next-shot continuation” paradigm, used in methods like VideoGPT, NUWA-XL, and Coach-LLM, is scalable but suffers from a critical blind spot: it never checks whether the accumulated memory still faithfully represents the subject’s identity. As generation proceeds, recurring subjects (e.g., a character’s face, a unique object) gradually drift—a phenomenon the authors call identity dilution: small errors compound, features are overwritten by newer context, and after a few shots the subject is effectively forgotten. No prior work treated subject preservation as an explicit reconstruction problem; they simply assumed that conditioning on history is enough. Memento shows this assumption is false, and replaces it with a learnable verification loop.

                   Problem:
    Recurring subjects diluted/forgotten across shots
                            |
                            v
                  Implicit Assumption:
   Conditioning on previous shots preserves identity
                            |
                            v (Assumption is false)
                    Explicit Method:
   Joint training of next-shot generation + memory-based
   subject reconstruction (recover target from memory alone)
                            |
                            v
                     Evidence:
   SOTA on subject consistency, cross-shot coherence, quality
                            |
                            v
                     Conclusion:
   Reconstruction-loop is necessary for long-term identity

The Increment

One sentence: Before Memento, long video generation trusted that sequential continuity would preserve subjects; after Memento, the generator must prove it can reconstruct the subject from its memory bank alone, making identity preservation a testable objective rather than a hope.

Core Mechanism

Memento consists of three interacting modules: a Memory Bank that stores past keyframes and their semantic features, a Generator that produces the next shot conditioned on both the global story caption and two separate memory queries, and a Reconstructor that takes only the memory bank and the global caption and tries to reconstruct a target subject appearance. The whole system is trained end-to-end with a dual loss: the standard next-shot generation loss (cross-entropy on visual tokens) plus a reconstruction loss that forces the memory bank to contain enough identity information to recover the subject.

Data flows as follows. During training, a full video is available. For a given shot boundary, the global caption (story-level description) and the current memory bank (from previous shots) are fed into the Generator. Additionally, a dual-query attention mechanism operates: one query (Q_long) attends over the entire memory bank to extract long-range identity cues, while another (Q_short) attends over a small set of recent keyframes to capture short-range coherence. The generator produces the next shot tokens. Separately, the Reconstructor receives the same memory bank and global caption, but *without the previous shot’s visual input, and tries to produce the target subject’s appearance (e.g., a close-up of the character’s face). The reconstruction loss penalizes failure, forcing the memory bank to be reconstructively sufficient—a property no previous method enforced.

                        +-------------------+
                        |   Global Caption  |
                        +---------+---------+
                                  |
                    +-------------+-------------+
                    |             |             |
                    v             v             v
             +-----------+  +-----------+  +-----------+
             | Q_long    |  | Q_short   |  | Reconst-  |
             | (over all |  | (over few |  | ructor    |
             | memory)   |  | keyframes)|  | (from     |
             +-----+-----+  +-----+-----+  | memory)   |
                   |              |         +-----+-----+
                   +------+-------+               |
                          |                       v
                    +-----v------+          +----------+
                    | Generator  |          | Reconst  |
                    | (produces  |          | loss     |
                    | next shot) |          +----------+
                    +-----+------+
                          |
                          v
                    +-----------+
                    | Next Shot |
                    +-----------+

Load-bearing metaphor: Think of Memento as a film continuity supervisor who works alongside the director.

• The Memory Bank is the supervisor’s binder of Polaroid photos: shots of the character from previous scenes, with notes about key outfit details, scars, or mannerisms.
• The Generator is the director shooting the next scene from a script (global caption) while glancing at the binder (long query) and the last few takes (short query).
• The Reconstructor is a pop quiz: the supervisor closes the binder and asks the assistant to sketch the character from memory alone. If the assistant fails, the binder didn’t contain enough identity info.
  The director and assistant are trained together: the director learns to store useful identity cues in the binder, and the assistant learns to reconstruct from those cues. Crucially, the assistant never sees the current shot, so the binder must be self-sufficient for identity—exactly what long-term consistency demands.

Key Concepts

• Identity reconstruction loss: Instead of just measuring how well the generated next shot matches ground truth, Memento adds a separate loss that asks: “Given only the memory bank (no recent context), can you still reconstruct the subject’s appearance?” This forces the memory to be a compact, identity-rich representation. Example: If a character has a unique scar on their cheek, the memory bank must encode it clearly; otherwise, the reconstructor will produce a blurry or wrong face, and the loss will spike.

• Dual-query memory: A single attention over all memory is prone to long-range distractions or short-range myopia. Memento splits the query into two: one attends to the entire memory bank (for stable identity features like hair color, face structure) and the other attends to only the most recent few keyframes (for motion continuity and scene transitions). Example: For a character walking through a door, the long query ensures they still look like the same person, while the short query picks up the correct door frame and walking pose from the last shot.

• Subject-aware cinematic data pipeline: To get clean reconstruction supervision, the authors built a custom dataset pipeline that extracts subject names from captions (e.g., “Alice” vs. a generic “person”) and aligns them with bounding-box annotations. This ensures that during training, the reconstruction loss targets specific subjects rather than aggregating over all entities. Without this, the reconstructor might learn to average over multiple characters and lose identity distinctiveness.

Framework Shift

Previous methods implicitly trusted sequential continuity; Memento explicitly verifies memory sufficiency. The shift is from generation-as-optimization-of-next-shot to generation-as-proof-of-identity-reconstruction.

Before (mainstream approach):
   Shot_1 --> Shot_2 --> Shot_3 --> ... (only continuity loss)
   [Memory is passively accumulated, never checked]
   Weakness: identity drifts silently.
   
After (Memento):
   Shot_1 --> Memory --> +--> Generator --> Shot_2
                         |        |
                         +--> Reconstructor (loss)
                         [Memory must pass reconstruction test]
   Strength: identity drift is penalized explicitly.

One sentence: From passive accumulation to active reconstruction verification, the core shift is that memory’s adequacy is no longer assumed but measured.

Expert Assessment

Problem choice: Real gap. Long video generation’s main failure mode is identity loss, and prior work (e.g., temporal decomposition) simply didn’t address it. This paper picks a fundamental, practically important problem.

Method maturity: Clever insight—framing identity preservation as a reconstruction task is elegant and principled. The dual-query mechanism is a reasonable engineering choice, though it could be argued that a single gated attention might suffice. The method is not brute force; it introduces a novel training objective that directly targets the problem. However, the reliance on a custom data pipeline (subject-aware annotations) may limit applicability to domains without such annotations.

Experimental integrity: Baselines include Coach-LLM, NUWA-XL, and a VideoGPT variant—standard competitors. Metrics cover subject consistency (CLIP score on subjects, face ID similarity), cross-shot coherence (temporal consistency), and visual quality (FID, FVD). The gains on subject consistency are sizable (e.g., +7% face ID similarity). One potential red flag: the reconstruction loss is computed on a single subject per shot—how do they handle videos with multiple subjects that share screen time? The paper mentions “consistent, pronoun-free subject descriptions” but multi-subject scenarios are not deeply discussed. Also, the ablation study only removes the reconstruction loss or the dual-query; removing the entire memory bank is not tested, which would be a strong sanity check.

Writing quality: The introduction and method sections are clear. However, the experiment section is a bit dense; a table summarizing the correlation between reconstruction loss value and subject consistency would add more intuition. The paper could also benefit from a failure case analysis (e.g., when does the reconstruction loss fail to prevent drift?). The appendix is thin.

Verdict: Weak accept — The core idea is sound and the results are promising, but the evaluation scope (single subject, custom pipeline) and missing comparisons to retrieval-based methods (e.g., storing reference images) leave room for future work. It’s worth reading for the reconstruction-loss framing.

Takeaways

• Reconstruction as verification: Any generative system with a long-term memory module can benefit from adding a reconstruction head that tests whether the memory is *self-sufficient for the most critical features (identity, style, layout). This is transferable to text-to-video, multi-image generation, and even long-document generation.
• Dual-query separation: Splitting attention into long-range identity queries and short-range coherence queries is a simple trick that can likely improve any sequential generation model. Practitioners should consider replacing a single cross-attention with two separate heads that have different receptive fields.
• Subject-aware annotation pipeline: The insight that modern LLMs can extract subject names from captions and tie them to visual annotations is useful beyond this paper. Anyone working on character consistency can adopt this cheap data augmentation strategy.