EntityBench: A Benchmark for Character-Consistent Long Video Generation

Paper: 2605.15199 Authors: Ruozhen He, Meng Wei, Ziyan Yang, Vicente Ordonez Categories: cs.CV, cs.AI

The Gap

Single-shot video generation has matured, but extending it to multi-shot narratives hits a wall: characters change appearance, objects vanish and reappear differently, locations drift. Existing benchmarks use independently generated prompts with limited entity coverage, and metrics that blur intra-shot quality with cross-shot consistency. You can’t tell if a model fails because it can’t render a face, or because it forgot what that face looked like 20 shots ago.

Prior work treats consistency as a side effect of better generation. This paper argues it’s a distinct problem requiring explicit tracking.

Problem: Character in shot 1 =/= same character in shot 30
   |
   v
Assumption: Consistency needs explicit memory, not just better generation
   |
   v
Method: EntityBench (structured episodes) + EntityMem (per-entity memory bank)
   |
   v
Evidence: Consistency degrades with recurrence distance; explicit memory yields +2.33 Cohen's d
   |
   v
Conclusion: Cross-shot consistency is a separable problem needing dedicated mechanisms

The Increment

One sentence: Before this paper, multi-shot video generation was evaluated on ad-hoc prompts with conflated metrics; after, we have a structured benchmark that isolates cross-shot entity consistency from generation quality, plus a memory system that explicitly tracks entities across recurrence gaps.

Core Mechanism

EntityBench is 140 episodes derived from real narrative media, with explicit entity schedules. Each episode specifies which characters, objects, and locations appear in which shots, with difficulty tiers based on sequence length (up to 50 shots), entity count (13 characters, 8 locations, 22 objects), and recurrence gaps (up to 48 shots between appearances).

The evaluation suite has three pillars: intra-shot quality (FVD, aesthetic score), prompt-following alignment (CLIP similarity), and cross-shot consistency. The consistency pillar uses a fidelity gate: only entity appearances that match the prompt are admitted into cross-shot scoring. This prevents rewarding a model that generates consistent faces that are consistently wrong.

EntityMem, the baseline system, works in two phases. First, it generates reference images for each entity in isolation and stores them in a memory bank. Second, during multi-shot generation, it retrieves relevant entity references and conditions the diffusion model on them via cross-attention. The memory is persistent across the entire episode, not just adjacent shots.

EntityBench Structure:
  Episode --> [Shot 1] [Shot 2] ... [Shot 50]
                 |        |            |
              Entity    Entity      Entity
              Schedule  Schedule    Schedule
                 |        |            |
              (Char A,  (Char B,    (Char A,
               Loc X)    Obj Y)      Loc Z)

EntityMem Flow:
  Phase 1: Generate & Store
    Prompt "Character A" --> Diffusion --> Image --> Memory Bank [A]
    Prompt "Location X"  --> Diffusion --> Image --> Memory Bank [X]
  
  Phase 2: Retrieve & Condition
    Shot N needs [A, X] --> Retrieve [A, X] from Memory
                        --> Cross-Attention Conditioning
                        --> Generate Shot N with consistent A, X

Think of EntityMem as a casting director’s reference book. Before filming starts, you photograph each actor, prop, and location in isolation—clean headshots, no context. When shooting a scene, the director flips to the relevant pages and says “make it look like this.” The book doesn’t change between scenes; it’s the single source of truth. If an actor appears in shot 1 and shot 40, you’re looking at the same headshot both times, not relying on the crew’s memory of what they shot 39 scenes ago.

The key insight: consistency is a retrieval problem, not a generation problem. You don’t ask the model to “remember” what it generated before; you show it what to match.

Key Concepts

• Recurrence Gap: The number of shots between two appearances of the same entity. If Character A appears in shot 5 and shot 32, the recurrence gap is 27. This matters because existing models degrade sharply with distance—they might keep a character consistent across 3 shots but fail at 20. EntityBench explicitly tracks this to measure how consistency decays with gap length, revealing that most methods collapse beyond 10-shot gaps.

• Fidelity Gate: A filtering mechanism that only admits entity appearances into cross-shot consistency scoring if they first pass a prompt-alignment check. Without this, a model could generate the same wrong face in every shot and score high on consistency. The gate uses CLIP similarity between the generated entity and the text prompt; only matches above a threshold proceed to cross-shot comparison. This disentangles “did you generate what I asked for?” from “did you keep it consistent?”

• Per-Entity Memory Bank: A data structure that stores verified visual references for each entity (character, object, location) before generation begins. Unlike methods that condition on the previous shot or a sliding window, the memory bank is global and persistent. When generating shot N, the system retrieves all entities scheduled for that shot from the bank and uses them as conditioning signals. This decouples consistency from temporal proximity—an entity can reappear after 48 shots and still match its original reference.

Framework Shift

Before (mainstream approach):        After (this paper):

  Shot 1 --> Model --> Video 1         Entity Catalog:
     |                                   [A] [B] [C] [X] [Y]
     v                                    |   |   |   |   |
  Shot 2 --> Model --> Video 2           +---+---+---+---+
     |         ^                          |
     v         |                          v
  Shot 3 --> Model --> Video 3         Shot N Schedule: [A, X]
  (model tries to remember                |
   what it generated before)              v
                                        Retrieve [A, X] --> Model --> Video N
                                        (explicit reference, not memory)

From implicit temporal coherence to explicit entity retrieval, the core shift is treating consistency as a lookup problem rather than a generation constraint.

Expert Assessment

Problem choice: Real gap. Multi-shot video generation is moving from tech demos to narrative applications, and character drift is the obvious blocker. The paper doesn’t manufacture the problem—anyone who’s tried to generate a 10-shot sequence has hit this. The benchmark design (derived from real media, explicit entity schedules) is more rigorous than prior work’s ad-hoc prompt sets.

Method maturity: EntityMem is straightforward—almost too straightforward. The two-phase design (generate references, then retrieve) is conceptually clean but computationally expensive. You’re running inference twice: once for the memory bank, once for the actual shots. The paper doesn’t explore cheaper alternatives like learned entity embeddings or distilled references. That said, the simplicity is a feature for a baseline; it isolates the effect of explicit memory without confounding factors.

Experimental integrity: Baselines are fair (Mora, CogVideoX, Kling, Gen-3). The fidelity gate is a smart design choice that prevents gaming the consistency metric. The Cohen’s d = +2.33 for character fidelity is a large effect size, but the paper is honest about where EntityMem doesn’t win (object consistency is weaker). One concern: the memory bank references are generated in isolation, which might not match the visual style of the final shots. The paper doesn’t ablate this.

Writing quality: The benchmark description is dense but necessary. The related work section could be tighter—it lists methods without clearly positioning them relative to the paper’s contribution. The results section is thorough but repetitive; Figure 4’s recurrence gap analysis is the key insight and could be foregrounded earlier. The appendix has useful details (entity schedules, failure cases) that belong in the main text.

Verdict: weak accept — Solid benchmark contribution with a simple but effective baseline; the recurrence gap analysis is the paper’s strongest result, but the method itself is more proof-of-concept than production-ready.

Takeaways

Fidelity gate as a design pattern: When evaluating consistency, separate “did you generate the right thing?” from “did you keep it consistent?” This applies beyond video—any multi-step generation task (dialogue, code, design iterations) benefits from gating consistency metrics on correctness first.

Recurrence gap as a diagnostic: Measuring how performance degrades with distance between entity appearances is a transferable evaluation strategy. In code generation, how well does a model maintain variable naming conventions across 100 lines? In dialogue, does character voice drift over 50 turns? The gap metric makes degradation visible.

Explicit memory vs. implicit coherence: EntityMem’s two-phase design (build reference catalog, then retrieve) is overkill for video but the principle generalizes. When consistency matters more than novelty, decouple the “what to generate” (retrieval) from the “how to generate it” (model). This shows up in RAG systems, style transfer, and any task where you want controlled variation around a fixed reference.