AnimationBench: The First Benchmark That Actually Gets Animation

Paper: 2604.15299 Authors: Leyi Wu, Pengjun Fang, Kai Sun, Yazhou Xing, Yinwei Wu, Songsong Wang, Ziqi Huang, Dan Zhou, Yingqing He, Ying-Cong Chen Categories: cs.CV

The Gap

Video generation models are getting good at making realistic videos. But when you ask them to animate—stylized characters, exaggerated motion, the stuff that makes Pixar Pixar—existing benchmarks can’t tell you if they’re any good. VBench, EvalCrafter, and others measure temporal consistency and motion smoothness, which matter for realism but miss what makes animation work: squash-and-stretch, anticipation, character identity preservation across frames. They use fixed prompt sets and rigid pipelines, so you can’t probe specific animation failures or test custom content. The field has been evaluating animation with the wrong ruler.

Problem: Realism benchmarks judge animation
   |
   v
Observation: Animation != realistic video
   |                (stylized, exaggerated, character-centric)
   v
Assumption: Need animation-specific principles
   |                (Disney's 12 principles + IP consistency)
   v
Method: AnimationBench with dual evaluation modes
   |           (close-set for comparison, open-set for diagnosis)
   v
Evidence: Better human alignment, exposes model weaknesses
   |              missed by realism benchmarks
   v
Conclusion: First systematic animation I2V benchmark

The Increment

One sentence: Before this paper, we evaluated animation generation with realism metrics; after, we have a benchmark grounded in actual animation principles that reveals character-centric and stylistic failures invisible to prior methods.

Core Mechanism

AnimationBench operationalizes animation quality into measurable dimensions. It takes the Twelve Basic Principles of Animation (Disney’s foundational rules: squash-and-stretch, anticipation, staging, etc.) and translates them into evaluation metrics. It adds IP Preservation—does the character look like the same character across frames?—and Broader Quality Dimensions like semantic consistency and motion rationality. The benchmark supports two modes: close-set evaluation uses a curated dataset of 200 animation images with standardized prompts for reproducible comparison; open-set evaluation lets you bring your own images and prompts for diagnostic analysis.

The evaluation pipeline uses vision-language models (VLMs) as judges. For each dimension, the system generates specific questions—“Does the character maintain consistent facial features?” “Is the motion exaggerated in an animation style?”—and feeds video frames to the VLM. The VLM outputs scores, which aggregate into dimension-level and overall quality metrics. This scales better than human annotation while maintaining alignment with human judgment.

Think of AnimationBench as a film school critique session, but automated. In traditional film school, you screen your animation and professors evaluate it against established principles—timing, spacing, appeal. Each principle has specific things to look for. AnimationBench does the same: it has a checklist derived from animation theory, watches your generated video frame-by-frame, and scores each principle. The close-set mode is like a standardized exam—everyone animates the same characters with the same prompts, so you can rank students fairly. The open-set mode is like office hours—bring your weird experimental piece, and the professor will still give you useful feedback using the same principles. The VLM is the professor who’s watched ten thousand animations and can spot when your character’s nose changes shape between frames.

Key Concepts

• The Twelve Principles as Evaluation Dimensions: Disney animators codified twelve principles in the 1930s—squash-and-stretch (objects deform to show weight/flexibility), anticipation (wind-up before action), staging (clear presentation), etc. These aren’t aesthetic preferences; they’re perceptual hacks that make motion readable and appealing. AnimationBench turns each into a measurable question. For squash-and-stretch: does the character deform during impact? For anticipation: is there preparatory motion before the main action? Prior benchmarks measured “is motion smooth?” This measures “does motion communicate intent the way animation should?”

• IP Preservation: In animation, character identity is sacred. Mickey Mouse must look like Mickey Mouse in every frame—same ear shape, same proportions, same design. Video models often drift: frame 1 has a character with round eyes, frame 30 has oval eyes. AnimationBench explicitly measures this by checking facial features, body proportions, and design consistency across the video. It’s not just “temporal consistency” (generic term); it’s “does this character maintain their identity?” (animation-specific requirement).

• Dual Evaluation Modes: Close-set gives you a leaderboard—200 curated images, fixed prompts, everyone generates the same content, scores are comparable. Open-set gives you a diagnostic tool—bring any image, write any prompt, get per-dimension feedback on what’s broken. Most benchmarks only do close-set. AnimationBench does both because animation production needs both: you want to know if Model A beats Model B generally (close-set), and you want to debug why your specific character’s hair keeps changing color (open-set).

Framework Shift

Before (realism benchmarks):        After (AnimationBench):

Input image                         Input image (animation style)
     |                                   |
     v                                   v
Generate video                      Generate video
     |                                   |
     v                                   v
Evaluate:                           Evaluate:
- Temporal consistency              - 12 Animation Principles
- Motion smoothness                   * Squash & stretch
- Object persistence                  * Anticipation
- Aesthetic quality                   * Staging, etc.
     |                              - IP Preservation
     v                                * Facial features
Score (realism-oriented)              * Body proportions
                                    - Broader Quality
                                      * Semantic consistency
                                      * Motion rationality
                                         |
                                         v
                                    Score (animation-oriented)
                                    + Diagnostic feedback

From measuring “does this look like a real video?” to “does this work as animation?”—the core shift is evaluating against domain-specific principles instead of generic realism.

Expert Assessment

Problem choice: Real gap. Video generation papers claim to handle animation, but nobody’s been checking if they actually do. The observation that realism benchmarks miss animation-specific failures is correct and important. This sits at the intersection of generative models and creative domains—timely as models move beyond photorealism.

Method maturity: Solid engineering, not groundbreaking science. Operationalizing the 12 principles is smart but straightforward—translate qualitative rules into VLM prompts. The dual-mode design is practical. The VLM-as-judge approach is trendy but appropriate here; human annotation doesn’t scale for animation evaluation. No major innovation, but doesn’t need to be—the contribution is the benchmark itself, not a novel evaluation algorithm.

Experimental integrity: Experiments are thorough. They validate against human judgment (correlation scores), compare multiple I2V models, and show AnimationBench discriminates where realism benchmarks don’t. Baselines are fair—they compare against VBench and others on the same content. One concern: VLM judges can be gamed if models are trained on VLM feedback, but that’s a general problem, not specific to this paper. The 200-image close-set is small but reasonable for a first benchmark.

Writing quality: Clear structure, good motivation. The related work section is dense—could trim some citations. The method section explains the principles well but gets repetitive listing all twelve. The experiments section is strong. Figures are helpful. Minor issue: they don’t discuss failure modes of VLM judges enough—when does the VLM get it wrong? Overall, solid paper that does what it says.

Verdict: weak accept — Fills a real need, execution is competent, and the benchmark will be useful to the community, but the method itself isn’t novel enough for a strong accept.

Takeaways

If you’re building benchmarks for creative domains, steal this: don’t evaluate with generic quality metrics—operationalize domain-specific principles. AnimationBench shows how to take qualitative expert knowledge (Disney’s 12 principles) and turn it into measurable dimensions. The dual-mode design (standardized comparison + flexible diagnosis) is smart for any benchmark where users need both leaderboards and debugging tools. For video generation specifically: character identity preservation is a distinct problem from temporal consistency, and current models are weak at it. If you’re training I2V models, add explicit losses for character feature consistency across frames.