CalibAtt: Making Video Generation 1.6x Faster by Learning Which Attention to Skip

Paper: 2603.05503 Authors: Shai Yehezkel, Shahar Yadin, Noam Elata, Yaron Ostrovsky-Berman, Bahjat Kawar Categories: cs.CV

The Gap

Text-to-video diffusion models like Wan 2.1 and Mochi 1 produce stunning results, but they’re painfully slow. The bottleneck? Spatiotemporal attention in their transformer backbones, which computes relationships between every token pair across space and time. Prior acceleration methods either require expensive retraining (pruning, distillation) or apply generic sparsity patterns that don’t adapt to the specific model and input characteristics. The field has reached a point where we can generate high-quality videos, but waiting minutes for a single clip makes iteration unbearable.

Problem: Video diffusion too slow
   |
   v
Observation: Many attention scores are consistently near-zero
   |                    across different inputs
   v
Assumption: These low-score patterns are predictable
   |          and can be identified offline
   v
Method: CalibAtt - offline calibration finds stable
   |     sparsity patterns, compile into fast ops
   v
Evidence: 1.58x speedup on Wan 2.1 14B, quality preserved
   |
   v
Conclusion: Input-stable sparsity exists and is exploitable

The Increment

One sentence: Before CalibAtt, you either retrained models for speed or applied blind sparsity; after CalibAtt, you run a one-time calibration that learns which attention connections each model actually needs, then skip the rest at inference without quality loss.

Core Mechanism

CalibAtt has two phases. First, an offline calibration pass runs the model on a small set of diverse prompts and records attention scores at every layer, head, and diffusion timestep. It analyzes these scores at the block level (groups of tokens) rather than individual tokens, identifying which block-to-block connections consistently produce negligible attention weights. It also detects repetition patterns—cases where multiple query blocks attend to the same key blocks with similar patterns, allowing computation reuse.

The calibration compiles these findings into optimized attention kernels for each layer-head-timestep combination. Some connections are marked for dense computation (input-dependent, high-variance scores), others for skipping (consistently low scores), and some for reuse (repetitive patterns). This compilation step is crucial—it translates statistical observations into hardware-efficient operations.

At inference time, CalibAtt loads these precompiled patterns. For each attention operation, it computes only the selected dense connections and skips the rest. The skipping isn’t just masking—it’s actual computation avoidance through kernel fusion and memory access optimization. The method is training-free; no model weights change.

Offline Calibration:                 Inference:
                                     
[Diverse prompts]                    [User prompt]
       |                                   |
       v                                   v
+-------------+                      +-------------+
| Run model   |                      | Load model  |
| Record all  |                      | + patterns  |
| attn scores |                      +-------------+
+-------------+                            |
       |                                   v
       v                              For each layer:
+-------------+                       +-------------+
| Analyze at  |                       | Compute     |
| block level |                       | selected    |
+-------------+                       | connections |
       |                              | Skip others |
       v                              +-------------+
+-------------+                            |
| Find stable |                            v
| low-score & |                      [Generated video]
| repetition  |
| patterns    |
+-------------+
       |
       v
+-------------+
| Compile to  |
| optimized   |
| kernels     |
+-------------+

Think of CalibAtt like optimizing a delivery route. Standard attention is like a delivery driver who visits every single address in the city every day, even the ones that never order anything. CalibAtt’s calibration phase is like analyzing a month of delivery data to find patterns: “These 20 addresses never order, these 10 always order the same thing on the same day, and these 5 have unpredictable orders.” The compilation step creates three route types: skip the never-orderers entirely, batch-deliver to the predictable ones, and visit the unpredictable ones individually. At inference (actual delivery day), the driver follows these optimized routes. The calibration data is the historical orders, the patterns are the route rules, and the compiled kernels are the actual optimized route maps the driver uses.

Key Concepts

• Block-level sparsity: Instead of deciding whether token A should attend to token B (which would require analyzing billions of pairs), CalibAtt groups tokens into blocks and decides whether block A should attend to block B. Imagine you’re organizing a conference: instead of tracking every possible one-on-one conversation between 1000 attendees, you organize people into 10 interest groups and decide which groups should have joint sessions. This reduces the decision space from 1M pairs to 100 pairs while capturing the essential interaction structure. In video generation, spatial blocks might be 8x8 pixel patches, and temporal blocks might be 4-frame chunks. If a background sky block consistently doesn’t attend to a foreground character block across different videos, that entire block-to-block connection can be skipped.

• Input-stable patterns: Not all sparsity is equal. Some attention connections have low scores for one input but high scores for another—these are input-dependent and must be computed. But CalibAtt discovered that many connections have consistently low scores across diverse inputs. These are input-stable. It’s like finding that certain ingredient combinations never work in cooking, regardless of the specific recipe. If garlic and chocolate consistently taste bad together across 100 different dishes, you can safely skip trying that combination in dish 101. CalibAtt’s calibration identifies these stable patterns by measuring score variance across calibration samples—low mean and low variance means safe to skip.

• Repetition exploitation: In transformer attention, different query tokens often compute nearly identical attention patterns over the keys. CalibAtt detects when multiple query blocks produce similar attention distributions and reuses the computation. Concretely, if query blocks Q1, Q2, Q3 all attend to the same key blocks with similar weights, compute the attention for Q1 fully, then copy and slightly adjust for Q2 and Q3 instead of recomputing from scratch. This is like a teacher grading essays: if five students wrote nearly identical answers, you carefully grade the first one, then quickly verify the others match rather than re-reading each word-by-word.

Framework Shift

Before (standard attention):         After (CalibAtt):

Every layer, every step:             Calibration (once):
                                     +------------------+
+------------------+                 | Analyze patterns |
| Compute all      |                 | across samples   |
| token-to-token   |                 +------------------+
| attention scores |                          |
+------------------+                          v
        |                            +------------------+
        v                            | Compile sparse   |
+------------------+                 | attention ops    |
| Apply softmax    |                 +------------------+
| to all scores    |                 
+------------------+                 Inference (fast):
        |                            +------------------+
        v                            | Load patterns    |
+------------------+                 +------------------+
| Weighted sum     |                          |
| over all values  |                          v
+------------------+                 +------------------+
                                     | Compute selected |
(Compute everything,                 | connections only |
 every time)                         +------------------+
                                              |
                                              v
                                     +------------------+
                                     | Skip & reuse     |
                                     | where possible   |
                                     +------------------+
                                     
                                     (Pay once, save forever)

From “compute everything every time” to “learn once what to skip, then skip it forever,” the core shift is trading upfront analysis cost for perpetual inference savings.

Expert Assessment

Problem choice: This is a real gap. Video generation models are genuinely bottlenecked by attention, and the problem will only worsen as models scale. The observation that sparsity patterns are input-stable is non-obvious and valuable. It sits at a sweet spot in the field’s trajectory—models are mature enough that acceleration matters, but not so optimized that low-hanging fruit is gone.

Method maturity: This is clever insight, not brute force. The key innovation is recognizing that block-level patterns are stable across inputs, which isn’t obvious from prior work on dynamic sparsity. However, the method does require offline calibration, which adds deployment friction. A simpler approach might be online adaptation, but that would sacrifice the compilation optimization. The block-level granularity is well-chosen—coarser than token-level (enabling hardware efficiency) but finer than layer-level (preserving flexibility).

Experimental integrity: The baselines are fair—they compare against other training-free methods and show results on multiple models (Wan 2.1, Mochi 1) at various resolutions. The 1.58x speedup is end-to-end, not just attention-layer speedup, which is honest. However, I’d want to see more analysis of failure cases. What happens when the calibration set doesn’t cover the input distribution well? The paper shows quality metrics (FVD, text alignment) are preserved, but doesn’t deeply probe edge cases. The calibration cost isn’t thoroughly discussed—how many samples and how long?

Writing quality: The paper is clear on the method but light on intuition. The related work section efficiently positions the contribution, but the method section could use more visual aids showing actual sparsity patterns discovered. The ablation studies are solid, but the paper would benefit from a “when does this fail?” section. The experimental setup is well-documented, but the calibration procedure details are scattered.

Verdict: weak accept — Solid contribution with practical impact, but needs more analysis of calibration robustness and failure modes.

Takeaways

The transferable idea here is input-stable pattern mining for inference optimization. This applies beyond video generation: any model with repeated computations across diverse inputs can benefit. Specifically:

1. Block-level analysis over token-level: When optimizing attention or other pairwise operations, analyze at a coarser granularity than the finest level. This reduces search space and enables hardware-efficient skipping.

2. Offline calibration as a deployment strategy: For models where inference cost dominates training cost, spending compute once to analyze patterns can pay off indefinitely. This is especially relevant for foundation models that serve many users.

3. Variance as a sparsity signal: Low mean + low variance across diverse inputs = safe to skip. This heuristic works for any operation where you’re deciding what to compute.

The specific technique of compiling patterns into optimized kernels per layer/head/timestep is also stealable—it’s essentially JIT compilation for neural network inference based on empirical statistics rather than static analysis.