MonarchRT: Efficient Attention for Real-Time Video Generation

Paper: 2602.12271 Authors: Krish Agarwal, Zhuoming Chen, Cheng Luo, Yongqi Chen, Haizhong Zheng, Xun Huang, Atri Rudra, Beidi Chen Categories: cs.CV, cs.LG

Abstract

Real-time video generation using Diffusion Transformers faces a critical bottleneck: the quadratic computational cost of 3D self-attention. This challenge becomes particularly acute in autoregressive, few-step generation regimes where errors accumulate over time and each denoising step must encode significantly more information. The authors discover that video attention patterns are fundamentally different from static images—they exhibit complex periodic spatiotemporal structures combined with dynamic semantic correspondences that exceed the capacity of traditional sparse attention methods. MonarchRT addresses this through a novel structured attention parameterization based on Monarch matrices, achieving up to 95% attention sparsity without quality loss and enabling true real-time generation at 16 FPS on a single RTX 5090 GPU.

Key Contributions

• Empirical Analysis of Video Attention: Demonstrates that video attention is not reliably sparse but combines periodic spatiotemporal structure with dynamic semantic correspondences, challenging assumptions behind existing sparse attention methods
• Monarch Matrix Parameterization: Introduces a structured attention factorization using extended tiled Monarch matrices with appropriately aligned block structures for high expressivity and computational efficiency
• Custom Kernel Implementation: Develops optimized Triton kernels that outperform FlashAttention-2/3/4 by 1.4-11.8× across RTX 5090, H100, and B200 GPUs
• Real-Time Performance: First demonstration of true real-time video generation with state-of-the-art Self-Forcing model at 16 FPS on consumer hardware
• Scalable Sparsity: Achieves 95% attention sparsity on Self-Forcing without quality degradation, validated against both sparse baselines and oracle top-k attention

The Real-Time Video Generation Challenge

The computational demands of video generation with Diffusion Transformers scale as $O(n^2d)$ for sequence length $n$ and dimension $d$, making 3D spatiotemporal attention prohibitively expensive. Unlike bidirectional many-step diffusion where each step contributes incrementally, autoregressive few-step generation requires each denoising operation to carry substantially more information—errors compound temporally, and the model cannot “look ahead” to future frames for context.

Previous sparse attention approaches like local windows, strided patterns, and learned top-k selection work well for bidirectional image generation but fail in this regime. The authors’ analysis reveals why: video attention patterns exhibit three simultaneous characteristics that exceed the representational capacity of simple sparsity patterns:

1. Periodic Structure: Strong spatiotemporal position-driven patterns that repeat across frames
2. Dynamic Semantics: Sparse but critical semantic correspondences that vary by content
3. Dense Mixing: Regions requiring dense attention that cannot be captured by fixed sparsity budgets

Even oracle top-k attention (which knows the optimal sparse pattern) struggles to capture this complexity within reasonable sparsity budgets.

Monarch Matrix Methodology

MonarchRT’s core innovation is parameterizing attention weights using Monarch matrices—a class of structured matrices that can be expressed as products of block-diagonal matrices. For an $n \times n$ attention matrix, the Monarch parameterization factorizes it as:

$A = L \cdot R$

where $L$ and $R$ are block-diagonal matrices with block size $b$. This reduces parameters from $O(n^2)$ to $O(nb)$ while maintaining expressivity through the interaction of block structures.

The authors extend this with a tiled Monarch parameterization that applies the factorization hierarchically across spatial and temporal dimensions. Key design choices include:

• Aligned Block Structure: Block boundaries align with natural video structure (spatial patches, temporal segments) to capture periodic patterns
• Extended Tiling: Multiple Monarch factors at different scales to handle both local and global dependencies
• Learnable Parameterization: Fine-tuning the Monarch factors on video data to adapt to dynamic semantic patterns

The computational cost reduces to $O(nb \log n)$ for the forward pass, with the block structure enabling efficient parallel computation.

Implementation and Performance

The authors implement MonarchRT with custom Triton kernels optimized for the block-diagonal structure of Monarch matrices. Key optimizations include:

• Fused Operations: Combining matrix multiplications and attention computations to reduce memory bandwidth
• Block-Aware Scheduling: Exploiting block structure for better cache utilization and parallelism
• Mixed Precision: Strategic use of FP16/BF16 for throughput while maintaining numerical stability

Performance results demonstrate significant speedups over state-of-the-art attention implementations:

GPU	vs FlashAttention-2	vs FlashAttention-3	vs FlashAttention-4
RTX 5090	1.4-3.2×	-	-
H100	2.1-5.7×	1.8-4.3×	-
B200	3.8-11.8×	2.9-8.4×	2.2-6.1×

When integrated with the Self-Forcing model, MonarchRT enables 16 FPS generation on a single RTX 5090—the first demonstration of true real-time performance for high-quality video generation.

Experimental Validation

The authors validate MonarchRT across multiple dimensions:

Sparsity vs Quality: MonarchRT maintains visual quality up to 95% sparsity on Self-Forcing, significantly outperforming sparse baselines (local attention, strided patterns, top-k) which degrade rapidly beyond 80% sparsity. Metrics include FVD (Fréchet Video Distance), temporal consistency, and human evaluation.

Comparison to Oracle: Even oracle top-k attention (with perfect knowledge of important positions) shows quality degradation at high sparsity, demonstrating that video attention requires structured expressivity beyond simple sparse selection.

Ablation Studies:

• Block size: Optimal at 16-32 for balancing expressivity and efficiency
• Tiling depth: 2-3 levels capture multi-scale dependencies effectively
• Fine-tuning: Essential for adapting Monarch structure to video-specific patterns

Generalization: MonarchRT transfers effectively across different video domains (natural scenes, synthetic content, human actions) and resolution scales.

Takeaways

1. Video attention in autoregressive few-step diffusion exhibits fundamentally different patterns than bidirectional image generation, combining periodic structure, dynamic semantics, and dense mixing that challenge simple sparse approximations.

2. Monarch matrix parameterization provides a structured middle ground between dense attention and naive sparsity, achieving 95% parameter reduction while maintaining the expressivity needed for high-quality video generation.

3. Custom kernel optimization is critical for realizing theoretical efficiency gains—MonarchRT’s Triton implementation achieves 1.4-11.8× speedups over FlashAttention variants across modern GPU architectures.

4. Real-time video generation (16 FPS) is now achievable on consumer hardware (RTX 5090) through the combination of efficient attention parameterization and optimized implementation.

5. The structured approach of MonarchRT opens new directions for efficient transformer architectures in other domains requiring complex attention patterns, particularly in autoregressive and real-time settings.