VideoRAE: Why Learn Video Latents from Scratch When Foundation Models Already Know So Much?

Paper: 2607.14088 Authors: Zhihao Xie, Junfeng Wu, Xinting Hu, Junchao Huang, Li Jiang Categories: cs.CV

The Gap

Video generation has exploded — Sora, Kling, LTX, you name it. But nearly all these systems share the same hidden assumption: learn a 3D Variational Autoencoder (3D-VAE) from scratch, optimized for pixel-level reconstruction, then build your generator on top of its latent space. The problem? Pixel-level reconstruction ≠ semantic fidelity. These VAEs often blur fine details, lose temporal coherence, or produce latents that are geometrically awkward for downstream generators (diffusion transformers, autoregressive models).

Meanwhile, the video understanding community has been building increasingly powerful Video Foundation Models — V-JEPA 2, VideoMAEv2 — that encode rich spatio-temporal semantics. But nobody has seriously asked: can these frozen understanding models serve as the backbone for generation? The gap is architectural: there’s no bridge between “models that understand video” and “models that generate video.”

VideoRAE builds that bridge. It takes a frozen VFM encoder, compresses its multi-scale features through a lightweight 1D self-attention projector, and trains a decoder with a representation alignment loss. The result: latents that are both semantically rich and generation-friendly, supporting both continuous (DiT) and discrete (AR) regimes.

Traditional Path:
[Learn 3D-VAE from scratch] --> [Pixel recon loss only] --> [Semantic-poor latents] --> [Slow generator convergence]

VideoRAE Path:
[Frozen VFM encoder (strong semantics)] --> [Lightweight 1D projector (compression)] --> [Decoder + rep alignment loss] --> [Rich latents, 5x faster convergence]

Gap being filled:
[VFMs exist] --[no bridge]--> [video generation]
VideoRAE: [VFMs] --[representation autoencoder]--> [generation-ready latents]

The Increment

One sentence: Before this paper, video generators had to learn their own latent spaces from scratch using pixel-reconstruction losses; after this paper, they can piggyback on frozen video foundation models that already understand spatio-temporal structure — getting better latents 5x faster.

Core Mechanism

VideoRAE has three main components working in sequence. First, a frozen VFM encoder (V-JEPA 2 or VideoMAEv2) processes the input video and produces multi-scale hierarchical features — think of these as rich, pre-computed “understanding” of the video at different spatial and temporal granularities. These features are frozen; no gradients flow back into the encoder.

Second, a lightweight 1D self-attention projector compresses these multi-scale features into a compact latent representation. This is where the engineering happens: the projector takes the high-dimensional VFM features and maps them down to a dimensionality suitable for generation (continuous vectors for diffusion, or discrete tokens for autoregressive models). The “1D” part means it treats the spatio-temporal features as a flat sequence and applies self-attention along that dimension — simple but effective for capturing global structure during compression.

Third, the decoder reconstructs the video from the compressed latents. But here’s the twist: instead of only using pixel-level reconstruction loss, VideoRAE adds a local-and-global representation alignment objective. The decoder’s intermediate features are aligned with the frozen VFM teacher’s features at both local (patch-level) and global (sequence-level) scales. This means the decoder doesn’t just need to get the pixels right — it needs to produce representations that “look like” the VFM’s understanding of the video. This alignment replaces the KL divergence regularization typically used in VAEs, which the authors argue is unnecessary when you have a strong teacher signal.

For discrete tokenization, VideoRAE uses multi-codebook high-dimensional quantization: instead of a single codebook, multiple codebooks operate on different sub-dimensions of the latent, enabling richer discrete representations without exploding codebook size.

Data Flow:

[Input Video]
      |
      v
[Frozen VFM Encoder] (V-JEPA 2 or VideoMAEv2)
      |
      | multi-scale features: [L1, L2, L3, ...]
      v
[1D Self-Attention Projector]
      |
      | compressed latents
      +---> [Continuous path] --> [Diffusion Transformer (DiT)]
      |
      +---> [Discrete path] --> [Multi-Codebook Quantization] --> [Autoregressive (AR) model]
      
[Decoder] <-- [latents (continuous or discrete)]
      |
      | reconstructed video
      v
[Losses]
      |--- Pixel reconstruction loss
      |--- Local rep alignment (patch-level, with frozen VFM)
      |--- Global rep alignment (sequence-level, with frozen VFM)
      (No KL divergence needed)

Here’s a structural metaphor that makes this click: imagine you’re a music producer. The traditional 3D-VAE approach is like training a new audio engineer from scratch — you hire someone with no experience and teach them to transcribe music purely by comparing their sheet music to the original waveform, note by note. They’ll eventually get the notes right, but they might miss the feel, the phrasing, the emotional arc.

VideoRAE is different. You hire a seasoned conductor (the frozen VFM) who already deeply understands music. You then pair them with a young transcriptionist (the lightweight projector) who listens to the conductor’s detailed analysis and learns to write compact lead sheets. The decoder is the orchestra that plays from those lead sheets. But instead of just checking “did the orchestra play the right notes?” (pixel loss), you also ask the conductor “does the orchestra’s performance feel like what you understood from the original?” (representation alignment). The conductor’s feedback (frozen VFM features) ensures the performance captures not just the notes but the musicality. And because the conductor’s understanding is so rich, the transcriptionist learns much faster — 5x faster convergence — than an engineer who only had waveform comparisons to learn from.

Key Concepts

• Representation Autoencoder (vs. VAE): A traditional VAE learns an encoder-decoder pair where the latent space is regularized by KL divergence to match a prior (usually Gaussian). The problem: this regularization can actually fight against semantic richness, forcing information into a shape that’s convenient for sampling but bad for reconstruction. A representation autoencoder replaces the KL prior with a representation alignment loss — instead of saying “your latents must look like a Gaussian,” it says “your decoded video must produce features similar to a strong teacher model.” Concretely: if you’re compressing a video of a dog catching a frisbee, a VAE might lose the precise moment of the catch; a representation autoencoder preserves it because the teacher model “notices” that moment as semantically important.

• 1D Self-Attention Projector: Think of this as a bottleneck with brains. A naive compression would just pool or average features (losing structure). A linear projection would compress but ignore relationships between different parts of the video. The 1D self-attention projector flattens the spatio-temporal features into a sequence, then lets each position attend to every other position before projecting down. This means when compressing, a patch in frame 50 can “consult” with a patch in frame 10 to decide what information to keep. The “1D” refers to treating the entire video as one long sequence, rather than maintaining a 2D spatial structure — a deliberate simplification that makes the projector fast and parameter-efficient.

• Multi-Codebook High-Dimensional Quantization: Standard vector quantization (like VQ-VAE) uses one codebook — say 8192 entries in 256 dimensions. The problem: 8192 isn’t enough to capture the diversity of video content, but making it bigger is computationally expensive. Multi-codebook quantization splits the latent vector into multiple sub-vectors, each quantized by its own codebook. If you have 4 codebooks each with 2048 entries, you get 2048^4 ≈ 1.8 × 10^13 possible combinations — vastly richer than a single large codebook. It’s like the difference between having one alphabet of 26 letters versus four alphabets of 26 letters each that can be combined in parallel.

Framework Shift

Before (mainstream approach):            After (this paper):

[Raw video]                              [Raw video]
      |                                        |
      v                                        v
[Train 3D-VAE from scratch]             [Frozen VFM encoder]
[Pixel recon loss only]                       |
[KL regularization to Gaussian]         [Multi-scale features]
      |                                        |
      v                                        v
[Latents: pixel-optimized]              [1D self-attn projector]
[May lose semantics]                          |
      |                                   [Compact latents]
      v                                        |
[Generator struggles, slow]             +------+------+
                                        |             |
                                   [Continuous]  [Discrete]
                                   (for DiT)     (multi-codebook
                                                  for AR)
                                        |
                                        v
                                   [Decoder + rep alignment
                                    with frozen VFM teacher]
                                        |
                                        v
                                   [Semantically rich latents]
                                   [5x faster convergence]

From “learn everything from scratch with pixel-level supervision” to “stand on the shoulders of frozen understanding models and only learn the translation to generation,” the core shift is treating video understanding models as reusable infrastructure rather than isolated tools for recognition tasks.

Expert Assessment

Problem choice: This is a genuine and timely gap. The video generation community has been locked into the “train your own VAE” paradigm, while the video understanding community has built increasingly powerful foundation models. The observation that nobody has seriously tried to bridge these two worlds is accurate — V-JEPA 2 and VideoMAEv2 are relatively recent, and the idea of using their frozen representations for generation is underexplored. This sits at a natural intersection point in the field’s trajectory: as foundation models get better, their representations become more valuable, and finding ways to reuse them for generation is an obvious but non-trivial question.

Method maturity: The approach is clever but not revolutionary — it’s more “elegant engineering insight” than “fundamental new principle.” The 1D self-attention projector is simple and the representation alignment loss is straightforward. One could argue that a simpler baseline (just fine-tuning a pretrained VFM decoder without the projector) might work nearly as well, and the paper doesn’t fully explore this ablation. The multi-codebook quantization is borrowed from prior work (VQ-VAE variants). That said, the combination is well-motivated and the frozen-encoder + alignment-loss design is clean.

Experimental integrity: The baselines are reasonably fair — they compare against established 3D-VAE approaches. The UCF-101 results are strong (40 and 93 gFVD for AR and DiT respectively), and the 5x convergence speedup is compelling. The 2B-scale text-to-video experiment replacing LTX-VAE is a good practical validation. However, UCF-101 is a relatively simple benchmark (101 action classes, short clips); the paper would benefit from results on more complex, longer-form video generation tasks. The “controlled 2B-scale study” sounds like it was designed carefully, but details on what “comparable settings” means exactly could be more transparent.

Writing quality: The paper reads clearly and the motivation is well-articulated. The weakest section is the related work — it could do more to position against concurrent efforts (are others working on VFM-to-generation bridges?). The ablation studies could be deeper: what happens when you remove the global alignment loss? When you use a deeper projector? When you don’t freeze the VFM? These are questions a careful reader will have, and the paper partially addresses them but not exhaustively.

Verdict: weak accept — The core insight (frozen VFMs as generation backbones) is sound and practically valuable, with solid experimental support, but the method novelty is incremental rather than fundamental, and the evaluation could be broader.

Takeaways

Three concrete things to steal:

1. Frozen understanding models as generation infrastructure: The paradigm of “don’t retrain your encoder for generation — just adapt the interface” transfers directly. If you’re working on image, audio, or 3D generation, ask: is there a strong foundation model whose representations I can reuse instead of training an encoder from scratch? This saves compute and often gives better representations.

2. Representation alignment as a replacement for KL regularization: Instead of forcing your latent space to match a simple prior (Gaussian), align your decoder’s intermediate features with a strong teacher. This is applicable to any autoencoder-based pipeline where you have access to a good feature extractor. The key insight: pixel loss + semantic alignment loss > pixel loss + KL divergence.

3. Multi-codebook quantization for discrete tokenization: If you need discrete tokens (for AR models, retrieval, or compression), splitting into multiple small codebooks is strictly better than one large codebook in terms of representational capacity per parameter. This is a simple technique that’s underused outside the video domain.