Geometric Latent Diffusion: Repurposing Foundation Models for Multi-view Synthesis

Paper: 2603.22275 Authors: Wooseok Jang, Seonghu Jeon, Jisang Han, Jinhyeok Choi, Minkyung Kwon, Seungryong Kim, Saining Xie, Sainan Liu Categories: cs.CV

The Gap

Novel view synthesis (NVS) needs to generate images from new camera angles that stay geometrically consistent — the same 3D point should look the same across views. Current diffusion models for NVS operate in VAE latent spaces, which compress images view-by-view without encoding cross-view geometric relationships. This creates a mismatch: the task demands geometric awareness, but the latent space is geometry-blind.

Prior work either trains diffusion directly in pixel space (slow, memory-hungry) or uses standard VAE latents (fast but geometrically inconsistent). The gap: no one asked whether existing geometric foundation models — already trained to extract geometrically consistent features — could serve as the latent space itself.

Problem: NVS needs geometric consistency across views
   |
   v
Observation: VAE latents are view-independent, ignore geometry
   |
   v
Hypothesis: Geometric foundation model features encode correspondences
   |
   v
Method: Use geometric features as diffusion latent space (GLD)
   |
   v
Evidence: 4.4x faster training, better 3D metrics than VAE
   |
   v
Conclusion: Geometric latents > generic compression latents for NVS

The Increment

One sentence: Before, multi-view diffusion used generic image compression latents; after, it uses geometry-aware feature spaces that already encode cross-view correspondences.

Core Mechanism

GLD has three components: a frozen geometric encoder (from DUSt3R or MASt3R foundation models), a lightweight decoder that reconstructs RGB from geometric features, and a diffusion model trained in this geometric latent space.

The geometric encoder takes multi-view images and outputs dense feature maps. These features aren’t just image descriptors — they’re trained to match across views when looking at the same 3D point. The decoder learns to invert this: given geometric features, reconstruct the RGB image. The diffusion model then learns to denoise in this geometric feature space, generating new views by sampling and decoding.

Data flows like this: input views → geometric encoder → feature maps → diffusion denoising → decoded RGB. The key trick: the encoder is frozen (pretrained), so training only touches the decoder and diffusion model. This is why it’s 4.4x faster than VAE approaches, which must train the encoder too.

Input views          Geometric encoder       Feature space
   |                  (frozen, pretrained)        |
   v                         |                    v
[img1, img2, ...]  --->  [feat1, feat2, ...]  <--- Diffusion
                             |                      model
                             v                      trains
                         Decoder                    here
                        (trainable)                 |
                             |                      v
                             v                  Denoising
                        RGB output              process

Think of it like using a translator’s brain as a compression algorithm. A professional translator doesn’t just memorize words — they build a mental model where “cat” in English and “猫” in Chinese point to the same concept. If you want to compress multilingual text, you could train a generic compressor from scratch, or you could repurpose the translator’s concept space. GLD does the latter: geometric foundation models are the “translator” that already learned to map different views to shared 3D concepts. Instead of training a new VAE to compress images, GLD borrows the geometric model’s feature space, which already knows that pixel (x1, y1) in view 1 and pixel (x2, y2) in view 2 might be the same 3D point. The decoder is like learning to “speak” from this concept space back to pixels, and the diffusion model learns to generate in this concept space where geometric relationships are already baked in.

Key Concepts

• Geometric latent space: Most latent spaces (like VAE) compress images independently — each image becomes a code, no relationship between codes. A geometric latent space encodes correspondences: if two features come from the same 3D point viewed from different angles, they’re similar in latent space. Imagine a warehouse where boxes are labeled by GPS coordinates instead of arbitrary IDs — you can tell which boxes contain items from the same location. Geometric features work similarly: they’re labeled by 3D position, not just image appearance.

• Frozen encoder repurposing: Training a VAE means learning both encoder (image → latent) and decoder (latent → image). GLD freezes a pretrained geometric encoder and only trains the decoder. Why does this work? The encoder already learned to extract geometrically meaningful features from millions of images. We just need to teach the decoder to invert this mapping. It’s like buying a high-end camera (encoder) and only learning to develop the film (decoder) — you don’t need to build the camera from scratch.

Framework Shift

Before (VAE-based NVS):              After (GLD):

View 1 ---> VAE encoder              View 1 ---> Geometric encoder
             |                                    (frozen, pretrained)
             v                                    |
         Latent code                              v
      (view-independent)                   Geometric features
             |                              (cross-view aware)
             v                                    |
      Diffusion model                             v
             |                              Diffusion model
             v                                    |
         VAE decoder                              v
             |                              Lightweight decoder
             v                                    |
         Output view                              v
                                             Output view

Key difference:                      Key difference:
Latent space has no                  Latent space encodes
geometric structure                  3D correspondences

From view-independent compression to geometry-aware compression, the core shift is embedding 3D structure directly into the latent space where generation happens.

Expert Assessment

Problem choice: Real gap. NVS is a hot area, and the VAE latent space mismatch is a genuine bottleneck — you’re asking a geometry-blind representation to maintain geometric consistency. The insight to repurpose geometric foundation models is clever, not manufactured.

Method maturity: This is elegant engineering, not a conceptual breakthrough. The heavy lifting (geometric feature learning) was done by DUSt3R/MASt3R. GLD’s contribution is recognizing these features can serve as latent space and showing it works. The 4.4x speedup comes mostly from freezing the encoder, which is a practical win but not a deep algorithmic innovation.

Experimental integrity: Baselines are fair (VAE, RAE, state-of-the-art NVS methods). The numbers hold up — 2D quality and 3D consistency metrics both improve. One concern: they compare against methods without text-to-image pretraining, then note GLD “remains competitive” with methods that do use it. This undersells the comparison — either compete directly or don’t claim competitiveness.

Writing quality: The abstract and intro are crisp. The method section could be tighter — there’s repetition about “geometric consistency” without showing failure cases where VAE latents break down. A single figure comparing VAE vs GLD latent space structure (e.g., t-SNE of features from the same 3D point across views) would make the geometric advantage visceral.

Verdict: weak accept — Solid engineering contribution with clear practical benefits, but the core idea is “use better features” rather than a new algorithmic principle. The speedup and quality gains justify publication, but this won’t reshape how people think about multi-view synthesis.

Takeaways

• Repurpose, don’t retrain: If a pretrained model already extracts the structure you need (here, geometric correspondences), freeze it and train only the decoder. This pattern applies beyond vision — language models’ embeddings, audio models’ spectrograms, etc.

• Latent space design matters: The choice of latent space isn’t just about compression ratio. For tasks with structural requirements (geometry, temporal coherence, physical constraints), use latents that encode that structure. Don’t force a diffusion model to learn structure that could be baked into the representation.

• Benchmark against the right baseline: GLD’s win over VAE is convincing. The comparison to text-to-image pretrained methods is muddled — either train GLD with pretraining or don’t claim competitiveness. When writing papers, pick one comparison story and commit.