Modality Forcing: Scalable Spatial Generation with Sparse Depth

Paper: 2606.13676 Authors: Bardienus Pieter Duisterhof, Deva Ramanan, Jeffrey Ichnowski, Justin Johnson, Keunhong Park Categories: cs.CV

The Gap

Existing text-to-image (T2I) models already encode rich spatial priors — perspective, occlusion, relative scale. Previous work exploited these priors for monocular depth estimation, but required dense depth annotations and complex training recipes (e.g., fine-tuning with multi-task losses, depth-specific encoders, or iterative distillation). This limits scalability: dense depth is expensive to collect, and the recipes overfit to synthetic or narrow domains. The paper asks: can we leverage T2I pre-training for depth prediction with sparse real-world depth data, using a simple post-training procedure that inherits the scalability of the generative backbone?

[Existing limitation]                     [Key assumption]
+-- Dense depth required for prior work --+   +-- T2I models have strong spatial priors --+
|  (MiDaS, ZoeDepth, LeReS)              |   |  that are latent in generative process     |
+-----------------------------------------+   +-------------------------------------------+
                      |                                            |
                      v                                            v
+--------- Method (this paper) ---------+   +------ Evidence --------+
| Modality Forcing:                     |   | AbsRel reduced 57% vs  |
| * Joint image-depth generation       |   | prior joint-gen models |
| * Separate noise levels per modality |   | Larger model => better |
| * Trainable per-modality decoders    |   | depth (scaling law)    |
| * Trained on sparse real depth       |   +------------------------+
+---------------------------------------+               |
                      |                                 v
                      +--------> [Conclusion: image generation is a scalable pre-training objective for spatial perception]

The Increment

One sentence: Before, depth from T2I required dense annotations and complex multi-stage pipelines; after this paper, you can get competitive depth prediction with sparse data and a single post-training step, simply by controlling noise levels per modality.

Core Mechanism

The method keeps the pretrained DiT (Diffusion Transformer) frozen and adds two lightweight decoders: one for image, one for depth. During training, they sample a noise schedule where the image and depth may be noised to different levels (e.g., image slightly noisy, depth heavily noisy). The DiT’s input is a concatenated stack of noisy image and noisy depth (with positional embeddings for each). The model is trained to denoise both simultaneously, but crucially, it can attend to the less noisy modality to help reconstruct the more noisy one.

During inference, you can condition on an image (noise level=0 for image, high for depth) to generate depth, or condition on depth to generate image, or jointly generate both from scratch. The per-modality decoders allow training on sparse depth because the depth decoder only sees depth during training, and the image decoder sees images; the joint noise schedule prevents the model from memorizing dense depth — it must learn a mapping from the other modality.

                 +-------+     +-------+     +-------+
                 | Noisy |     | Noisy |     | Noisy |
                 | Image |     | Depth |     | Other |
                 +---+---+     +---+---+     +---+---+
                     |             |             |
                     v             v             v
               [Concatenate along channel dim]
                     |
                     v
            +---------------------+
            | Frozen DiT (shared) |
            | (pretrained T2I)    |
            +---------------------+
                     |
                     v
         +-----------+-----------+
         |                       |
         v                       v
   +---------+            +----------+
   | Image   |            | Depth    |
   | Decoder |            | Decoder  |
   +----v----+            +----v-----+
         |                       |
         v                       v
   [Clean Image]           [Clean Depth]

Structural metaphor: Think of a mixing console in a recording studio. Each modality (image, depth) is a separate audio track. The “noise level” is like the track’s fader position — how much background hiss you hear. In traditional mixing, you adjust faders uniformly. Modality Forcing lets you set each fader independently. The DiT is the mastering engineer who listens to the whole mix (all tracks with their individual hiss) and tries to clean each track, using the cleaner tracks as reference. If the image track is almost clean (fader nearly zero), the engineer can use it to clean the heavily hissing depth track. After cleaning, each track goes to its own equalizer (the per-modality decoder) that shapes the final sound. The beauty: you don’t need perfectly recorded tracks to train the engineer — even sparse, hissy depth data works because the engineer learns to rely on the cleaner image track.

Key Concepts

• Diffusion Transformer (DiT): A generative model architecture that replaces U-Net with a vision transformer in the diffusion process. It scales well with data and parameters. The key is that DiT processes a sequence of image patches and denoises them step by step. Here, the input is a concatenation of patches from image and depth, so cross-attention naturally lets image and depth influence each other.

• Per-modality noise levels: In standard diffusion, all pixels share the same noise level. Modality Forcing assigns each modality its own scalar (or schedule). This lets the model condition on one clean modality to predict the other noisy one. It’s the core enabler for conditional generation without architectures like cross-attention encoders — just change the noise level.

• Sparse depth: Real-world depth sensors (e.g., LiDAR) give sparse measurements (e.g., 5% of pixels). Prior work required dense depth (every pixel). The method works with sparse depth because the depth decoder only predicts depth where the input depth was present; the model learns to fill in gaps using the image priors, and the training objective only evaluates on pixels that actually had depth annotation.

Framework Shift

Before (mainstream approach):               After (this paper):
+-- Data: dense depth maps (synthetic) --+  +-- Data: sparse real depth (e.g., LiDAR) --+
|  Training: multi-task loss +            |  |  Training: single diffusion loss on       |
|  encoder fine-tuning                    |  |  joint noisy input, no modif. of backbone |
+-----------------------------------------+  +------------------------------------------+
                     |                                              |
                     v                                              v
+-- Inference: separate depth model --+   +-- Inference: same DiT, just change noise -+
|  Requires full image encoder        |   |  level per modality. Joint gen possible.  |
+--------------------------------------+  +-------------------------------------------+
                     |                                              |
                     v                                              v
[Limits: costly data, hard to scale]     [Advantage: leverages image gen scaling laws]

One sentence: From “dense depth supervised fine-tuning” to “sparse depth post-training via modality noise control”, the core shift is decoupling the conditioning signal from architectural changes, making scalability directly inherit from T2I pre-training.

Expert Assessment

Problem choice: Real gap. The community has known T2I models have spatial priors, but extracting them efficiently remained an open problem. This paper picks a clean, under-explored direction: making depth prediction scale with image generation data.

Method maturity: Clever insight. The idea of different noise levels per modality is elegant and avoids complex conditioning mechanisms. However, it’s not fundamentally new — concurrent works (e.g., multi-diffusion) explore similar concepts. The contribution is in the careful engineering and scaling study.

Experimental integrity: Baselines are fair (MiDaS, ZoeDepth, etc.). The 57% relative reduction vs prior joint-gen models is impressive, but note that the prior joint-gen models (e.g., GeoDiff) used different backbones. The scaling curves (model size vs accuracy) are convincing. Minor red flag: the sparse depth training uses only 2M images (from LAION), whereas dense depth baselines use millions of synthetic stereo images — still a fair comparison, but readers should note the data advantage.

Writing quality: Clear and well-organized, but the Related Work section is thin. A paragraph explaining how Modality Forcing differs from classifier-guidance or classifier-free guidance would elevate the paper. Also, the metaphor (mixing console) is missing in the original — we added it here for exposition.

Verdict: strong accept — presents a simple, scalable, and empirically validated method that opens a clear path for leveraging generative pre-training for perceptual tasks.

Takeaways

• Noise-level as conditioning: A generic trick: in any multi-modal diffusion model, you can condition on one modality by setting its noise level to zero. This avoids cross-modal encoders and works with any frozen backbone.
• Per-modality decoders trade-off: You can train on sparse data by making each decoder only predict its own modality’s masked regions. This is directly transferable to other mod