One View Is Enough: Training Novel View Synthesis on Unpaired Internet Images

Paper: 2603.23488 Authors: Adrien Ramanana Rahary, Nicolas Dufour, Patrick Perez, David Picard Categories: cs.CV

The Gap

Existing novel view synthesis methods (NeRF, 3D Gaussian Splatting, diffusion-based approaches) require multi-view image pairs during training. This creates a data bottleneck: you need carefully captured multi-view datasets, which limits scale and diversity. Methods like Zero-1-to-3 try to work around this with synthetic data or fine-tuning, but still depend on paired supervision or slow 3D optimization at inference.

The core question: can we train a model to generate novel views using only single images from the internet, without any multi-view pairs?

Problem: Multi-view pairs needed
    |
    v
Assumption: Monocular depth can create pseudo-pairs
    |
    v
Method: Lift → Transform → Project + Masked Training
    |
    v
Evidence: 30M unpaired images → SOTA zero-shot performance
    |
    v
Conclusion: Paired supervision is unnecessary

The Increment

One sentence: Before this paper, novel view synthesis required multi-view training data; after, you can train on billions of single internet images and generate views 600x faster.

Core Mechanism

OVIE uses a two-stage pipeline. At training time, it takes a single image, estimates its depth using a frozen monocular depth estimator (DINOv2), lifts the image into a 3D point cloud, applies a random camera transformation, and projects back to 2D to create a “pseudo-target” view. This pseudo-target has holes (disocclusions) where geometry was hidden in the source view.

The key innovation is masked training: OVIE computes a validity mask that identifies which pixels in the pseudo-target are geometrically sound. It applies losses (geometric, perceptual, textural) only to valid regions, ignoring disoccluded areas. This lets the model learn from imperfect pseudo-targets without being corrupted by invalid pixels.

At inference, OVIE discards all 3D machinery. It’s a pure 2D diffusion model conditioned on source image and target camera pose, generating novel views directly without depth estimation or 3D reconstruction.

Training:
Source Image → Depth Est. → 3D Lift → Camera Transform
                                            |
                                            v
                                    Project to 2D
                                            |
                                            v
                                    Validity Mask
                                            |
                                            v
                            Masked Loss ← Pseudo-Target

Inference:
Source Image + Target Pose → Diffusion Model → Novel View
(no depth, no 3D)

Think of OVIE like a chef learning to cook by watching cooking videos. During training (learning phase), the chef uses a recipe book (depth estimator) as scaffolding: they imagine rotating the dish (camera transform), predict what it would look like from another angle (pseudo-target), but only trust the parts they can actually see (validity mask). They practice on millions of dishes from random YouTube videos (unpaired images). At serving time (inference), the chef throws away the recipe book and cooks from memory alone—they’ve internalized the 3D understanding and can generate new views directly without geometric crutches.

Key Concepts

• Pseudo-target generation: You have one image. You want to train a model that generates views from other angles, but you don’t have those other angles. Solution: use depth to fake them. Lift the image into 3D using estimated depth, rotate the camera, project back. The result is noisy and has holes, but it’s a training signal. It’s like tracing a 2D drawing onto tracing paper, rotating the paper, and projecting it back—you get a rough approximation of what the rotated view should look like.

• Validity masking: When you rotate and reproject, some pixels become invalid: they were occluded in the source view, or the depth estimate was wrong. If you train on these pixels, you teach the model garbage. The validity mask identifies trustworthy pixels using geometric checks (depth consistency, reprojection error). Only these pixels contribute to the loss. It’s like learning to draw faces from photos where some faces are partially covered—you only learn from the visible parts, ignoring the covered regions.

• Geometry-free inference: Training uses depth as a teacher, but inference doesn’t. The diffusion model learns to internalize 3D reasoning without explicit 3D representation. This is why it’s fast: no depth estimation, no point cloud rendering, just direct image generation. It’s like learning to ride a bike with training wheels, then removing them—the training wheels (depth) helped you learn balance (3D understanding), but you don’t need them anymore.

Framework Shift

Before (mainstream approach):        After (this paper):

Multi-view Dataset                   Single-view Dataset
    |                                    |
    v                                    v
Paired Supervision                   Depth Estimator (training only)
    |                                    |
    v                                    v
3D Reconstruction                    Pseudo-target + Mask
    |                                    |
    v                                    v
Novel View Rendering                 Diffusion Model
    |                                    |
    v                                    v
Slow (3D optimization)               Fast (direct generation)

From paired-supervised 3D reconstruction to self-supervised 2D generation with geometric scaffolding, the core shift is decoupling training geometry from inference speed.

Expert Assessment

Problem choice: This is a real gap. Multi-view data is genuinely scarce and expensive to collect. The field has been stuck scaling on synthetic data or small real-world datasets. Unlocking internet-scale single images is a legitimate unlock. The problem sits at the intersection of 3D vision and generative models, a hot area with practical applications (AR/VR, e-commerce).

Method maturity: The core idea—use depth to create pseudo-pairs—is not novel (prior work in self-supervised depth estimation does this). The contribution is making it work at scale with masked training and showing you can discard geometry at inference. It’s more engineering insight than algorithmic breakthrough. The masked loss formulation is straightforward but effective. No obvious simpler approach is being overlooked; the method is well-motivated.

Experimental integrity: Baselines are fair (Zero-1-to-3, ViewCrafter, etc.). The 600x speedup claim is legitimate but somewhat cherry-picked—it compares against methods that do 3D optimization, not other direct generation methods. Zero-shot evaluation on RealEstate10K and ACID is appropriate. Ablations are thorough (mask components, depth estimator choice, dataset scale). One red flag: no user study or perceptual metrics beyond LPIPS/FID. Generated samples look good but could use human evaluation.

Writing quality: The paper is clear and well-structured. The abstract oversells slightly (“one view is enough” is catchy but imprecise—you still need depth estimation during training). Section 3.2 (masked training) could be more concise; the validity mask computation is described in excessive detail. The related work section is comprehensive but could better position the work relative to self-supervised depth methods.

Verdict: weak accept — Solid engineering contribution that removes a real bottleneck, but the novelty is incremental and the evaluation could be more rigorous on perceptual quality.

Takeaways

Masked training for noisy supervision: When your training signal is imperfect (pseudo-labels, weak annotations), compute a confidence mask and apply losses only to high-confidence regions. This is broadly applicable beyond novel view synthesis—think semi-supervised learning, noisy label handling, or any scenario where you have cheap but unreliable supervision.

Scaffolding at training, not inference: Use expensive operations (depth estimation, 3D reconstruction) as training-time teachers, then discard them at inference. The model learns to internalize the structure without needing the crutch. This pattern appears in knowledge distillation but is underused in 3D vision.

Scale matters more than pairing: 30M unpaired images outperform smaller paired datasets. If you can generate pseudo-supervision at scale, data quantity can compensate for label quality. This challenges the conventional wisdom that paired data is always superior.