GenWildSplat: Feed-Forward 3D Reconstruction from Wild Internet Photos

Paper: 2604.28193 Authors: Vinayak Gupta, Chih-Hao Lin, Shenlong Wang, Anand Bhattad, Jia-Bin Huang Categories: cs.CV

The Gap

Existing 3D reconstruction methods hit a wall when dealing with real-world internet photos: varying lighting across images, tourists walking through scenes, sparse viewpoints. Methods like NeRF-W and Ha-NeRF handle these issues through per-scene optimization—they learn appearance embeddings or dynamic masks specific to each location. This works, but requires hours of training per scene and collapses when you only have a handful of images. The evaluation problem compounds this: most papers test on 5-10 landmark scenes, leaving generalization as an open question.

The core tension: you need scene-specific knowledge to handle lighting variations and transients, but you can’t afford per-scene optimization if you want a practical system.

Problem: Wild photos (lighting + occlusions + sparse views)
    |
    v
Prior approach: Per-scene optimization
    |
    +---> Appearance embeddings (NeRF-W)
    +---> Dynamic masks (Ha-NeRF)
    |
    v
Limitation: Hours per scene, fails at <10 views
    |
    v
This paper: Feed-forward with learned priors
    |
    +---> Geometric priors (depth, pose)
    +---> Appearance adapter (lighting)
    +---> Semantic segmentation (transients)
    |
    v
Evidence: PhotoTourism + MegaScenes (100+ scenes)
    |
    v
Conclusion: Real-time inference, generalizes across scenes

The Increment

One sentence: Before this paper, handling wild photo variations required per-scene optimization; after, a single feed-forward network generalizes across scenes using learned priors.

Core Mechanism

GenWildSplat processes unposed images through three parallel streams. First, a depth and pose estimator predicts camera parameters and per-pixel depth using DUSt3R-style geometric priors. Second, an appearance adapter takes a reference image and target lighting condition, producing modulation parameters that adjust 3D Gaussian appearance. Third, semantic segmentation identifies transient objects (people, cars) to exclude from reconstruction.

These streams feed into a 3D Gaussian predictor that outputs Gaussians in a canonical space—a normalized coordinate frame independent of camera pose. The appearance adapter modulates Gaussian colors based on target lighting, while transient masks prevent temporary objects from polluting the geometry. At inference, you provide sparse images and a target lighting condition; the network predicts depth, solves for poses, generates Gaussians, and renders novel views—all in one forward pass.

Training uses curriculum learning: start with synthetic data (perfect geometry, controlled lighting), then mix in real PhotoTourism data (wild lighting, real occlusions). The curriculum teaches geometric priors first, then appearance adaptation, preventing the network from overfitting to scene-specific patterns.

Structural metaphor: Think of GenWildSplat as a film colorization studio that learned from thousands of movies. When you bring in black-and-white frames from a new film, the studio doesn’t need to study your specific movie for weeks. Instead, it applies learned priors: faces are usually flesh-toned, skies are blue, shadows follow light direction. The depth/pose estimator is the geometry department figuring out spatial layout. The appearance adapter is the color grading team adjusting for your target lighting (golden hour vs overcast). The semantic segmentation is the cleanup crew removing boom mics and crew members from shots. Each department learned general patterns from training data, so they handle your new film immediately without custom training.

Key Concepts

• Canonical space: Imagine you’re building a 3D model of the Eiffel Tower from tourist photos. Each photo has a different camera position and orientation—some from ground level, some from across the river, some from helicopters. Canonical space is like building the tower model in a fixed coordinate system (say, centered at the tower’s base, aligned with compass directions) rather than in each camera’s local coordinates. This matters because it decouples geometry from viewpoint: the network learns “what the Eiffel Tower looks like” independent of “where the camera was.” When a new photo arrives, you just figure out how that camera relates to the canonical frame, rather than rebuilding the whole scene. It’s the difference between having one master blueprint vs redrawing the building for every photographer’s perspective.

• Appearance adapter: Real-world photos of the same scene look different depending on time of day, weather, and season. An appearance adapter is a small neural network module that takes two inputs: (1) a reference image showing what lighting you want, and (2) the 3D Gaussians representing geometry. It outputs modulation parameters—essentially, “multiply these Gaussian colors by 1.2, shift those hues by 15 degrees”—that adjust the scene’s appearance to match the target lighting. The key insight: instead of baking lighting into the geometry (which would require retraining for each lighting condition), you separate geometry from appearance and learn a function that maps between lighting conditions. It’s like having a single 3D model of a building with a shader that can render it at dawn, noon, or dusk by adjusting parameters, rather than storing three separate models.

• Curriculum learning on synthetic-to-real: Training directly on messy real-world data often fails because the network latches onto spurious correlations (e.g., “tourists always appear near landmarks, so include them in geometry”). Curriculum learning stages the training: start with clean synthetic data where ground truth is perfect—you know exact depth, exact lighting, exact transient locations. The network learns fundamental geometric relationships without noise. Then gradually introduce real data with its messiness. By this point, the network has strong geometric priors, so it interprets real-world ambiguities correctly rather than memorizing scene-specific quirks. It’s like teaching someone to drive: start in an empty parking lot (synthetic), then quiet streets (mixed), then rush hour (real). Each stage builds on skills from the previous one.

Framework Shift

Before (per-scene optimization):        After (feed-forward generalization):

Input: Photos of Scene A                Input: Photos of Scene A
    |                                       |
    v                                       v
Initialize scene-specific                Load pretrained network
embeddings + masks                       (trained on 1000s of scenes)
    |                                       |
    v                                       v
Optimize for hours:                      Single forward pass:
  - Fit appearance codes                   - Predict depth/pose
  - Learn transient masks                  - Apply appearance adapter
  - Adjust geometry                        - Segment transients
    |                                       |
    v                                       v
Scene A model (non-transferable)         Scene A reconstruction
                                            |
[Repeat full process for Scene B]          v
                                         Input: Photos of Scene B
                                            |
                                            v
                                         Same network, instant result

From per-scene fitting to cross-scene generalization, the core shift is replacing optimization with learned priors that transfer.

Expert Assessment

Problem choice: Real gap. The per-scene optimization bottleneck has blocked practical deployment of neural 3D reconstruction for years. PhotoTourism has 100+ scenes, but most papers cherry-pick 5-10 for evaluation—this paper actually tests on the full benchmark, which is overdue. The sparse-view angle is also genuine: tourist photo collections are inherently sparse and unstructured.

Method maturity: Solid engineering rather than conceptual breakthrough. The components (DUSt3R for geometry, appearance modulation, semantic segmentation) are established techniques; the contribution is integrating them into a feed-forward pipeline with curriculum learning. The canonical space formulation is clean. However, the appearance adapter architecture isn’t deeply motivated—why this specific modulation scheme over alternatives? Feels like the first thing that worked rather than a principled design.

Experimental integrity: Baselines are fair (NeRF-W, Ha-NeRF, pixelSplat). The PhotoTourism and MegaScenes evaluations are comprehensive. Numbers look credible. One concern: the paper claims “real-time inference” but doesn’t report actual frame rates or compare inference time against baselines. “Real-time” might mean 10 FPS or 0.1 FPS—the ambiguity is suspicious. Also, the curriculum learning ablation is thin: they show it helps, but don’t isolate which stage (synthetic pretraining vs mixing ratio) matters most.

Writing quality: The method section is dense and assumes familiarity with 3D Gaussians and DUSt3R. A reader unfamiliar with these would struggle. The related work section does a good job positioning against prior work. The results section front-loads qualitative comparisons (good) but buries quantitative tables (bad—put numbers up front). The biggest weakness: no failure case analysis. When does GenWildSplat break? Extremely sparse views? Extreme lighting changes? The paper is silent on limitations, which undermines trust.

Verdict: weak accept — Solves a real problem with solid engineering and comprehensive evaluation, but lacks conceptual depth and honest limitation discussion.

Takeaways

Canonical space for multi-view tasks: If you’re building any system that processes multiple views of a scene (SLAM, multi-view stereo, video understanding), consider predicting into a canonical coordinate frame rather than camera-relative coordinates. It decouples geometry from viewpoint and makes learned representations more transferable. The cost is solving for the canonical-to-camera transform, but that’s often easier than learning viewpoint-invariant features.

Appearance modulation over baked appearance: When your data has appearance variations (lighting, weather, style), don’t bake appearance into your representation. Instead, learn a base representation and a modulation function. This applies beyond 3D: image editing, video generation, style transfer. The pattern is: separate content from style, learn a mapping between styles.

Curriculum from synthetic to real: If your real-world data is noisy and your task has clear ground truth in simulation, start training on synthetic data to learn fundamental structure, then transition to real data. The synthetic stage acts as a regularizer, preventing overfitting to real-world spurious correlations. This works when synthetic data captures the core task structure even if it doesn’t match real data distribution perfectly.