GenRecon: Scaling Generative 3D Priors to Multi-View Scene Reconstruction

Paper: 2605.23888 Authors: Katharina Schmid, Nicolas von Lützow, Jozef Hladký, Angela Dai, Matthias Nießner Categories: cs.CV

The Gap

Multi-view 3D reconstruction has reached a point where methods can produce geometrically consistent meshes from posed RGB images. Neural radiance fields (NeRFs) and Gaussian splatting excel at novel view synthesis but struggle with complete, watertight geometry. Traditional multi-view stereo (MVS) produces meshes but lacks detail in textureless or occluded regions. Meanwhile, generative 3D models like Trellis have learned rich priors from large-scale 3D datasets, producing high-fidelity object meshes with complete geometry and PBR materials. The gap: these generative priors operate at object scale and don’t naturally extend to multi-view, scene-scale reconstruction where spatial extent is large and input views are sparse and irregularly distributed.

Problem: Object-level generative priors =/=> Scene-level reconstruction
   |
   v
Assumption: Chunk scenes into overlapping regions, condition generation
            on multi-view features via projection
   |
   v
Method: Spatially-localized generation + projection-based conditioning
   |
   v
Evidence: 16% improvement over baselines, high-fidelity PBR meshes
   |
   v
Conclusion: Generative priors scale to scenes when properly conditioned

The Increment

One sentence: Before this paper, generative 3D priors were confined to single objects; after, they can be applied to multi-view scene reconstruction by chunking space and projecting image features into a coherent 3D representation.

Core Mechanism

GenRecon operates in two stages. First, it tiles the scene into overlapping 3D chunks, each small enough for a generative model to handle. Second, for each chunk, it conditions the generative model (Trellis.2) using a projection-based mechanism: multi-view image features are lifted into a 3D voxel grid aligned with the chunk, then fed as conditioning input to the generative model’s denoising process.

The projection mechanism works by extracting features from each input view using a 2D encoder, then projecting these features into 3D space using camera poses. Features from multiple views are aggregated at each voxel location, creating a spatially-anchored 3D feature volume. This volume is independent of view ordering and provides dense conditioning even in regions visible to only a few cameras. The generative model then produces a structured latent representation (SLAT) for the chunk, which decodes into a mesh with PBR materials.

Input: Multi-view RGB images + camera poses
   |
   v
[2D Feature Extraction] ---> Per-view feature maps
   |
   v
[Projection + Aggregation] ---> 3D feature volume (chunk-aligned)
   |
   v
[Conditional Generation] ---> SLAT latent (via Trellis.2)
   |
   v
[Decoding] ---> PBR mesh for chunk
   |
   v
[Chunk Merging] ---> Full scene mesh

Think of it like assembling a large mosaic from photographs. You can’t print the entire mosaic at once—your printer (the generative model) only handles letter-sized pages (chunks). So you divide the mosaic into overlapping tiles. For each tile, you don’t just guess what should be there; you project the relevant parts of your reference photos onto that tile’s area, blending information from multiple angles. The printer then fills in the tile using both the projected photo evidence and its learned understanding of what mosaics should look like (the generative prior). Finally, you stitch the tiles together, using the overlaps to ensure seamless joins. The key insight: the projection step anchors the generative process to actual observations, preventing hallucination while preserving the prior’s ability to complete occluded regions.

Key Concepts

• Projection-based conditioning: Instead of encoding multi-view images into a global latent code (which loses spatial information) or processing views sequentially (which creates order dependence), GenRecon projects 2D image features into 3D space using known camera geometry. Each voxel in the 3D volume accumulates features from all views that see that location. This creates a spatially-grounded conditioning signal: the generative model knows not just “what the scene looks like” but “what each specific region looks like from multiple viewpoints.” It’s the difference between describing a building from memory versus pointing at it from several street corners—the latter grounds your description in physical space.

• Structured Latent (SLAT): Trellis.2 represents 3D shapes as SLATs—latent codes organized in a spatial grid rather than a single global vector. Each grid cell encodes local geometry and appearance. This structure is crucial for scene-scale generation: you can condition different spatial regions independently, generate them in parallel, and merge them coherently. A global latent would force the entire scene through a single bottleneck; SLAT lets you scale by dividing and conquering.

• Chunk-based generation: Large scenes exceed the spatial extent that generative models can handle in one shot (both in terms of memory and learned prior distribution). GenRecon divides the scene into overlapping 3D chunks, generates each independently, then merges them. The overlap ensures smooth transitions—like shingles on a roof, each chunk covers part of its neighbor’s territory, allowing blending at boundaries. This scales generation from object-level (1-2 meters) to room-level (10+ meters) without retraining the base model.

Framework Shift

Before (mainstream approach):        After (this paper):

Multi-view images                    Multi-view images
       |                                    |
       v                                    v
  [MVS / NeRF]                    [Feature Projection]
       |                                    |
       v                                    v
  Geometry only                    3D Feature Volume
  (no strong prior)                        |
       |                                    v
       v                          [Chunked Generation]
  Incomplete mesh                   (with generative prior)
                                           |
                                           v
                                   Complete PBR mesh

[One sentence: From direct geometric reconstruction without priors to conditioning generative priors on multi-view observations, the core shift is treating reconstruction as spatially-grounded conditional generation rather than pure geometry estimation.]

Expert Assessment

Problem choice: This is a real gap. Generative 3D models have demonstrated impressive object-level quality, but their application to multi-view reconstruction has been limited. The problem sits at the intersection of two mature subfields (MVS and generative modeling) that haven’t been tightly coupled before. The timing is right—generative models are now good enough to provide useful priors, and the question of how to scale them to scenes is natural.

Method maturity: The approach is well-engineered rather than conceptually novel. Chunking is a standard scaling strategy, and projection-based feature aggregation has precedent in multi-view stereo. The contribution is in the integration: showing that you can condition a generative model this way and get coherent results. The method is somewhat brute-force (generate many chunks, merge them), but given the constraints of existing generative models, it’s a reasonable path. A simpler approach might be to fine-tune the generative model on scene-scale data, but that requires new datasets and training infrastructure.

Experimental integrity: The 16% improvement claim is based on standard metrics (Chamfer distance, F-score) against recent baselines (SuGaR, 2DGS, GOF). The baselines are fair—they represent the current state of multi-view reconstruction. However, the paper doesn’t deeply analyze failure modes or provide ablations on key design choices (e.g., chunk size, overlap amount, aggregation strategy). The results look strong visually, but quantitative analysis is somewhat surface-level. No red flags, but more rigor would strengthen the claims.

Writing quality: The paper is clearly written but front-loads technical details without sufficient motivation. The introduction jumps quickly into method description before establishing why projection-based conditioning is necessary. The related work section is thorough but could better position the work relative to concurrent efforts in scene-scale generation. The experiments section would benefit from failure case analysis—when does the method struggle? The supplementary material likely contains more detail, but the main paper should be more self-contained.

Verdict: weak accept — Solid engineering contribution that successfully bridges object-level generative priors to scene-scale reconstruction, with convincing results, but lacks conceptual novelty and deep experimental analysis.

Takeaways

Projection as a conditioning mechanism: When you have a generative model trained on one domain (objects) and want to apply it to another (scenes), don’t just feed it global features—project observations into the model’s native spatial structure. This pattern transfers: if you’re adapting a 2D generative model to video, project frame features into a 3D spacetime volume; if you’re adapting a text model to structured documents, project sentence embeddings into a document layout grid.

Chunking with overlap for spatial coherence: When scaling generation beyond the model’s training distribution, divide the output space into overlapping regions and blend at boundaries. The overlap size is a hyperparameter that trades off redundancy (more computation) against seam artifacts (less blending). This applies to any spatial generation task: large images, long videos, extended audio.

Structured latents for scalability: If you’re designing a generative model for data with spatial or temporal extent, avoid global latent codes. Use spatially-organized latents (grids, octrees, hash tables) so you can condition and generate different regions independently. This architectural choice determines whether your model can scale beyond its training resolution.