VisDom: Sparse Novel View Synthesis with Visible Domain Constraint

Paper: 2606.20531 Authors: Mariia Gladkova**, Tarun Yenamandra*, Edmond Boyer, Robert Maier, Tony Tung, Daniel Cremers Categories: cs.CV

The Gap

Existing NeRF and Gaussian Splatting (GS) methods work well with dense views but collapse in sparse settings: they overfit to silhouette consistency, producing floating floaters and hollow geometry. Silhouette carving alone is ambiguous because any region that projects inside all silhouettes — even empty space — passes the test. Prior regularizers (depth, smoothness) are either data-hungry or add learned parameters that still fail to enforce actual occupancy. The field needed a cheap, deterministic, and theoretically grounded geometric prior that tightens the reconstruction hull without extra training.

[ASCII logic topology]

Problem: Sparse views -> 3D geometry underdetermined
            |
            v
Assumption: Silhouette consistency = object occupancy  ---> WRONG (overestimates)
            |                                                 |
            v                                                 v
Method: VisDom = silhouette carving + min K-view visibility constraint
            |
            v
Evidence: VisDom improves NeRF (DVGO) & GS (3DGS, GaussianObject) 
            on 3 datasets (Omni3D, MipNeRF360, DTU). 
            | 
            v
Conclusion: Learning-free filter reduces floaters; works from 4 views.

The Increment

One sentence: Before VisDom, sparse NVS relied on silhouettes (or learned priors) that overestimate occupancy; after VisDom, any point must be seen by at least K cameras to survive, producing a tighter hull with fewer artifacts.

Core Mechanism

VisDom operates as a pre-processing geometric filter. Given input images with known cameras and silhouettes, it builds a voxel grid in space and labels each voxel by how many cameras observe its projection inside the silhouette. Only voxels seen by >= K cameras (typically K=3) are kept; the rest are discarded. This “visible domain” is the intersection of the pure visual hull (all silhouette-consistent points) with a multi-view visibility mask. Crucially, it requires no learned parameters — just ray-casting and counting.

The visible domain then constrains downstream NVS pipelines in two ways. For implicit methods (NeRF), it restricts volumetric sampling: only sample points inside the domain, so the network never wastes capacity on floaters outside the real object. For explicit methods (Gaussian Splatting), it provides an initialization mask: only place Gaussians inside the domain, and optionally prune those that drift outside during optimization. The constraint is applied at inference time and does not modify the training loss; it’s a hard geometric prior.

[ASCII diagram of method internals]

Input: silhouettes + cameras (N views)
    |
    v
+---------------------------+
|   Voxel grid (fine)       |
|   For each voxel:         |
|     ray-cast to each cam  |
|     count silhouette hits |
+---------------------------+
    |
    v
+---------------------------+
|   Visible Domain =        |
|   {voxels with count>=K}  |
+---------------------------+
    |
    v
Apply to NeRF or GS:
   NeRF: restrict sample volume
   GS  : initialize Gaussians inside domain

Structural metaphor: Think of VisDom as a coffee filter. The ground truth solid object is the coffee grounds. Silhouette carving is a coarse sieve that lets through all large chunks (even clumps of air bubbles that appear solid from some angles). VisDom adds a finer mesh: “must be seen by at least K cameras” — that’s like requiring each ground particle to be visible from multiple pour angles. Particles that float in silhouette-consistent air space (e.g., a peak seen from only one angle) get filtered out. The remaining grounds form a tighter, more realistic puck. The downstream NeRF/GS brewer then only works with this compact puck, avoiding the murky floaters that cloud the cup.

Key Concepts

• Visual hull: The maximal shape that projects inside all silhouettes. It’s the intersection of all viewing frustums back-projected from silhouette masks. Problem: it overestimates volume because concave regions or disconnected supports are included. VisDom shrinks this hull by requiring each point to be *visible from enough views.
• Multi-view visibility count: For a 3D point, the number of cameras that both (a) have the point in their field of view and (b) project the point inside the silhouette. A point behind a surface from one view is “observed” but not inside the silhouette — it gets zero count. This count is purely geometric: no depth map or learning needed.
• K-threshold: The minimum number of cameras that must contain the point in their silhouette. Low K (e.g., K=2) still allows some floaters; high K (e.g., K=4 from 4 input views) is very restrictive. The paper recommends K=3 as a sweet spot that preserves most real geometry while cutting hallucinations.

Framework Shift

Before (mainstream approach):        After (this paper):
+-------------------+                +-------------------+
| Silhouette mask   |                | Silhouette mask   |
| + carving         |                | + carving         |
| |                 |                | |                 |
| v                 |                | v                 |
+-------------------+                +-------------------+
        |                                     |
        v                                     v
+--------------------------+      +---------------------------+
| Occupancy / density      |      | Visible domain filter     |
| (learned, may overfit)   |      | (hard geometric prior)   |
+--------------------------+      +---------------------------+
        |                                     |
        v                                     v
+--------------------------+      +---------------------------+
| NeRF or GS optimization  |      | NeRF or GS optimization   |
| over entire volume       |      | only inside visible domain|
| (floaters possible)      |      | (tight hull, few floaters)|
+--------------------------+      +---------------------------+

One sentence: From a soft learned occupancy that can hallucinate, to a hard geometric filter based on multi-view visibility counts; the core shift is replacing a learned prior with a deterministic one that uses information already present in the input.

Expert Assessment

Problem choice: Real gap. Sparse NVS is a practical bottleneck (AR/VR, robotics) and the ambiguity of silhouette-only constraints is well known but rarely attacked with such a simple fix. The paper targets a sweet spot: cheap, train-free, and effective.

Method maturity: Clever but not groundbreaking. The idea of multi-view visibility as a geometric constraint is not new (it’s related to space carving from 1999), but the way it’s integrated into modern NVS pipelines and the systematic study of K thresholds is novel. The main insight is the *combination and the practical effectiveness. Could a smarter baseline be a learned binary occupancy network? Possibly, but that would be heavier and domain-specific.

Experimental integrity: Baselines are fair (DVGO for NeRF, 3DGS and GaussianObject for GS). Metrics (PSNR, SSIM, LPIPS) show consistent improvement. A red flag: the experiments use only synthetic or controlled real datasets with clean silhouettes; performance on noisy masks (in-the-wild) is not shown. Also, comparison to depth-regularized methods like DS-NeRF would strengthen the narrative.

Writing quality: Well structured, but the paper rushes through the metaphor and intuition in the abstract. Section 4 (implementation) is too concise; the exact K sensitivity analysis is buried in the appendix. A good rewrite would move the “why K=3” experiment into the main text.

Verdict: weak accept — a solid, practical contribution that is likely to be used as a plug-in in many sparse-view pipelines, but lacks theoretical depth and challenging real-world validation.

Takeaways

• Plug-in prior: Any NVS pipeline can adopt VisDom in a few lines of code: just ray-cast silhouettes to a voxel grid and pass the domain mask as a sampling constraint. No training, no hyperparameter tuning beyond K.
• K-threshold as a dial: Practitioners can tune K per dataset. For 4 views, K=3 is near-optimal; for 6 views, K=4 or 5 may further tighten. This gives a controllable trade-off between recall and precision.
• Silhouette quality matters: The method inherits all limitations of the silhouette extraction. If masks are noisy (e.g., from a segmentation network), the visible domain may be overly restrictive or miss parts. A simple erosion/dilation on masks before ray-casting could help.