SurGe: Fixing the Bumpy Surfaces in Single-Image 3D Reconstruction

Paper: 2605.31577 Authors: Karim Knaebel, Gonzalo Martin Garcia, Christian Schmidt, Ilya Fradlin, Lucas Nunes, Daan de Geus, Bastian Leibe Categories: cs.CV

The Gap

Recent feedforward methods (Depth Anything, Metric3D) predict point maps from single images and nail the global 3D structure—room layouts, object positions, overall depth ordering. But zoom in and the surfaces look like crumpled aluminum foil: bumpy, noisy, with normals pointing in inconsistent directions. The problem is invisible to standard metrics like AbsRel, which only measure per-pixel depth error and miss local surface coherence.

Prior work optimized for global accuracy. No one asked: “Are neighboring predictions forming smooth, plausible surfaces?” The evaluation gap enabled a training gap—models learned to place points at roughly the right depth without caring if those points implied a sensible surface.

Problem: Global metrics hide local surface errors
   |
   v
Assumption: Supervising 3D gradients + local feature mixing
            will enforce surface coherence
   |
   v
Method: Point gradient matching loss (supervise finite differences)
        + Neighborhood Attention Decoder (local upsampling)
   |
   v
Evidence: New "point map normal" metric shows improvement
          + SOTA on 8 zero-shot benchmarks
   |
   v
Conclusion: Explicit surface supervision fixes the bumpiness

The Increment

One sentence: Before SurGe, feedforward 3D reconstruction gave you the right room but wrong walls; after SurGe, the walls are smooth and the corners meet cleanly.

Core Mechanism

SurGe has two components working in tandem. First, a point gradient matching loss that supervises the 3D finite differences between neighboring predictions. Instead of just checking if each point is at the right depth, it checks if the *change from one point to the next matches the ground truth surface gradient. This is depth-normalized—scaled by the predicted depth—so gradients at near and far surfaces are weighted equally.

Second, a Neighborhood Attention Decoder (NAD) that progressively upsamples features from coarse to fine resolution. At each scale, it uses Neighborhood Attention (a windowed self-attention mechanism) to mix features within local spatial windows. This lets each pixel’s prediction incorporate information from its immediate neighbors, enforcing local consistency. The decoder starts from a ViT encoder’s output and upsamples through four stages, each with NAD blocks that refine the point map.

Input Image
    |
    v
ViT Encoder --> Coarse Features (low res)
    |
    v
NAD Stage 1: Upsample + Neighborhood Attention
    |                (local feature mixing)
    v
NAD Stage 2: Upsample + Neighborhood Attention
    |
    v
NAD Stage 3: Upsample + Neighborhood Attention
    |
    v
NAD Stage 4: Upsample + Neighborhood Attention
    |
    v
Point Map (X, Y, Z per pixel)
    |
    +---> Point Gradient Loss: Compare finite differences
          (supervise dX/dx, dY/dy, dZ/dz between neighbors)

Think of it like sculpting clay. Standard methods drop clay blobs at roughly the right depths—the sculpture’s silhouette looks right from a distance, but up close it’s lumpy. The point gradient loss is like running a smoothing tool between each blob and its neighbors, checking that the slope from one to the next matches the reference. The Neighborhood Attention Decoder is like having multiple sculpting passes, where each pass looks at a small neighborhood and adjusts the clay so it flows smoothly into adjacent areas. By the final pass, the surface is coherent—not just a collection of independent depth guesses, but a connected mesh where each point respects its neighbors.

Key Concepts

• Point Map: A dense 3D representation where every pixel in the input image maps to an (X, Y, Z) coordinate in 3D space. Unlike depth maps (which only give distance along the camera ray), point maps give full 3D positions, making it easier to reason about surface geometry. If you have a 480×640 image, you get 480×640 points in 3D—a point cloud with known pixel correspondence.

• Point Gradient Matching: Supervising the *derivative of predictions rather than the predictions themselves. For a surface, the gradient (how Z changes as you move in X or Y) encodes the surface orientation. By matching gradients to ground truth, you enforce that neighboring points form a plausible surface tangent. This is computed as finite differences: (point[i+1] - point[i]) / depth[i], normalized by depth so near and far surfaces are treated equally. Without this, the model can place points at correct depths but with inconsistent slopes—like a staircase where each step is the right height but tilted at random angles.

• Neighborhood Attention: A windowed self-attention mechanism where each token (pixel feature) only attends to tokens within a local spatial window (e.g., 7×7 or 15×15). This is cheaper than full self-attention (which scales quadratically with image size) but still allows local feature mixing. In the context of point map prediction, it lets each pixel’s 3D coordinate be influenced by nearby pixels, enforcing smoothness. It’s the architectural counterpart to the gradient loss—one supervises the output, the other structures the computation to naturally produce smooth outputs.

Framework Shift

Before (mainstream approach):        After (SurGe):

Image --> Encoder --> Decoder        Image --> Encoder --> NAD
            |            |                        |            |
            v            v                        v            v
      Per-pixel      Point Map              Local mixing   Point Map
      prediction                             at each scale      |
            |                                                   |
            v                                                   v
      Loss on absolute                    Loss on gradients
      depth/position                      (finite differences)
      
Optimization target:                 Optimization target:
"Put each point at                   "Make neighboring points
 the right depth"                     form smooth surfaces"

One sentence: From independent per-pixel depth regression to neighborhood-aware surface reconstruction, the core shift is treating 3D prediction as a surface coherence problem rather than a point placement problem.

Expert Assessment

Problem choice: Real gap. The qualitative bumpiness in feedforward 3D reconstructions has been obvious for years—look at any Depth Anything output under raking light and you see noise. But the field kept optimizing AbsRel because that’s what the benchmarks measured. Introducing a normal-based metric is overdue and makes the problem legible to optimization. This sits at the intersection of “everyone knows it’s a problem” and “no one formalized it,” which is a sweet spot.

Method maturity: The gradient matching loss is elegant—supervising derivatives is a classic trick from physics-informed learning, applied cleanly here. Neighborhood Attention is borrowed from NAT (2022) but used appropriately. The depth normalization in the gradient loss is a nice touch that prevents near-surface bias. However, the paper doesn’t explore simpler baselines: would a standard smoothness regularizer (Laplacian penalty) get you 80% of the way there? The ablation shows NAD helps, but doesn’t isolate whether it’s the attention mechanism or just the progressive upsampling doing the work.

Experimental integrity: Eight zero-shot benchmarks is thorough. The normal metric is well-motivated and the results are consistent across datasets. Baselines are recent SOTA methods (Depth Anything V2, Metric3D V2), which is fair. One concern: the normal metric is computed from the predicted point map itself, so it’s not fully independent—a model could game it by producing locally smooth but globally wrong surfaces. The paper doesn’t show failure cases or discuss when the method breaks down (e.g., on textureless regions or thin structures).

Writing quality: The abstract and intro are crisp. The method section is clear but could use a diagram showing the gradient computation explicitly—the text describes it but a visual would help. The related work section is thorough but reads like a literature dump; cutting it by 30% and focusing on the most relevant comparisons would tighten the narrative. The results section is strong, but the qualitative figures (if present) should be larger—surface errors are subtle and need high-resolution visualization.

Verdict: weak accept — Solid incremental contribution with a useful new metric and clean method, but lacks depth in ablations and doesn’t explore the boundary of when the approach fails.

Takeaways

Supervise derivatives, not just values: When your output has spatial structure (images, meshes, point clouds), adding a loss on finite differences or gradients can enforce coherence that per-element losses miss. This transfers to any dense prediction task—optical flow, surface normals, semantic segmentation boundaries.

Windowed attention for local consistency: If full self-attention is too expensive but you need local feature mixing, Neighborhood Attention is a practical middle ground. Useful for high-resolution dense prediction where global context matters less than local smoothness.

Metrics shape research: The field optimized AbsRel for years and got globally accurate but locally noisy outputs. The moment someone introduced a normal-based metric, the problem became solvable. If your domain has a quality gap that metrics don’t capture, formalizing it is half the battle.

Depth normalization matters: When supervising gradients or differences, normalizing by depth (or scale) prevents near-field bias. This is a small implementation detail with large impact—without it, the model focuses on nearby surfaces and ignores distant ones.