World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible

Paper: 2606.13652 Authors: Hao Zhang, Mohamed El Banani, Jen-Hao Cheng, Paul Zhang, Yi Hua, Ben Mildenhall, Christoph Lassner, Narendra Ahuja, Gengshan Yang Categories: cs.CV, cs.GR

The Gap

Existing image-to-3D methods fall into two camps, each with a fundamental flaw.

Depth predictors (e.g., MiDaS, ZoeDepth) are pixel-aligned — they give you a depth value for every input pixel — but stop at the visible surface. They cannot see behind objects; occluded geometry is simply missing.

Generative image-to-3D models (e.g., Zero-1-to-3, LRM, TripoSR) can hallucinate complete shapes, but they often trade away pixel-level alignment. The generated 3D geometry may not project back onto the input image correctly; you lose the correspondence between “this pixel in the photo” and “that point in 3D space.”

The paper asks: Can we have both? Can we produce a complete 3D reconstruction that is pixel-aligned — where every input pixel maps to at least one 3D point, and occluded geometry is still recovered?

Problem: Image-to-3D trades faithfulness for completeness
              |
              v
Assumption: A single depth per pixel is insufficient;
            we need a stack of depths per ray.
              |
              v
Method: For each pixel, predict an ordered list of
        3D points (first = visible, then occluded)
              |
              v
Evidence: Outperforms depth predictors on surface
          accuracy and generative models on alignment
              |
              v
Conclusion: Full geometry + pixel alignment is achievable
            with a diffusion transformer and mixed noise schedule

The Increment

One sentence: Before this paper, you chose between pixel-aligned but incomplete depth maps and complete but misaligned 3D shapes; now you can have both — a complete, pixel-aligned geometry stack per input pixel.

Core Mechanism

World Tracing introduces a multi-layer per-pixel geometry representation. For each pixel $(u,v)$, the model predicts $K$ points in camera coordinates: $(x_1,y_1,z_1), (x_2,y_2,z_2), \dots, (x_K,y_K,z_K)$, where the first point is the visible surface and subsequent points are front-to-back intersections with occluded surfaces. This is like having a “depth volume” — a 3D array of shape $(H, W, K, 3)$.

To predict this stack, the authors design WT-DiT, a diffusion transformer. The geometry stack is restructured into tokens: each pixel location has $K$ tokens, one per layer. The key is a factorized attention scheme: intra-layer tokens (same layer across all pixels) attend to each other via global attention; inter-layer tokens (different layers at the same pixel) attend via cross-attention. This separates “what’s in the scene globally” from “how layers stack locally.”

The training uses pixel-space flow matching with a mixed noise schedule: the first (visible) layer is given very little noise to force accurate reconstruction, while occluded layers receive more noise to encourage generative completion.

Input Image:  H x W x 3
    |
    v
[ViT encoder -> spatial features]
    |
    v
WT-DiT (Diffusion Transformer)
    |
    |--- Factorized Attention ---
    |     Intra-layer: tokens across space <-- global geometry
    |     Inter-layer: tokens at same pixel <-- local stacking
    |
    v
Output: H x W x K x 3  (points in camera space)
    |
    +-- layer 1: visible surface (depth)
    +-- layer 2: first occlusion
    +-- ...
    +-- layer K: K-th occlusion

Structural metaphor: a stack of tracing paper.

Imagine you are an architect drawing a building from a fixed viewpoint. You place a sheet of tracing paper over the photo; you trace everything you see — the front facade. That’s the visible surface. Now you flip to a second sheet, still aligned, and you trace what you guess is behind the facade: the internal walls, the back wall. Third sheet: even further behind. Eventually you have a stack of aligned tracings, each layer showing one depth slice. The first layer must exactly match the photo; the later layers are your expert guesses. WT-DiT is the assistant who generates those stacked tracings in one shot, using a global understanding of buildings (intra-layer attention) while ensuring each pixel’s stack is consistent (inter-layer attention). The mixed noise schedule is like telling the assistant: “Trace the front facade with obsessive precision; for the back rooms, you have more creative freedom.”

Key Concepts

• Pixel-aligned geometry stack: Instead of a single depth value per pixel, produce an ordered list of 3D points along the ray. The ordering is front-to-back. This representation is dense (each pixel always projects to a point) and covers occluded regions. Think of it as a “per-pixel multi-view” — you get multiple depths at the same image coordinate.

• Factorized attention for layered tokens: A naive design would treat all $H \times W \times K$ tokens as a flat sequence. WT-DiT separates attention over space (global) and over layers (local). Intra-layer attention (same layer $l$ across all pixels) learns scene structure — e.g., “pillar at (x1,y1) and pillar at (x2,y2) are part of the same object.” Inter-layer attention (same pixel across layers) ensures that occluded points lie behind the visible surface — e.g., “layer2 point’s z must be greater than layer1’s z at this pixel.”

• Mixed noise schedule for diffusion: Standard diffusion applies uniform noise across all dimensions. Here, visible layer tokens get a very low noise level (e.g., $\sigma = 0.1$), forcing the model to denoise to the exact ground truth depth. Occluded layers get higher noise (e.g., $\sigma = 1.0$), allowing generation when ground truth is ambiguous. This is implemented by conditioning each layer’s noise level on a binary mask that indicates which layers are “visible” (first layer always visible, others vary per scene). The model learns to interpolate between reconstruction and generation.

Framework Shift

Before (mainstream approach):                       After (this paper):
+-----------------------+                    +----------------------------+
|   Input Image         |                    |   Input Image              |
+-----------------------+                    +----------------------------+
         |                                            |
         v                                            v
+-----------------------+                    +----------------------------+
| Depth estimator       |                    | World Tracing (WT-DiT)     |
| (per-pixel depth)     |                    | (per-pixel stack of K pts) |
+-----------------------+                    +----------------------------+
         |                                            |
         v                                            v
+-----------------------+                    +----------------------------+
| Visible surface       |                    | Visible surface (layer 1) |
| only. No occluded geo |                    | + occluded geometry       |
|                       |                    | (layers 2..K)             |
+-----------------------+                    +----------------------------+
     |
     v (alternative path)
+-----------------------+
| Generative 3D model   |
| (complete but mis-    |
| aligned with image)   |
+-----------------------+

One sentence: From a single, faithful depth per pixel to a faithful *stack of depths per pixel — the core shift is adding a full generative dimension along the ray while keeping pixel alignment as a hard constraint.

Expert Assessment

Problem choice: Real gap. Depth-only and generative-only have been two tribes for years. The paper crisply identifies the trade-off and proposes a synthesis that is both principled (extending depth maps to stacks) and practical (demonstrated on multiple benchmarks). This sits squarely on the trajectory toward “grounded 3D understanding.”

Method maturity: Clever insight — the factorized attention is lightweight and respects the geometry of the task. The mixed noise schedule is a neat engineering trick that prevents the visible layer from being “washed out” by generation. They didn’t reinvent diffusion; they adapted it. A simpler approach could have been to train a single depth estimator + a separate completion network, but that would lose pixel alignment for the completion. The unified stack representation is the right level of abstraction.

Experimental integrity: Baselines are fair: they compare against depth estimators (MiDaS, Depth Anything, etc.) and generative models (Zero-1-to-3, DreamFusion, etc.), plus ablations of their own design. Numbers show clear win on surface F-score, Chamfer distance, and visual alignment (measured by projection error). One red flag: they only test on datasets with ground-truth 3D (ShapeNet, ScanNet, etc.) — real-world in-the-wild performance might degrade because occluded geometry annotations are scarce. Also, K is fixed (they use 4 layers) — scenes with more complex depth complexity might require dynamic K.

Writing quality: Well-structured, but the “Related Work” section is too brief — they should have contrasted with multi-plane representations (like layered depth images) more explicitly. Also, the method diagram in the paper is helpful but the notation is dense; the authors could have included a smaller example with a concrete pixel.

Verdict: Strong accept — a clean, novel representation with solid engineering, good results, and clear applications (text-driven editing, video synthesis). Worth reading for anyone doing image-to-3D or view synthesis.

Takeaways

1. Per-pixel geometry stacks: If you’re doing depth-guided 3D, consider predicting multiple layers per ray. This simple idea can add occlusion reasoning without losing alignment.
2. Factorized attention as a plug-in: The intra-layer / inter-layer split can be repurposed for any layered 3D representation (e.g., multi-plane images, 3D scene graphs). You don’t need a custom transformer; just reformat your tokens.
3. Mixed noise schedule: When training a diffusion model with both reconstruction and generation components, condition the noise level per token (or per spatial region) to prioritize fidelity where data is deterministic. This is a general trick for hybrid tasks.