Fixing Holes in Real-Time 3D Streams: A Transformer Approach to Multi-View Inpainting

Paper: 2603.05507 Authors: Leif Van Holland, Domenic Zingsheim, Mana Takhsha, Hannah Dröge, Patrick Stotko, Markus Plack, Reinhard Klein Categories: cs.CV, cs.GR

The Gap

AR/VR streaming needs multiple camera views to reconstruct 3D scenes, but real-time constraints force you to use fewer cameras than ideal. Fewer cameras mean blind spots — parts of the scene no camera sees directly. When you render a novel viewpoint, these blind spots show up as holes or incomplete surfaces.

Current systems handle this with simple heuristics: copy nearby pixels, blur edges, or use basic interpolation. These methods are fast but dumb — they don’t understand scene structure, so you get blurry patches, temporal flickering, and artifacts that break immersion. Existing inpainting networks (like LaMa or MAT) are designed for single images and don’t leverage the multi-view geometry or temporal coherence available in streaming setups.

The gap: no inpainting method exploits both multi-camera spatial relationships and temporal consistency while running fast enough for real-time streaming.

Sparse cameras -> Incomplete 3D -> Novel view with holes
                                          |
                                          v
    Assumption: Multi-view + temporal info can guide inpainting
                                          |
                                          v
    Method: Transformer with spatio-temporal embeddings
            + adaptive patch selection
                                          |
                                          v
    Evidence: Outperforms LaMa/MAT on video metrics
              while maintaining real-time speed
                                          |
                                          v
    Conclusion: Context-aware inpainting beats heuristics
                when designed for the streaming use case

The Increment

One sentence: Before this paper, 3D streaming systems filled holes with blind pixel-copying; after, they can use a transformer that understands multi-view geometry and temporal flow to synthesize plausible textures in real time.

Core Mechanism

The method sits as a post-processing module after any 3D rendering pipeline. It takes the rendered frame (with holes marked) plus metadata about camera positions and previous frames. The core is a transformer encoder that processes image patches, but with two key additions: spatio-temporal embeddings and adaptive patch selection.

Spatio-temporal embeddings encode where each patch sits in 3D space (using camera calibration) and when it appears in the video sequence. This lets the transformer attend to patches from other camera views that see the same 3D point, and to patches from previous frames showing the same region. The transformer learns to aggregate information across views and time, filling holes with textures consistent with what nearby cameras see and what the scene looked like a moment ago.

Adaptive patch selection is the speed trick. Instead of processing every patch in every frame, the system identifies which patches contain holes or sit near hole boundaries, then only runs the transformer on those. Clean regions pass through untouched. This cuts computation by 60-80% depending on hole density, making real-time performance feasible.

Input frame with holes
        |
        v
    [Patch Extractor]
        |
        +---> Clean patches -----> Pass through
        |
        +---> Hole patches ------> [Spatio-Temporal Embedder]
                                            |
                                            v
                                   [Transformer Encoder]
                                   (attends across views + time)
                                            |
                                            v
                                   [Decoder: synthesize texture]
                                            |
                                            v
                                   Filled patches
                                            |
                                            v
                                   [Compositor]
                                            |
                                            v
                              Complete output frame

Think of it like a photo restoration studio with multiple archivists. You bring in a damaged photo (the frame with holes). Instead of one person guessing what’s missing, you have archivists who can pull related photos from different angles (other camera views) and earlier versions of the same scene (previous frames). They lay these references on a table, compare details, and collaboratively sketch in the missing parts. The head archivist (transformer) decides which references are most relevant and how to blend them. To save time, they only work on damaged sections — pristine areas don’t need attention. The spatio-temporal embeddings are like catalog tags telling archivists which photos show the same physical location and time period, so they know which references to consult.

Key Concepts

• Spatio-temporal embeddings: Imagine you’re watching a concert from multiple phone cameras held by friends. Each video shows a different angle, and each frame has a timestamp. If one camera’s view is blocked, you can fill in the missing part by looking at what other cameras saw from their angles at that moment, or what the same camera saw a second earlier. Spatio-temporal embeddings encode this “where in 3D space” and “when in time” information as numbers the network can use. For space, they use camera calibration to project each image patch back to its 3D location. For time, they encode frame numbers. This lets the transformer know “patch A from camera 1 and patch B from camera 2 both show the same wall corner” or “this patch looks similar to what we saw 3 frames ago,” so it can intelligently borrow texture from the right sources.

• Adaptive patch selection: Not all parts of an image need expensive neural network processing. If a region is already complete and clean, running a transformer on it wastes computation. Adaptive patch selection is like triaging patients in an ER — you quickly scan the image, identify which patches contain holes or sit near hole edges (the “injured” regions), and only send those through the heavy machinery. Everything else gets a fast pass. This is crucial for real-time performance: if 70% of the image is fine, you save 70% of the compute. The system uses a simple mask to detect holes, dilates it slightly to catch boundary regions, then extracts only those patches for processing.

• Resolution independence: Most neural networks are trained on fixed-size images (say, 512x512) and break if you feed them a different size. This paper’s architecture processes images as collections of patches, and the transformer operates on patch-level features. Since transformers don’t care about sequence length (they just attend over whatever tokens you give them), you can feed in patches from a 1080p image or a 4K image without retraining. The network sees “here are N patches with holes” and processes them the same way regardless of N. This means one trained model works across different camera resolutions and aspect ratios — useful when your AR/VR setup might have a mix of camera hardware.

Framework Shift

Before (mainstream approach):        After (this paper):

  [3D Render]                          [3D Render]
       |                                    |
       v                                    v
  Holes appear                         Holes appear
       |                                    |
       v                                    v
  Simple heuristic:                    Transformer inpainting:
  - Copy neighbor pixels               - Query other views
  - Blur edges                         - Check previous frames
  - Interpolate                        - Synthesize texture
       |                                    |
       v                                    v
  Fast but artifacts                   Slower but coherent
  (no scene understanding)             (multi-view + temporal aware)

From “fill holes with local pixel math” to “fill holes by understanding the scene across space and time,” the core shift is treating inpainting as a multi-view, multi-frame reasoning problem rather than a single-image pixel problem.

Expert Assessment

Problem choice: This is a real gap. Anyone building AR/VR streaming hits this — you can’t afford 50 cameras for full coverage, so you live with holes. Current heuristics genuinely produce bad results (I’ve seen the flickering artifacts in VR demos). The problem sits at the intersection of real-time graphics and learned inpainting, which is timely given the push toward neural rendering in production systems.

Method maturity: The core idea — use transformers with spatio-temporal embeddings — is solid but not groundbreaking. Transformers for video are well-explored (see VideoMAE, TimeSformer). The novelty is in the application-specific design: adaptive patch selection for speed, resolution independence for deployment flexibility, and the multi-view embedding scheme. It’s more engineering insight than algorithmic breakthrough, but that’s fine — the field needs practical solutions. I’d like to see ablations on whether simpler architectures (say, a 3D CNN) could achieve similar results with less complexity.

Experimental integrity: The baselines (LaMa, MAT, CoModGAN) are fair — these are strong single-image inpainting methods. The authors test under the same real-time constraints, which is honest. However, the paper doesn’t compare against multi-view-specific methods (like neural radiance fields with inpainting, or depth-guided inpainting), which feels like a missed opportunity. The metrics (PSNR, SSIM, LPIPS, plus video-specific FVD and temporal consistency) are appropriate. One red flag: no user study. For AR/VR, perceptual quality matters more than PSNR, and users are the ground truth.

Writing quality: The abstract and intro are clear. The method section gets dense — the spatio-temporal embedding formulation could use a worked example with actual numbers. The experiments section front-loads tables without enough qualitative analysis; I’d swap some tables for more visual comparisons showing failure cases. The related work section is thorough but reads like a list. Rewriting the method section with a running example (e.g., “consider a 3-camera setup capturing a room…”) would make it much more accessible.

Verdict: weak accept — Solid engineering work addressing a real problem with a practical solution, but limited algorithmic novelty and missing key comparisons.

Takeaways

Adaptive computation based on content: The patch selection strategy is broadly applicable. If your neural network processes images but only some regions need heavy computation, build a cheap pre-filter to identify those regions and skip the rest. This is especially valuable for real-time systems where you’re compute-bound. The idea transfers to any domain with sparse “interesting” regions: medical imaging (only process suspicious areas), satellite imagery (only analyze changed regions), video compression (only encode complex patches).

Spatio-temporal embeddings for multi-sensor fusion: If you have multiple sensors observing the same scene (cameras, LiDAR, radar), encoding their spatial relationships and temporal alignment as learnable embeddings lets a transformer reason about correspondences without hand-crafted geometry. This pattern works beyond vision — think multi-microphone audio, distributed IoT sensors, or multi-robot coordination.

Resolution independence through patch-based processing: Designing networks to operate on patches rather than full images gives deployment flexibility. You can train once and deploy across different hardware without retraining. The cost is losing some global context, but for tasks where local structure dominates (inpainting, super-resolution, denoising), it’s a good trade.