ReCoSplat: Teaching 3D Reconstruction to Self-Correct Through Visual Feedback

Paper: 2603.09968 Authors: Freeman Cheng, Botao Ye, Xueting Li, Junqi You, Fangneng Zhan, Ming-Hsuan Yang Categories: cs.CV

The Gap

Existing online novel view synthesis methods face a training-inference dilemma. Feed-forward approaches like pixelSplat assemble local Gaussians using camera poses, which scales better than predicting in canonical space. But here’s the catch: training with ground-truth poses gives stable gradients, while inference uses predicted poses that inevitably contain errors. This distribution mismatch causes reconstruction quality to degrade when the model encounters pose prediction errors it never saw during training.

Prior work either accepts this mismatch (training with GT poses, hoping for generalization) or trains end-to-end with predicted poses (unstable, harder to converge). Neither approach directly addresses the core issue: the model has no mechanism to compensate for pose errors once they occur.

Problem: Training-inference pose distribution mismatch
   |
   v
Assumption: Visual feedback can compensate for pose errors
   |
   v
Method: Render-and-Compare (ReCo) module
   |
   +---> Render current reconstruction from predicted view
   +---> Compare with incoming observation
   +---> Use difference as conditioning signal
   |
   v
Evidence: SOTA on RealEstate10K, ACID, Tanks&Temples
   |
   v
Conclusion: Visual feedback bridges training-inference gap

The Increment

One sentence: Before ReCoSplat, models trained on perfect poses failed when given imperfect ones; after ReCoSplat, models use visual feedback to self-correct pose errors during reconstruction.

Core Mechanism

ReCoSplat is an autoregressive transformer that processes observations sequentially. At each step, it receives an image and (optionally) camera parameters, predicts the camera pose if needed, then generates local 3D Gaussians for that view. These Gaussians accumulate into a growing scene representation.

The key innovation is the Render-and-Compare (ReCo) module. Before processing a new observation, ReCo renders the current reconstruction from the predicted camera viewpoint. This rendered image is compared with the actual incoming observation, producing a difference map. This difference map becomes an additional conditioning signal fed into the transformer, alongside the observation itself.

Input Stream:  [Img_t] ---> Pose Predictor ---> [Pose_t]
                 |                                  |
                 v                                  v
Current Scene: [Gaussians_1...t-1] ---> Render --> [Rendered_t]
                                            |
                                            v
                                    Compare with [Img_t]
                                            |
                                            v
                                      [Difference Map]
                                            |
                                            v
Transformer: [Img_t] + [Difference Map] --> [New Gaussians_t]
                                                    |
                                                    v
                                            Add to Scene

Think of ReCoSplat like a GPS navigation system with visual confirmation. Traditional approaches are like following GPS directions blindly—if the GPS drifts (pose error), you end up in the wrong place. ReCoSplat adds “looking out the window”: it checks what the scene should look like from the predicted position (renders), compares it with what it actually sees (incoming image), and uses the mismatch to correct its understanding. If the GPS says you’re at an intersection but you see a park, that visual discrepancy tells you the GPS is wrong and helps you recalibrate.

The ReCo module acts as the “looking out the window” component. When pose predictions drift, the rendered view won’t match the actual observation. This mismatch provides a training signal that teaches the model to compensate for pose errors. During training, even with ground-truth poses, the model learns to use visual feedback. At inference, when poses are predicted and imperfect, the visual feedback mechanism kicks in to stabilize reconstruction.

Key Concepts

• Render-and-Compare (ReCo): Imagine you’re assembling a 3D puzzle but your position sensor is slightly off. Instead of trusting the sensor blindly, you render what you think you should see from your current position, then compare it with what you actually see. The difference tells you how wrong your position estimate is. ReCo does exactly this: it renders the current 3D reconstruction from the predicted camera pose, compares the rendering with the actual incoming image, and uses the visual discrepancy as a correction signal. This creates a feedback loop where visual evidence compensates for pose prediction errors. Concrete example: if the pose predictor thinks the camera moved 2 meters left but actually moved 1.5 meters, the rendered view will be slightly off. The difference map highlights this mismatch, allowing the model to adjust its reconstruction accordingly.

• Training-Inference Distribution Mismatch: During training, you have perfect camera poses from ground truth. The model learns to assemble Gaussians assuming poses are accurate. But at inference, poses come from a predictor that makes errors. The model now faces inputs (imperfect poses) it never saw during training—like training a driver on smooth roads then putting them on a bumpy one. ReCoSplat solves this by making visual feedback (ReCo) part of training, so the model learns to handle pose uncertainty from the start.

• KV Cache Compression: Transformers remember previous tokens using key-value (KV) caches. For 100-frame sequences, this memory explodes. ReCoSplat uses a hybrid strategy: (1) early transformer layers see truncated history (only recent frames matter for low-level features), (2) later layers use chunk-level selective retention (keep important chunks, discard redundant ones). This cuts KV cache by 90%+ while maintaining reconstruction quality. Think of it like taking notes: you don’t transcribe everything verbatim (truncation), and you highlight key points while summarizing the rest (selective retention).

Framework Shift

Before (mainstream approach):          After (ReCoSplat):

Training:                              Training:
  GT Pose --> Model --> Gaussians        GT Pose --> Model --> Gaussians
                                                      ^
                                                      |
                                              ReCo Feedback
                                              (visual check)

Inference:                             Inference:
  Pred Pose --> Model --> Gaussians      Pred Pose --> Model --> Gaussians
  (mismatch!)                                         ^
                                                      |
                                              ReCo Feedback
                                              (compensates error)

From blind pose trust to visual feedback loop, the core shift is making the model aware of its own reconstruction quality through rendering.

Expert Assessment

Problem choice: This is a genuine gap. The training-inference mismatch in pose-conditioned models is well-known but rarely addressed directly. Most papers either ignore it or use end-to-end training (which is unstable). ReCoSplat tackles it head-on with a principled solution. The problem sits at the intersection of feed-forward efficiency and robustness—a sweet spot for practical impact.

Method maturity: The ReCo module is clever, not brute force. It’s conceptually simple (render and compare) but addresses the root cause rather than symptoms. The KV cache compression is practical engineering. However, I wonder if simpler approaches exist—could pose refinement networks or uncertainty-aware pose prediction achieve similar results with less machinery? The paper doesn’t explore these alternatives thoroughly.

Experimental integrity: Baselines are fair, covering both canonical-space methods (pixelSplat) and pose-free approaches (DUSt3R). Results are strong across multiple datasets (RealEstate10K, ACID, Tanks&Temples). The ablations clearly show ReCo’s contribution. One minor flag: the paper doesn’t deeply analyze failure cases or discuss when ReCo might struggle (e.g., textureless scenes where visual comparison is ambiguous).

Writing quality: The abstract and method sections are clear. However, the related work section feels rushed—it lists prior work without clearly positioning ReCoSplat’s unique contribution. The ablation study could be more thorough: what happens with different rendering resolutions for ReCo? How sensitive is the method to the quality of the pose predictor? Rewriting Section 4.3 (ablations) with deeper analysis would elevate the paper.

Verdict: weak accept — Addresses a real problem with a principled solution and strong empirical results, though the novelty is incremental and some design choices lack deeper justification.

Takeaways

The render-and-compare feedback loop is the transferable idea here. Anytime you have a model making predictions that affect downstream processing, consider adding a “sanity check” module that renders/simulates the prediction’s consequences and compares with ground truth. This pattern applies beyond 3D reconstruction: in robotics (predict action, simulate outcome, compare with sensor feedback), in generative models (predict next token, render full sequence, check coherence), or in planning systems (predict trajectory, simulate physics, compare with constraints).

The KV cache compression strategy is also practical: early layers get truncated history, late layers get selective retention. This hierarchical memory management could apply to any long-sequence transformer task where different layers need different temporal contexts.

Finally, the training-inference mismatch framing is useful. When your training data has privileged information (like GT poses) that won’t be available at inference, don’t just hope for generalization—explicitly train the model to handle the degraded inputs it will actually see.