LongStream: Enabling Kilometer-Scale 3D Reconstruction with Gauge-Decoupled Streaming

Paper: 2602.13172 Authors: Chong Cheng, Xianda Chen, Tao Xie, Wei Yin, Weiqiang Ren, Qian Zhang, Xiaoyuang Guo, Hao Wang Categories: cs.CV

Abstract

Long-sequence 3D reconstruction has been a persistent challenge in computer vision, with existing autoregressive models struggling to maintain accuracy over extended sequences. LongStream addresses this by introducing a gauge-decoupled streaming architecture that fundamentally rethinks how visual geometry models handle long sequences. The key insight is abandoning the traditional first-frame anchor paradigm in favor of keyframe-relative pose prediction, transforming the problem from long-range extrapolation to consistent local estimation. Combined with orthogonal scale learning and cache-consistent training strategies, LongStream achieves state-of-the-art performance on kilometer-scale sequences at 18 FPS with stable metric-scale reconstruction.

Key Contributions

• Keyframe-relative pose prediction that eliminates first-frame anchoring, converting long-range extrapolation into constant-difficulty local tasks
• Orthogonal scale learning framework that fully disentangles geometric estimation from scale prediction to suppress drift accumulation
• Cache-consistent training methodology with periodic refresh mechanisms to address attention-sink reliance and KV-cache contamination
• Demonstrated stable metric-scale reconstruction over thousands of frames at real-time performance (18 FPS)

Technical Methodology

The architecture of LongStream revolves around three core innovations that work synergistically to enable ultra-long sequence processing.

The first innovation addresses the fundamental limitation of first-frame anchoring. Traditional autoregressive models predict poses relative to the initial frame, which creates an increasingly difficult extrapolation problem as sequences grow longer. This leads to attention decay where the model struggles to maintain focus on distant reference frames, and scale drift where small errors compound over time. LongStream reformulates this by predicting poses relative to recent keyframes instead. Mathematically, instead of estimating $\mathbf{T}_{i \to 0}$ (transformation from frame $i$ to frame 0), the model predicts $\mathbf{T}_{i \to k}$ where $k$ is a recent keyframe. This keeps the prediction task within a bounded difficulty range regardless of sequence length.

The second innovation introduces orthogonal scale learning. In traditional approaches, scale and geometry are entangled in the estimation process, causing scale drift to corrupt geometric predictions. LongStream decouples these by learning scale in a separate orthogonal subspace. The model maintains two parallel estimation streams: one for relative geometry (rotation and translation direction) and another for absolute scale. This disentanglement is crucial because geometric relationships are more stable across frames than absolute scale, which is prone to accumulation errors.

The third innovation tackles Transformer-specific challenges in streaming scenarios. Long sequences create two problems: attention-sink effects where the model over-relies on initial tokens, and KV-cache contamination where outdated information degrades prediction quality. LongStream introduces cache-consistent training where the model is trained with the same cache management strategy used during inference. Additionally, periodic cache refresh discards stale KV pairs at regular intervals, preventing long-term contamination. This reduces the train-inference gap that typically causes performance degradation in production deployments.

Experimental Results

LongStream demonstrates substantial improvements over existing methods across multiple benchmarks. On long-sequence datasets, the model maintains consistent accuracy over thousands of frames where baseline methods experience catastrophic drift. The metric-scale reconstruction remains stable even on kilometer-scale trajectories, with absolute trajectory error (ATE) remaining bounded rather than growing linearly with sequence length.

Performance metrics show LongStream achieves 18 FPS processing speed, making it viable for real-time applications. This is particularly impressive given the model processes full sequences rather than isolated frame pairs. The throughput is sufficient for autonomous navigation, AR/VR applications, and real-time mapping scenarios.

Ablation studies validate each component’s contribution. Removing keyframe-relative prediction causes rapid accuracy degradation beyond 500 frames. Disabling orthogonal scale learning leads to scale drift that corrupts reconstruction quality. Without cache-consistent training and periodic refresh, attention degradation becomes severe after 1000 frames, with prediction quality dropping significantly.

The model also shows strong generalization to diverse environments. Testing across indoor scenes, urban environments, and natural landscapes demonstrates that the architectural innovations are not dataset-specific but address fundamental challenges in long-sequence processing.

Implications and Future Directions

LongStream represents a significant step toward practical long-sequence 3D reconstruction. The gauge-decoupled approach offers a principled solution to problems that have plagued autoregressive visual geometry models. By reformulating the task structure rather than simply scaling up model capacity, the work demonstrates that architectural innovation can overcome fundamental limitations.

The real-time performance opens new application possibilities. Autonomous vehicles could maintain consistent maps over extended drives without periodic reinitialization. AR devices could track environments across long sessions without drift. Robotics applications could perform continuous SLAM over extended operation periods.

Several directions emerge for future work. Extending the approach to handle dynamic scenes would broaden applicability. Incorporating semantic understanding could improve robustness in challenging conditions. Investigating how the gauge-decoupled framework applies to other sequential prediction tasks beyond visual geometry could reveal broader principles for long-sequence modeling.

The cache management strategies also suggest general solutions for deploying Transformers in streaming scenarios. The train-inference gap problem affects many production systems, and cache-consistent training with periodic refresh could benefit other domains requiring long-context processing.

Takeaways

1. First-frame anchoring is a fundamental bottleneck for long-sequence autoregressive models; keyframe-relative prediction transforms unbounded extrapolation into consistent local estimation
2. Disentangling scale from geometry through orthogonal learning prevents drift accumulation that corrupts reconstruction quality over long sequences
3. Cache-consistent training with periodic refresh addresses Transformer-specific degradation in streaming scenarios, reducing the train-inference gap
4. Real-time performance (18 FPS) on kilometer-scale sequences demonstrates that architectural innovation can achieve both accuracy and efficiency
5. The gauge-decoupled framework offers a principled approach to long-sequence modeling that may generalize beyond visual geometry tasks