HorizonStream: Taming Temporal Chaos in Streaming 3D Reconstruction

Paper: 2605.23889 Authors: Chong Cheng, Peilin Tao, Nanjie Yao, Guanzhi Ding, Xianda Chen, Yuansen Du, Xiaoyang Guo, Wei Yin, Weiqiang Ren, Qian Zhang Categories: cs.CV

The Gap

Online 3D reconstruction from streaming video must estimate camera pose and scene geometry under two brutal constraints: causality (no future frames) and bounded memory (no infinite history). Existing methods handle this poorly. Sliding windows (DROID-SLAM, DepthCRFs) impose hard cutoffs that discard long-range evidence, causing drift on extended sequences. Recurrent networks (DeepV2D) and causal attention (StreamingFlow) suffer from cache saturation—early tokens dominate the hidden state or attention distribution, creating “attention sinks” that block new information. The core problem: geometric evidence is temporally heterogeneous. A feature match might matter for 5 frames (short-lived correspondence) or 5000 frames (persistent global scale), but current architectures force uniform influence patterns that fit neither timescale.

Problem: Streaming 3D reconstruction drifts on long sequences
   |
   v
Root cause: Temporal heterogeneity of geometric evidence
   |
   +---> Short-lived: feature matches (5-50 frames)
   +---> Long-lived: global scale, loop closures (1000+ frames)
   |
   v
Existing architectures impose uniform influence:
   |
   +---> Sliding windows: hard cutoff -> discard long-range
   +---> Ungated recurrence: saturation -> early tokens dominate
   +---> Causal attention: attention sinks -> new info blocked
   |
   v
Solution: Factorize evidence influence kernel
   |
   +---> Temporal factor: learnable multi-timescale decay
   +---> Spatial factor: local attention + spatiotemporal RoPE
   |
   v
Evidence: 48-frame training generalizes to 10,000+ frames
   |
   v
Conclusion: Explicit factorization resolves temporal mismatch

The Increment

One sentence: Before this paper, streaming 3D reconstruction architectures imposed uniform temporal influence patterns that caused drift or saturation; after, factorized evidence kernels enable stable multi-timescale propagation on arbitrarily long sequences.

Core Mechanism

HorizonStream treats geometric propagation as an evidence influence kernel and factorizes it into temporal and spatial components. The temporal factor is handled by Geometric Linear Attention (GLA), which learns per-channel exponential decay rates. Instead of uniform attention weights or fixed recurrence gates, each feature channel gets its own decay timescale—some channels retain information for thousands of frames (global scale), others decay quickly (transient matches). This prevents cache saturation because the model explicitly controls how long each piece of evidence persists.

The spatial factor uses Geometric Local Attention with Spatiotemporal RoPE. Rather than attending to all past tokens (which creates attention sinks), each query only attends to spatially nearby tokens within a local window. Spatiotemporal RoPE encodes both 3D position and time into rotary embeddings, so the model learns which spatial neighborhoods matter at which temporal distances. This suppresses attention sinks by limiting the receptive field while preserving geometric coherence.

Finally, Metric Readout Tokens act as persistent geometric state. These special tokens accumulate scale and pose information across the entire sequence. At each timestep, the model reads out camera pose and depth scale directly from these tokens, which have been updated by the long-horizon attention. This decouples metric estimation from per-frame features, preventing drift.

Input: Frame t, Hidden State H(t-1)
   |
   v
[Geometric Linear Attention]
   |  (per-channel decay rates)
   |  channel 1: decay=0.99 (slow, global scale)
   |  channel 2: decay=0.90 (medium, structure)
   |  channel 3: decay=0.50 (fast, transient matches)
   |
   v
[Geometric Local Attention + Spatiotemporal RoPE]
   |  (spatial window + 3D position encoding)
   |  query at (x,y,z,t) attends to neighbors in space-time
   |
   v
Updated Hidden State H(t)
   |
   +---> [Metric Readout Tokens]
   |        |
   |        v
   |     Camera Pose, Depth Scale
   |
   +---> H(t) -> next frame

Think of HorizonStream as a river delta system. Water (geometric evidence) flows from upstream (past frames) to downstream (current frame). In a naive system, all water flows at the same speed—either it rushes through and disappears (sliding window), or it stagnates in pools (attention sinks). HorizonStream builds channels with different flow rates. The main channel (high decay rate) carries fast-moving water for immediate matches. Side channels (low decay rate) carry slow-moving water for global scale—this water persists for miles downstream. Spatiotemporal RoPE acts as terrain: water only flows between nearby elevation points (spatial locality), and the terrain changes over time (temporal encoding). The readout tokens are measurement stations at the river mouth, integrating flow from all channels to estimate the total volume (pose and scale). This way, evidence flows at its natural timescale without clogging or vanishing.

Key Concepts

• Evidence Influence Kernel: In streaming reconstruction, each past observation (a feature match, a depth estimate) influences future estimates with some strength that decays over time. The influence kernel describes this decay pattern. A uniform kernel (like standard attention) treats all evidence equally regardless of age. A factorized kernel separates temporal decay (how long evidence persists) from spatial structure (which observations interact). Example: A loop closure detected 1000 frames ago should still influence global scale (slow temporal decay), but only affect geometrically nearby regions (spatial locality). HorizonStream learns this kernel explicitly instead of imposing a fixed pattern.

• Attention Sink: In causal attention, early tokens accumulate disproportionate attention mass because they appear in every subsequent token’s context. The model learns to “dump” attention onto these early tokens as a way to ignore irrelevant information, but this blocks new information from being incorporated. It’s like a traffic jam where cars (attention) pile up at the first exit (early tokens) instead of flowing to their actual destinations (relevant recent tokens). GLA avoids this by using exponential decay instead of softmax normalization—attention naturally fades for old tokens without creating artificial sinks.

• Spatiotemporal RoPE: Rotary Position Embedding (RoPE) encodes position by rotating feature vectors in high-dimensional space. Standard RoPE encodes 1D sequence position. Spatiotemporal RoPE encodes both 3D spatial position (x, y, z in the scene) and temporal position (frame index). When computing attention, the dot product between rotated queries and keys naturally emphasizes tokens that are close in both space and time. This lets the model learn “this feature at (x, y, z, t) should attend to features within 1 meter and 10 frames” without hard-coded windows. The rotation angles are derived from the 4D spatiotemporal coordinates, so geometric proximity directly modulates attention strength.

Framework Shift

Before (sliding window + uniform attention):

Frame:  t-3    t-2    t-1     t
        |      |      |       |
        v      v      v       v
      [CNN] [CNN] [CNN]   [CNN]
        |      |      |       |
        +------+------+-------+
               |
               v
          [Attention]  <-- all frames equal weight
               |
               v
          Pose, Depth

Problem: Hard cutoff at window boundary
         Uniform influence -> no timescale separation


After (HorizonStream):

Frame:  ...  t-1000  ...  t-10  ...  t
             |            |          |
             v            v          v
          [Encoder]   [Encoder]  [Encoder]
             |            |          |
             +------------+----------+
                          |
                          v
                  [Geometric Linear Attention]
                   (per-channel decay rates)
                    fast: ~~~>
                    slow: ============>
                          |
                          v
                  [Local Attention + RoPE]
                   (spatial neighbors only)
                          |
                          v
                  [Readout Tokens]
                          |
                          v
                    Pose, Depth

Benefit: No cutoff, multi-timescale propagation
         Spatial locality prevents attention sinks

From uniform temporal influence to factorized multi-timescale propagation, the core shift is explicit control over evidence persistence.

Expert Assessment

Problem choice: Real and well-motivated. Drift on long sequences is a documented failure mode in SLAM and streaming reconstruction. The temporal heterogeneity framing is insightful—it explains why both sliding windows and recurrence fail, rather than treating them as unrelated issues. The problem sits at the intersection of geometric vision and sequence modeling, which is timely given the Transformer’s dominance in both domains.

Method maturity: The factorization into temporal and spatial components is elegant, but the individual pieces (linear attention, local attention, RoPE) are borrowed from language modeling. The novelty is in the combination and the geometric interpretation. GLA’s per-channel decay is a clever way to avoid attention sinks, though it’s essentially a gated linear RNN with learned gates. The spatiotemporal RoPE extension is straightforward. The method feels like careful engineering rather than a fundamental insight, but that’s appropriate for a systems problem like streaming reconstruction.

Experimental integrity: Strong. Training on 48-frame clips and testing on 10,000+ frames is a genuine generalization test. The ablations isolate each component’s contribution. Baselines include recent methods (DROID-SLAM, DepthCRFs, StreamingFlow), though I’d want to see comparisons with more classical SLAM systems (ORB-SLAM3) to verify the gains aren’t just from learned features. The constant memory claim is validated with profiling. One concern: the paper doesn’t report failure cases or sequences where the method still drifts. All methods fail somewhere; hiding that weakens credibility.

Writing quality: The abstract and introduction are crisp. The method section is dense—Figure 2 (architecture diagram) does heavy lifting, but the text doesn’t walk through a concrete example of how evidence flows through the system. The “evidence influence kernel” framing is introduced but not formalized mathematically until the appendix, which makes the main text feel hand-wavy. The related work section is thorough but could better position the work relative to recent Transformer-based SLAM methods. The conclusion oversells slightly (“resolves the fundamental mismatch”)—the method works well empirically, but the theoretical justification for why this factorization is optimal is thin.

Verdict: weak accept — Solid engineering contribution with strong empirical results, but the novelty is incremental and the theoretical framing could be tighter.

Takeaways

Factorize temporal influence explicitly: If your sequential model suffers from drift or saturation, don’t just add more layers or bigger windows. Identify the timescales in your data (transient vs. persistent evidence) and build separate pathways for each. Per-channel decay rates in linear attention are a simple way to achieve this without architectural complexity.

Spatiotemporal embeddings for geometric sequences: If your data has both spatial and temporal structure (video, sensor streams, trajectories), encode both dimensions into position embeddings. Spatiotemporal RoPE is a drop-in replacement for standard positional encoding that lets the model learn which spatiotemporal neighborhoods matter.

Readout tokens for persistent state: Instead of decoding directly from per-frame features, maintain special tokens that accumulate information across the sequence. This decouples metric estimation from local features and reduces sensitivity to per-frame noise. Applicable beyond reconstruction—any task where you need stable global state (tracking, forecasting) could benefit.

Test generalization to longer sequences: If you’re building a streaming model, train on short clips but test on sequences 10-100x longer. This exposes failure modes (drift, saturation, memory leaks) that don’t appear in fixed-length evaluation. It’s a cheap way to validate that your architecture actually handles unbounded sequences.