OmniStream: One Backbone to See, Reconstruct, and Act in Real Time

Paper: 2603.12265 Authors: Yibin Yan, Jilan Xu, Shangzhe Di, Haoning Wu, Weidi Xie Categories: cs.CV

The Gap

Vision foundation models have gotten very good — but only in narrow lanes. CLIP and its descendants nail semantic alignment but are blind to geometry. DPT and depth estimators understand 3D structure but can’t reason about language. Video transformers like VideoMAE model temporal dynamics but process clips offline in batch mode. And none of them are designed to run frame-by-frame in a live stream while maintaining memory of what came before.

The result is that anyone building a real-world visual agent — a robot, a drone, an AR system — has to stitch together a pipeline of specialists. That stitching is expensive, brittle, and architecturally incoherent. Each specialist was trained with different assumptions about what “a good representation” means.

The specific boundary this paper pushes against: no single frozen backbone can simultaneously serve semantic understanding, streaming 3D reconstruction, and downstream robotic control. The authors argue this fragmentation isn’t inevitable — it’s a consequence of how we’ve been training, not a fundamental limit of what a single model can learn.

Problem: Vision models are specialists, not generalists
  |
  v
Assumption: A shared causal spatiotemporal representation
            can serve semantic + geometric + action tasks
  |
  v
Method: OmniStream
  - Causal spatiotemporal attention (no future leakage)
  - 3D-RoPE for position encoding across space + time
  - Multi-task pre-training on 29 datasets
  - Persistent KV-cache for online streaming
  |
  v
Evidence:
  - Competitive with specialists on image/video probing
  - Streaming depth/geometry reconstruction
  - Zero-shot transfer to robotic manipulation
  |
  v
Conclusion: A single frozen backbone can generalize
            across semantic, spatial, and temporal tasks

The Increment

One sentence: Before this paper, you needed a different backbone for each modality of visual understanding; after it, there’s evidence that one causally-structured, multi-task-trained backbone can cover all three without fine-tuning.

Core Mechanism

OmniStream is a transformer backbone with two architectural choices that do most of the heavy lifting. First, all attention is causal — each token can only attend to past and present tokens, never future ones. This isn’t just a training trick; it’s what makes the model deployable in a real-time stream where future frames literally don’t exist yet. Second, positional encoding uses 3D-RoPE, which extends rotary position embeddings to encode height, width, and time simultaneously. This gives the model a coherent sense of where and when each patch is, without needing separate positional heads for spatial vs. temporal reasoning.

The pre-training is a four-way multi-task setup. Task one: static image representation learning (think masked autoencoding or contrastive objectives). Task two: temporal video representation learning across clips. Task three: streaming geometric reconstruction — predicting depth and 3D structure from the causal stream. Task four: vision-language alignment, connecting visual tokens to text. These four tasks are trained jointly, not sequentially, which forces the shared backbone to find representations that are simultaneously useful for all of them.

At inference, the model maintains a persistent KV-cache across frames. As each new frame arrives, its keys and values are appended to the cache, and attention is computed over the full causal history. This is the same mechanism that makes autoregressive language models efficient — here it’s applied to video, enabling constant-time per-frame processing regardless of how long the stream has been running.

Input stream: ...frame_t-2, frame_t-1, frame_t, frame_t+1...
                                              ^
                                         only up to here

Each frame -> patch tokens
                  |
                  v
         3D-RoPE encoding
         (x, y, t coordinates)
                  |
                  v
    Causal Spatiotemporal Attention
    [attends to KV-cache of all past frames]
                  |
         +--------+--------+--------+--------+
         |        |        |        |        |
         v        v        v        v        v
      Semantic  Temporal  Depth   3D Geo   Lang
      probing   probing   recon   recon    align

Think of OmniStream like a court stenographer who’s also a structural engineer and a translator — all at once. The stenographer writes down everything in real time (causal, no peeking ahead). The 3D-RoPE is like the page-and-line numbering system: every word gets tagged with exactly when and where it appeared, so you can always reconstruct the spatial and temporal layout of the room. The KV-cache is the stenographer’s running transcript — they don’t re-read the whole session from scratch each time someone speaks; they just append the new line and reference back as needed. The multi-task pre-training is what makes this stenographer unusual: they were trained simultaneously in courtrooms, construction sites, and translation booths, so the same note-taking habits serve all three contexts. When a robotics task shows up — a context they’ve never seen — the underlying note-taking discipline transfers anyway.

Key Concepts

• Causal attention: In a standard transformer, every token can attend to every other token — past and future. That’s fine for offline processing but impossible in a live stream. Causal attention adds a mask that blocks any token from looking at tokens that come after it in time. Concretely: when processing frame 47, the model can use information from frames 1–47 but not frame 48. This constraint is what makes the model deployable in real time, and it’s also what forces the model to learn representations that are genuinely predictive rather than retrospective.

• 3D-RoPE (Rotary Position Embedding in 3D): RoPE is a way of encoding position by rotating the query and key vectors in attention by an angle proportional to their position. The rotation means that the dot product between two tokens naturally encodes their relative distance — nearby tokens have similar rotations, distant ones diverge. The 3D extension applies this idea across three axes: horizontal pixel position, vertical pixel position, and time. So every patch token carries a compact, geometry-aware address in space-time, and the attention mechanism can reason about “how far apart are these two patches in 3D?” without any explicit coordinate regression head.

• Persistent KV-cache for video: In autoregressive language models, the KV-cache stores the key and value projections of all previous tokens so you don’t recompute them on every new token. OmniStream applies this to video: as each frame is processed, its KV pairs are appended to a running cache. The next frame attends over the full cache. This means per-frame compute is O(1) in the number of past frames (just one attention pass over the cache), not O(n²). The “persistent” part means the cache isn’t cleared between frames — it accumulates across the entire stream, giving the model a long-term memory of what it has seen.

• Multi-task pre-training: Rather than training separate models for semantic understanding, depth estimation, and temporal modeling, OmniStream trains one backbone on all tasks simultaneously. The loss is a weighted sum of task-specific objectives. This forces the model to learn a shared representation space where semantic, geometric, and temporal information coexist. The result is a backbone that can be frozen and probed for any of these tasks without task-specific fine-tuning.

Why It Matters

The practical impact is that you can now deploy a single frozen backbone for real-time visual agents. No more stitching together CLIP for semantics, DPT for depth, and VideoMAE for temporal reasoning. One model, one forward pass per frame, constant memory, and it generalizes to robotic manipulation tasks it was never explicitly trained on.

The conceptual impact is that causal spatiotemporal attention might be the right inductive bias for visual foundation models. If you want a model that can run in the real world — where the future doesn’t exist yet and you need to make decisions frame by frame — then causal attention isn’t just a nice-to-have, it’s the only coherent choice. And if you train that causally-structured model on a diverse enough mix of tasks, it learns representations that transfer across modalities without further tuning.

This paper doesn’t claim to have solved vision. But it does show that the fragmentation of vision models into specialists isn’t inevitable. A single backbone, trained the right way, can be a generalist.