OmniRoam: Panoramic Video Generation for Infinite Scene Exploration

Paper: 2603.30045 Authors: Yuheng Liu, Xin Lin, Xinke Li, Baihan Yang, Chen Wang, Kalyan Sunkavalli, Yannick Hold-Geoffroy, Hao Tan, Kai Zhang, Xiaohui Xie Categories: cs.CV

The Gap

Existing video generation models (like Sora, Runway, Pika) work with perspective videos — they’re essentially looking through a narrow camera viewfinder. This creates three problems: (1) you only see a slice of the scene at any moment, (2) when you want to “walk” through a scene for a long time, the model loses track of where things are, and (3) different generated frames contradict each other spatially because there’s no global map.

Prior work tried trajectory-controlled generation (CameraCtrl, MotionCtrl) but still used perspective representation. Some explored panoramic images (PanoDreamer) but not video. The gap: no one combined panoramic representation with long-horizon video generation under trajectory control.

Problem: Perspective video = limited FOV + poor long-term consistency
    |
    v
Assumption: Panoramic representation inherently encodes global spatial structure
    |
    v
Method: Two-stage generation (preview + refine) with panoramic video
    |
    v
Evidence: Outperforms baselines on FID, consistency metrics, user studies
    |
    v
Conclusion: Panoramic representation enables longer, more consistent scene wandering

The Increment

One sentence: Before this paper, video generation models could create short perspective clips with shaky consistency; after it, you can generate hour-long panoramic journeys through scenes with stable spatial structure.

Core Mechanism

OmniRoam works in two stages. The preview stage takes an input image or video and a camera trajectory, then generates a low-resolution panoramic video that sketches out the scene. Think of it as a rough draft — fast, covers the whole journey, but blurry. This uses a diffusion model conditioned on trajectory (camera poses) and the input.

The refine stage takes that draft and does two things simultaneously: extends it temporally (makes it longer) and upsamples it spatially (makes it sharper). This isn’t just upscaling — it’s generating new content that’s consistent with the preview. The model uses a sliding window approach: it looks at a chunk of the preview video, refines it, then slides forward with overlap to maintain consistency.

Input Image/Video + Trajectory
         |
         v
    [Preview Stage]
    Diffusion Model
    (trajectory-conditioned)
         |
         v
    Low-res Panoramic Video (draft)
         |
         v
    [Refine Stage]
    Sliding Window Processing
    (temporal extend + spatial upsample)
         |
         v
    High-res Long Panoramic Video

Here’s the structural metaphor: Think of OmniRoam as a cartographer exploring unknown territory. The preview stage is like flying over the land in a helicopter, quickly sketching a rough map of mountains, rivers, and forests — you see everything but details are fuzzy. The refine stage is like walking the same route on foot with a surveyor’s tools, filling in precise elevations, textures, and landmarks. The sliding window is your surveyor’s notebook: you focus on one section, record it carefully, then move forward with some overlap so the next section connects seamlessly to what you just mapped. The panoramic representation is like using a 360° compass instead of looking through binoculars — you always know where everything is relative to everything else.

Key Concepts

• Panoramic representation: Instead of a rectangular image (like a photo), imagine wrapping the image around a sphere so it covers all directions at once. Technically, it’s an equirectangular projection — the sphere is “unwrapped” into a rectangle where horizontal position = azimuth angle and vertical position = elevation angle. Why does this help? Because when you move through a scene, objects that disappear from a perspective camera’s view are still present in the panoramic representation — they just shift horizontally. This makes long-term consistency easier because the model has a persistent “memory” of the entire scene encoded in each frame.

• Trajectory control: You specify a path through 3D space as a sequence of camera poses (position + orientation). The model conditions its generation on this trajectory, meaning it learns to generate content that matches what you’d see if a camera actually moved along that path. The trick is encoding these poses into the diffusion model’s conditioning mechanism — they use a combination of pose embeddings and cross-attention so the model “knows” where it is at each timestep.

• Sliding window refinement: When extending a video, you can’t just generate frame 1001 after seeing frame 1000 — errors accumulate. Instead, you generate frames 1-100, then frames 51-150 (overlapping with the first batch), then frames 101-200 (overlapping with the second), and so on. The overlap regions act as “anchors” that keep the new content consistent with what came before. It’s like writing a story where you always re-read the last paragraph before writing the next one.

Framework Shift

Before (mainstream approach):        After (this paper):

Perspective Video Generation         Panoramic Video Generation
                                    
[Camera] ---> [Limited FOV]         [Camera] ---> [360° Coverage]
    |              |                     |              |
    v              v                     v              v
Frame 1        Frame 2                Frame 1        Frame 2
[  |  ]        [  |  ]                [======]       [======]
(narrow)       (narrow)               (full scene)   (full scene)
    |              |                     |              |
    +------+-------+                     +------+-------+
           |                                    |
           v                                    v
    Inconsistent                          Consistent
    (objects vanish/reappear)             (objects persist)

From narrow-view frame sequences to full-scene panoramic flows, the core shift is from partial observations with implicit spatial relationships to complete observations with explicit global structure.

Expert Assessment

Problem choice: This is a real gap. Video generation has been stuck in perspective-land, and the consistency issues are well-documented (anyone who’s used Runway knows objects morph weirdly after a few seconds). Panoramic representation is an elegant solution that’s been underexplored in the video domain. The problem sits at the intersection of 3D vision and generative models — timely given the push toward embodied AI and virtual worlds.

Method maturity: The two-stage design is clever but not groundbreaking — it’s essentially coarse-to-fine generation, which is standard practice. The real contribution is adapting diffusion models to panoramic video and showing it works. However, I’m skeptical about the sliding window approach scaling to truly long videos (they show up to 10 minutes, but what about hours?). The method feels like 70% engineering and 30% insight, which is fine for a systems paper.

Experimental integrity: Baselines are fair (CameraCtrl, MotionCtrl, DreamScene4D). The metrics are appropriate (FID for quality, LPIPS for consistency, user studies for realism). However, the datasets are small (1000 synthetic + 500 real panoramic videos) — I’d want to see this tested on more diverse data. The ablation studies are thorough. One red flag: they don’t report failure cases or discuss when the method breaks down.

Writing quality: The paper is well-structured but overly dense in the method section. The “preview + refine” framing is clear, but the technical details about pose encoding and attention mechanisms are buried in walls of text. If they rewrote Section 3.2 with more diagrams and less notation, the paper would be much more accessible. The related work section is comprehensive but reads like a literature dump.

Verdict: weak accept — Solid execution on a real problem with clear improvements over baselines, but the novelty is incremental (panoramic + diffusion isn’t a huge conceptual leap) and the evaluation could be more rigorous.

Takeaways

Steal the representation choice: When building generative models for spatial tasks, ask whether your representation encodes the structure you care about. Panoramic representation naturally encodes “everything around you” — this idea transfers to other domains (e.g., audio generation could use ambisonics instead of stereo).

Steal the two-stage strategy: Preview-then-refine is a general pattern for long-horizon generation. Generate a low-fidelity plan first (cheap, fast, covers the whole task), then refine it incrementally. This works for text (outline → draft → polish), code (pseudocode → implementation → optimization), and now video.

Steal the sliding window trick: When you need to generate long sequences with consistency, overlapping windows are your friend. The overlap acts as a “consistency buffer” that prevents drift. This applies to any autoregressive generation task where errors accumulate (music generation, long-form text, animation).

Don’t steal the dataset approach: Building custom datasets for every new representation is expensive and doesn’t scale. The field needs better ways to adapt existing data or use synthetic data more effectively.