Seoul World Model: Grounding Video Generation in Real Cities

Paper: 2603.15583 Authors: Junyoung Seo, Hyunwook Choi, Minkyung Kwon, Jinhyeok Choi, Siyoon Jin, Gayoung Lee, Junho Kim, JoungBin Lee, Geonmo Gu, Dongyoon Han Categories: cs.CV

The Gap

Existing world models like GAIA-1, DriveDreamer, and GenAD synthesize driving videos by imagining everything from scratch. They produce visually plausible scenes but these environments are entirely artificial—the buildings, streets, and landmarks don’t correspond to any real location. This limits their utility for applications requiring spatial fidelity: autonomous vehicle testing in specific cities, urban planning simulations, or location-aware scenario generation. The fundamental limitation is that prior models operate in a vacuum, disconnected from geographic reality.

Problem: Synthetic world models lack spatial grounding
   |
   v
Assumption: Real street-view images can anchor generation
   |
   v
Method: Retrieval-augmented conditioning + temporal alignment
   |
   v
Evidence: Outperforms baselines on spatial fidelity across 3 cities
   |
   v
Conclusion: Real-world grounding enables city-scale simulation

The Increment

One sentence: Before this paper, world models imagined plausible but fictional environments; after, they can render videos of actual cities by anchoring generation to retrieved street-view images.

Core Mechanism

SWM operates through three interconnected stages. First, given a target trajectory through Seoul, it retrieves nearby street-view images from a database indexed by GPS coordinates. These retrieved images serve as spatial anchors—visual evidence of what actually exists at those locations. Second, it processes these anchors through cross-temporal pairing, deliberately misaligning the temporal correspondence between reference images and target frames to force the model to learn dynamic scene understanding rather than simple frame copying. Third, it feeds these processed anchors into an autoregressive video generator that produces each frame conditioned on both previous frames and the retrieved spatial context.

The data pipeline addresses sparsity through view interpolation. Street-view captures happen at sparse intervals (every few meters), but training requires dense video sequences. SWM synthesizes intermediate views by interpolating between sparse captures, creating coherent training videos. For diverse trajectories beyond vehicle-mounted captures, they generate a large-scale synthetic dataset with varied camera movements—turns, elevation changes, different speeds—all registered to Seoul’s actual geography.

For long-horizon generation (hundreds of meters), SWM introduces Virtual Lookahead Sink: at each generation chunk, it retrieves an image from a future location along the trajectory and uses it as an additional conditioning signal. This acts like a compass, continuously re-grounding the generation to prevent spatial drift over long sequences.

Trajectory Input
   |
   v
[GPS Coordinates] ---> Retrieval Database
   |                        |
   v                        v
[Retrieved Images] <--- [Street-view Index]
   |
   v
Cross-temporal Pairing (temporal misalignment)
   |
   v
[Spatial Anchors] + [Previous Frames] + [Lookahead Image]
   |                                           |
   +-------------------------------------------+
   |
   v
Autoregressive Video Generator
   |
   v
Generated Frame ---> (loop for next frame)

Think of SWM like a tour guide navigating a city with a photo album. Traditional world models are like guides who describe the city entirely from imagination—they might describe plausible buildings and streets, but nothing corresponds to reality. SWM is like a guide who carries a photo album indexed by location. As you walk, the guide pulls out photos taken near your current position and uses them as reference points: “See this building in the photo? That’s what’s actually here.” The cross-temporal pairing is like deliberately using photos taken at different times of day or seasons, forcing the guide to understand the underlying structure rather than just memorizing specific lighting conditions. The Virtual Lookahead Sink is like the guide occasionally checking photos from a few blocks ahead to ensure you’re heading in the right direction and won’t drift off course. The view interpolation is like filling gaps between sparse photos by intelligently blending nearby shots to create a smooth visual journey.

Key Concepts

• Cross-temporal Pairing: When you retrieve a street-view image from location A to condition generation at location A, the naive approach would match them temporally—use a summer photo for summer generation, morning photo for morning. But this creates a shortcut: the model learns to copy lighting and weather rather than understanding spatial structure. Cross-temporal pairing deliberately breaks this correspondence—it might use a winter reference image to generate a summer scene at the same location. This forces the model to extract geometric and structural information (building shapes, road layout) while ignoring temporal attributes (lighting, weather, traffic). It’s like learning to recognize your house by its architecture rather than by whether the lights are on. The model must learn: “This is what exists here, regardless of when.”

• Virtual Lookahead Sink: Autoregressive generation accumulates errors—each frame depends on previous frames, so mistakes compound over long sequences. In spatial terms, this manifests as drift: after generating 100 frames, you might end up in the wrong neighborhood. The Virtual Lookahead Sink addresses this by retrieving an image from a future location (say, 50 meters ahead) and using it as an additional conditioning signal. It’s like hiking with both a map of where you are and a photo of your destination—the destination photo keeps you oriented even if you take small wrong turns. Technically, this future image acts as a “sink” that pulls the generation trajectory toward the correct spatial location, preventing accumulated drift from derailing long-horizon sequences.

• Retrieval-Augmented Conditioning: Instead of generating everything from learned parameters, the model retrieves actual images from a database and conditions generation on them. This is fundamentally different from pure generative models. Think of it like the difference between drawing a portrait from memory versus drawing while looking at a reference photo. The reference photo (retrieved image) provides ground truth about spatial structure, while the generative model handles dynamics, viewpoint changes, and temporal evolution. The retrieval database acts as an external memory that stores real-world visual facts, while the neural network learns to interpret and animate those facts.

Framework Shift

Before (mainstream approach):        After (this paper):

[Learned Parameters]                 [Learned Parameters] + [Retrieval DB]
        |                                     |                    |
        v                                     v                    v
  Imagine Scene                        Query Location -----> Get Real Images
        |                                     |                    |
        v                                     +--------------------+
  Generate Video                                     |
   (fictional)                                       v
                                              Condition on Reality
                                                     |
                                                     v
                                              Generate Video
                                               (grounded)

From imagination-based synthesis to retrieval-augmented grounding, the core shift is anchoring generation in external visual evidence rather than purely internal representations.

Expert Assessment

Problem choice: This is a genuine gap with practical implications. Autonomous vehicle companies need to test in specific cities, not generic environments. Urban planners want to visualize changes to actual locations. The problem sits at the intersection of world models (hot area) and practical deployment (real need). Not manufactured.

Method maturity: The core insight—using retrieval for spatial grounding—is elegant and somewhat obvious in hindsight. The engineering around it (cross-temporal pairing, lookahead sink, view interpolation) shows thoughtful problem-solving rather than brute force. However, the reliance on pre-existing street-view databases limits generalization. What about cities without comprehensive street-view coverage? The synthetic data generation for diverse trajectories feels like a workaround for data limitations rather than a principled solution.

Experimental integrity: Baselines are reasonable (recent world models), but the evaluation heavily favors their approach by design—of course a retrieval-based method wins on spatial fidelity when compared to imagination-based methods. The more interesting question is: how much does retrieval help versus just training on more Seoul-specific data? The ablations address some of this, but I’d want to see comparisons against models trained exclusively on Seoul without retrieval. The three-city evaluation (Seoul, Busan, Ann Arbor) is good for generalization claims, but details on data availability per city are sparse.

Writing quality: The paper front-loads motivation well but buries important technical details. The cross-temporal pairing mechanism deserves earlier, clearer explanation—it’s central to why this works. The synthetic data generation section feels rushed; given its importance for trajectory diversity, it warrants more depth. The related work section could better position this against retrieval-augmented generation in other domains (not just world models).

Verdict: weak accept — Solid contribution with clear practical value, but the method feels more like careful engineering of existing components than a fundamental breakthrough; the evaluation could be more rigorous about isolating the retrieval contribution.

Takeaways

The cross-temporal pairing trick transfers broadly: whenever you have paired data with both spatial and temporal dimensions, deliberately misaligning the temporal correspondence forces models to learn spatial structure. This applies to any domain where you want to disentangle “what exists” from “what it looks like right now”—medical imaging across different scanners, satellite imagery across seasons, or audio across recording conditions.

The Virtual Lookahead Sink pattern—conditioning on future states to prevent drift—is applicable beyond video generation. Any autoregressive process over spatial or structured domains can benefit: trajectory prediction (condition on destination), code generation (condition on function signature), or long-form text generation (condition on outline points).

The broader lesson: when your domain has external ground truth available (maps, databases, knowledge graphs), retrieval-augmented approaches can outperform pure generation by anchoring outputs in reality. The key is designing the conditioning mechanism so the model learns to interpret and animate the retrieved information rather than just copying it.