Latent-WAM: Compact World Models for Autonomous Driving via Spatial-Aware Compression

Paper: 2603.24581 Authors: Linbo Wang, Yupeng Zheng, Qiang Chen, Shiwei Li, Yichen Zhang, Zebin Xing, Qichao Zhang, Xiang Li, Deheng Qian, Pengxuan Yang Categories: cs.CV, cs.RO

The Gap

World-model-based autonomous driving planners have hit a compression wall. Methods like GAIA-1, MILE, and UniAD either keep raw pixel representations (computationally expensive), use naive compression (losing spatial structure), or ignore temporal dynamics (poor future prediction). The result: you need massive datasets and compute to get decent planning, and even then performance plateaus.

The specific bottleneck: existing approaches treat compression, spatial understanding, and temporal modeling as separate problems. They compress first (losing geometry), then try to recover spatial reasoning later. Or they model dynamics in pixel space (wasteful). No one’s asking: what if we compress with spatial awareness baked in, then model dynamics in that compressed space?

Problem: World models waste compute on redundant representations
   |
   v
Assumption: Compression + spatial reasoning + dynamics can be unified
   |
   v
Method: SCWE (compress with geometry) + DLWM (dynamics in latent space)
   |
   v
Evidence: 89.3 EPDMS on NAVSIM v2 with 104M params, 10x less data
   |
   v
Conclusion: Spatially-aware latent compression enables efficient planning

The Increment

One sentence: Before, you needed billions of parameters and massive datasets to get world models working for driving; now you can match or beat that performance with 104M parameters and a fraction of the data by doing compression and dynamics modeling in a spatially-aware latent space.

Core Mechanism

Latent-WAM has two modules working in sequence. First, the Spatial-Aware Compressive World Encoder (SCWE) takes multi-view camera images and squashes them into a small set of “scene tokens” — think of it as distilling six camera views into 64 compact descriptors. It does this using learnable queries (like DETR) but crucially distills geometric knowledge from a frozen foundation model (DINOv2) so the tokens retain spatial structure even after aggressive compression.

Second, the Dynamic Latent World Model (DLWM) takes these scene tokens plus the vehicle’s motion history and predicts future scene tokens autoregressively. It’s a causal Transformer that rolls forward in time, generating what the world will look like at future timesteps. Finally, a lightweight planner head reads these predicted future tokens and outputs a trajectory.

Multi-view images (6 cameras)
   |
   v
[SCWE: Compress with geometry]
   |
   +---> Learnable queries (64 tokens)
   |
   +---> Distill from DINOv2 (spatial structure)
   |
   v
Scene tokens (compact, spatially-aware)
   |
   v
[DLWM: Predict future in latent space]
   |
   +---> Causal Transformer
   |
   +---> Autoregressive rollout (t+1, t+2, ...)
   |
   v
Future scene tokens
   |
   v
[Planner head]
   |
   v
Trajectory waypoints

Think of SCWE as a skilled sketch artist at a crime scene. Instead of photographing every detail (raw pixels), the artist draws a compact schematic that captures the essential geometry — where the cars are, road boundaries, spatial relationships. The sketch is tiny compared to photos but preserves what matters for reasoning. Then DLWM is like an animator who takes that sketch and draws the next frames, showing how the scene evolves. The animator works entirely in “sketch space” (latent tokens), never going back to full photos. Finally, the planner is a director who looks at the animated sketches and decides where the car should go. The magic: by working in sketch space throughout, you avoid the computational cost of manipulating full images while keeping the spatial intelligence needed for planning.

Key Concepts

• Learnable Queries for Compression: Instead of encoding every pixel, you start with a fixed set of learnable vectors (queries) that act like questions: “Where are the vehicles?” “What’s the road geometry?” These queries attend to the image features and pull out only the information needed to answer those questions, compressing the scene into a small set of tokens. It’s like having a checklist of important things to notice rather than memorizing everything you see. The queries are learned during training to extract exactly what’s useful for driving decisions.

• Distillation from Foundation Models: DINOv2 is a vision model pre-trained on millions of images to understand general visual concepts and geometry. Latent-WAM “distills” this knowledge by training SCWE to match DINOv2’s feature representations. Concretely, they add a loss term that penalizes differences between SCWE’s internal features and DINOv2’s features for the same image. This forces SCWE to inherit DINOv2’s spatial understanding even though SCWE is compressing aggressively. It’s like learning to paint by copying a master — you absorb their understanding of perspective and composition.

• Autoregressive Latent Dynamics: The DLWM predicts future scene tokens one timestep at a time, where each prediction depends on all previous tokens (causal). At time t, it sees tokens from t-2, t-1, t and predicts t+1. Then it uses that predicted t+1 to predict t+2, and so on. This is “autoregressive” — each step feeds into the next. Crucially, this happens entirely in token space (64 tokens per timestep), not pixel space (millions of values). It’s like predicting tomorrow’s weather by looking at today’s summary statistics rather than re-simulating every air molecule.

Framework Shift

Before (mainstream approach):        After (this paper):

Raw pixels (H x W x C x T)           Raw pixels
    |                                    |
    v                                    v
Encode each frame                    SCWE (geometry-aware)
    |                                    |
    v                                    v
Pixel/feature sequences              Scene tokens (64 x D)
    |                                    |
    v                                    v
Model dynamics in                    DLWM (latent dynamics)
high-dim space                           |
    |                                    v
    v                                Future tokens (64 x D)
Decode to actions                        |
                                         v
(Expensive, needs                    Plan from tokens
 massive data)                           
                                     (Efficient, less data)

From modeling the world in observation space to modeling it in a compressed, spatially-aware latent space, the core shift is moving the dynamics prediction bottleneck from high-dimensional representations to compact tokens that retain geometric structure.

Expert Assessment

Problem choice: Real gap. World models for driving have been stuck in a compute-data trap — you need both to scale, but most teams have neither. Attacking the representation efficiency is the right lever. This sits at the intersection of two trends: foundation model distillation and latent world models, both hot but not yet combined effectively for driving.

Method maturity: Mostly clever composition of existing ideas (learnable queries from DETR, distillation from DINOv2, causal Transformers). The novelty is in the integration and the specific design choices (what to distill, where to compress). Not groundbreaking conceptually, but solid engineering. One concern: the reliance on DINOv2 means you’re inheriting its biases and limitations — if DINOv2 fails on certain geometric patterns, so will this.

Experimental integrity: Strong results on two benchmarks (NAVSIM v2, HUGSIM) with fair baselines. The 10x data efficiency claim is compelling but needs scrutiny — they compare against methods trained on different data distributions, so it’s not apples-to-apples. The ablations are thorough (Table 2 shows each component’s contribution). One red flag: no failure case analysis. What scenarios does this fail on? When does latent compression lose critical information?

Writing quality: Section 3.2 (SCWE) is dense and could use a diagram showing the distillation process explicitly. The related work section name-drops too many methods without clearly positioning this work’s unique angle until the method section. The results section is solid but buries the data efficiency story — that should be front and center since it’s the main practical win.

Verdict: weak accept — Solid execution on an important problem with strong empirical results, but limited conceptual novelty and missing critical failure analysis. The data efficiency gains are valuable for practitioners even if the method is incremental.

Takeaways

Steal the distillation strategy: if you’re building a task-specific model and have access to a strong foundation model, add a distillation loss to inherit its representations while compressing. This is cheaper than fine-tuning the foundation model directly and lets you control the bottleneck size.

The learnable query pattern for compression is underused outside object detection. If you’re dealing with high-dimensional inputs (images, point clouds, sensor arrays) and need compact representations, learnable queries let you compress with task-awareness baked in from the start.

For temporal modeling, consider whether you really need to work in observation space. If your task has a natural bottleneck (like scene understanding for driving), push the dynamics modeling to happen after that bottleneck in a lower-dimensional space. The compute savings compound over time.