Vega: Teaching Self-Driving Cars to Follow Natural Language Instructions

Paper: 2603.25741 Authors: Sicheng Zuo, Yuxuan Li, Wenzhao Zheng, Zheng Zhu, Jie Zhou, Jiwen Lu Categories: cs.CV, cs.AI, cs.RO

The Gap

Existing vision-language-action models in autonomous driving (like GAIA-1, DriveVLM) use language primarily for scene understanding or reasoning — they describe what’s happening or explain decisions. But they can’t actually follow user instructions like “drive more cautiously” or “take the scenic route.” The car understands language but doesn’t obey it. This limits personalization: every passenger gets the same driving style regardless of their preferences.

The core issue: prior work treats language as an input for perception, not as a control signal for behavior. There’s no dataset pairing diverse instructions with corresponding trajectories, and no architecture designed to condition planning on free-form commands.

Problem: Language used for description, not control
   |
   v
Assumption: Instructions can directly shape trajectory generation
   |
   v
Method: Autoregressive (vision+language) + Diffusion (world+action)
   |
   v
Evidence: InstructScene dataset (100K scenes) + superior planning metrics
   |
   v
Conclusion: Instruction-conditioned driving is feasible and effective

The Increment

One sentence: Before Vega, autonomous vehicles could understand language but not follow instructions; after Vega, they can adjust driving behavior based on natural language commands.

Core Mechanism

Vega splits the problem into two paradigms running in parallel. The autoregressive branch processes vision (camera frames) and language (user instructions) token by token, building a joint representation. Think of it as reading a scene and a command simultaneously, creating a shared understanding. The diffusion branch then takes this understanding and generates two things: future scene predictions (world modeling) and the actual driving trajectory (action).

Data flows like this: raw images get tokenized into visual patches, instructions become text tokens, both feed into a transformer with joint attention. This produces a context vector. The diffusion model receives this context and iteratively denoises random noise into coherent future frames and waypoints. The key innovation is using different projection layers for each modality — vision, language, world, and action each get specialized heads, allowing the model to handle heterogeneous outputs without interference.

The architecture resembles a restaurant kitchen with two stations. The prep station (autoregressive) takes orders (instructions) and ingredients (visual input), combining them into a unified recipe. The cooking station (diffusion) follows that recipe to produce both a preview of the dish (future scene) and the actual meal (trajectory). Each station has specialized tools (projection layers) for different tasks, but they share the same base understanding of what needs to be made.

Key Concepts

• Joint Attention Across Modalities: Instead of processing vision and language separately then fusing them, Vega lets visual tokens and text tokens attend to each other directly in the same transformer. Imagine a conversation where you’re simultaneously watching a video and listening to someone describe what to do — your brain doesn’t process these separately then merge them; it integrates them on the fly. This allows instructions like “slow down near the pedestrian” to directly influence which visual features get emphasized (the pedestrian’s position and velocity).

• Autoregressive + Diffusion Hybrid: Autoregressive models excel at discrete, sequential reasoning (like understanding language and scene context), while diffusion models excel at continuous, multi-modal generation (like smooth trajectories and realistic future frames). Vega uses autoregressive for the “thinking” part and diffusion for the “doing” part. It’s like planning a route with GPS (discrete waypoints, sequential decisions) versus actually steering the car (continuous control, smooth motion). Most prior work picks one paradigm; Vega uses both where each is strongest.

• InstructScene Dataset: Existing driving datasets have trajectories but no instructions explaining why the car behaved that way. InstructScene annotates 100K scenes with diverse commands (“drive defensively,” “merge aggressively,” “avoid the left lane”) paired with trajectories that follow those instructions. This is the training data that teaches the model the instruction → behavior mapping. Without it, the model would have no examples of how different commands should change driving style.

Framework Shift

Before (mainstream approach):        After (this paper):

Language -----> [Scene Description]  Language -----> [Behavior Control]
                       |                                    |
Vision -------> [Perception Module]  Vision -------> [Joint Attention] 
                       |                                    |
                       v                                    v
                [Planning Module]            [Autoregressive Context]
                       |                                    |
                       v                                    v
                  Trajectory              [Diffusion: World + Action]
                                                           |
                                                           v
                                              Future Frames + Trajectory

From language-as-annotation to language-as-command, the core shift is treating instructions as first-class control inputs rather than auxiliary explanations.

Expert Assessment

Problem choice: Real gap. Personalized driving is a genuine user need (comfort preferences, risk tolerance, route priorities), and current systems are one-size-fits-all. This sits at the intersection of embodied AI and human-AI interaction — timely given the push toward consumer autonomous vehicles.

Method maturity: The autoregressive + diffusion split is clever but not groundbreaking — it’s a natural division of labor given each paradigm’s strengths. The joint attention mechanism is standard practice in multimodal transformers. The real contribution is the dataset and the end-to-end integration. No simpler approach is obviously overlooked, though one could question whether diffusion is necessary for trajectory generation (simpler regression might suffice).

Experimental integrity: Baselines are reasonable (GAIA-1, DriveVLM) but the paper doesn’t compare against instruction-tuned versions of these models, which would be fairer. The metrics (L2 error, collision rate) are standard but don’t directly measure instruction-following quality — there’s no human evaluation of whether the car actually drives “cautiously” when asked. The InstructScene annotation process isn’t fully detailed, raising questions about label quality.

Writing quality: The method section is dense and assumes familiarity with both autoregressive and diffusion models. The paper would benefit from a clearer explanation of how the two paradigms interact during inference. The related work section undersells the novelty by not clearly distinguishing instruction-following from scene understanding.

Verdict: weak accept — Solid contribution with a useful dataset, but experimental evaluation could be more rigorous and the writing could better highlight the core innovation.

Takeaways

The autoregressive + diffusion split is transferable: use autoregressive for discrete reasoning over heterogeneous inputs (text, structured data, discrete observations) and diffusion for continuous generation tasks (trajectories, images, audio). This division of labor could apply to robotics manipulation (autoregressive for task planning, diffusion for motion generation) or creative tools (autoregressive for style/content understanding, diffusion for image synthesis).

The joint attention mechanism for vision + language is now table stakes for any multimodal system — don’t fuse modalities late; let them interact from the start.

The bigger lesson: instruction-following requires paired data (command + behavior), not just behavior alone. If you want a model to respond to user preferences, you need training examples of those preferences being expressed and followed. This applies beyond driving to any personalized AI system.