VLGA: Adding Dense 3D Geometry to Vision-Language-Action Models for Autonomous Driving

Paper: 2606.12396 Authors: Jin Yao, Dhruva Dixith Kurra, Tom Lampo, Zezhou Cheng, Danhua Guo, Burhan Yaman Categories: cs.CV, cs.RO

The Gap

Vision-Language-Action (VLA) models have become popular for end-to-end driving because they can interpret scenes and reason in natural language. However, they still struggle to ground actions in the dense 3D world. Existing approaches fall into two camps:

1. Feature injection: Inject features from a frozen 3D foundation model (e.g., BEV-based), but without any loss that forces the policy to *use those features. The policy can ignore them.
2. Sparse geometry losses: Supervise with 3D box or HD map losses. These only constrain a handful of objects or lane boundaries, not every pixel. The policy has no signal for the continuous shape of the road, curbs, or obstacles.

The core limitation: no dense, per-pixel 3D supervision exists for the geometric understanding that driving decisions require. The paper argues that a policy needs to know the 3D location of every visible point – not just boxes – to make safe, smooth plans.

[Problem: VLA models lack dense 3D grounding]
        |
        v
[Assumption: Dense 3D signal helps action prediction]
        |
        v
[Method: Add a Geometry Expert, train with per-pixel pointmap loss vs LiDAR]
        |
        v
[Evidence: Open-loop nuScenes (L2=0.50m, crash=0.18%) and closed-loop Bench2Drive (score=79.08)]
        |
        v
[Conclusion: Geometry as 4th modality is effective and efficient]

The Increment

One sentence: Before VLGA, VLA models either ignored 3D geometry or used only sparse box/map signals; after VLGA, models are explicitly trained to reconstruct every pixel’s 3D position, and that dense geometry significantly improves driving performance.

Core Mechanism

VLGA adds a fourth modality – geometry – alongside vision, language, and action. The architecture has four experts (encoders): Vision (image backbone), Language (text encoder), Geometry (a small 3D neural network), and Action (a planning head). The key addition is the Geometry Expert, which takes the vision features and outputs a *per-pixel pointmap – i.e., for every pixel in the input image, it predicts a 3D coordinate. This pointmap is supervised with an L1 regression loss against LiDAR point clouds projected into the camera view (after time-alignment).

Data flow: camera images go to Vision Expert -> produces multi-scale feature maps. The Geometry Expert takes these features and predicts a dense pointmap. Meanwhile, the Language Expert encodes the driving command (e.g., “turn left”). All three feature streams are fused (via cross-attention and MLP) into a joint representation, which the Action Expert decodes into ego-vehicle waypoints (x,y) for 2 seconds ahead. The geometry loss is applied before fusion, ensuring the policy sees a faithful 3D reconstruction – not just a latent feature that may be ignored.

         [Image]           [Text command]       [LiDAR (train only)]
            |                    |                      |
   [Vision Expert]       [Lang Expert]          [Pointmap GT]
            |                    |                      |
     [Feat maps]          [Text tokens]                |
            \                   /                       |
             [Feature Fusion] <---

![Hero diagram](/arxiv-visuals/vlga-vision-language-geometry-action-models/HeroScene.png)

----------------------+
            |                    |                 [Geometry Expert]
            |                    |                       |
            |                    |                [Pred pointmap]
            |                    |                       |
            |                    |                   [L1 Loss]
            |                    |
         [Action Expert]
            |
         [Waypoints]

Structural metaphor: Imagine a surveyor who must navigate a complex terrain. The surveyor has eyes (vision), a radio to receive instructions (language), and a notebook to sketch the landscape. Traditional VLA models are like a surveyor who jots down only the locations of major landmarks (sparse boxes) and then guesses the ground shape. VLGA provides the surveyor with a detailed topographic map that shows the elevation at every point. This map is created by a dedicated mapper (Geometry Expert) who uses laser range-finding data (LiDAR) as ground truth during training. Every pixel in the camera view gets a height and distance, so the surveyor’s route planning is based on continuous 3D geometry, not just a few pins on a map. The surveyor can now avoid unseen bumps and dips because they have dense 3D knowledge – not just “there’s a car at (10, -2, 0)” but “the entire road surface undulates like this.”

Key Concepts

• Pointmap: A per-pixel prediction of 3D coordinates. For each pixel (u,v) in the image, the model outputs (x,y,z) in the vehicle’s reference frame. Think of it as a depth map that also gives lateral and vertical positions – a full 3D reconstruction. The loss compares this to LiDAR points projected into the same camera pixels (with occlusion handling). This gives a dense signal: every pixel that sees a solid surface gets a supervision target.

• Geometry Expert: A small neural network (e.g., lightweight Conv + Transformer) that takes vision features and outputs the pointmap. It’s separate from the action head – the geometry is an intermediate representation, not the final output. This forces the model to reconstruct the 3D world explicitly, rather than encoding geometry implicitly in a latent vector.

• Dense Spatial Signal: Unlike box or lane losses that constrain only a few hundred points, the per-pixel pointmap loss supervises ~100k points per image (depending on resolution). This provides a rich gradient that shapes the vision features to be 3D-aware. The authors show that without this dense loss, the policy’s feature space collapses and driving performance degrades.

Framework Shift

Before (mainstream VLA):
   Vision -> (shallow 3D) -> Language -> Action
                |  sparse loss (boxes) ignored
                v
            Policy can see only "there is a car at (x,y,z)"
            No pixel-level 3D understanding

After (VLGA):
   Vision -> Geometry Expert -> dense pointmap (supervised)
           \-> Language Expert
                \-> Fusion -> Action
            Every pixel mapped to a real 3D point
            Policy sees the full road surface, curbs, obstacles as continuous geometry

One sentence: From *sparse 3D labels (boxes/maps) with no guarantee the policy uses them to dense per-pixel 3D reconstruction explicitly trained and fused into action, the core shift is making geometry a first-class, supervised modality.

Expert Assessment

Problem choice: This is a genuine gap. Existing VLA models do suffer from poor spatial grounding – they can describe “a car is turning left” but fail to stay in lane during tight curves. The paper targets the root cause: the absence of dense 3D signal. It sits at the right place in the field – just as VLA models are maturing and need better world models.

Method maturity: More of a clever insight than brute force. The idea of adding a pointmap head is simple, but the implementation details (expert architecture, fusion, loss design) are well-engineered. Simpler alternatives (e.g., just adding depth estimation as a multitask loss) could be tried but might not provide the same explicit 3D coordinate signal. The authors show that pointmap > depth alone.

Experimental integrity: Baselines are fair – they compare against state-of-the-art VLA methods (e.g., DriveLM, Lingo). The hybrid open-loop (nuScenes) and closed-loop (Bench2Drive) evaluation is appropriate. No obvious red flags; the results are consistent. However, they only test on two datasets; generalization to other camera configs (e.g., surround-view) is unclear.

Writing quality: Clear enough, but the architecture section could use more diagrams. The ablation study is tucked away; it should be highlighted. The paper would be stronger if they showed *why the pointmap has to be dense (e.g., vs. depth map or occupancy grid) with more intuitive visualizations.

Verdict: Weak accept – a solid incremental contribution that fixes a real weakness in VLA models, but not a breakthrough. The dense geometry idea is likely to be adopted by future VLA papers.

Takeaways

• Per-pixel 3D supervision is a direct and effective way to ground visuomotor policies. For any domain where depth/range is available (e.g., robotics with LiDAR or stereo), consider a pointmap head instead of sparse object losses.
• Geometry as a separate expert (not just an auxiliary loss) allows the model to allocate dedicated capacity to 3D reconstruction without interfering with the language or action pathways.
• The structural metaphor of “topographic map vs. landmark notes” can be used to explain dense vs. sparse 3D understanding to non-experts – a useful communication tool for papers.