Geometric Action Model for Robot Policy Learning

Paper: 2606.17046
Authors: Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong
Categories: cs.RO, cs.CV, cs.LG

The Gap

Existing vision-language-action models (VLAs) and video world-action models (WAMs) inherit rich semantic or temporal priors from large-scale foundation models, but they operate on 2D image frames or 2D-derived latent spaces. For contact-rich manipulation — pushing, screwing, inserting — the robot needs explicit 3D geometry: which point on the object touches which surface, where the camera’s line-of-sight occludes or distorts. VLAs ignore this by design; WAMs predict future frames in 2D pixel space, losing geometric structure.

This paper asks: can we take a geometric foundation model (GFM) — pretrained on vast 3D data to reason about point clouds, meshes, and spatial relations — and turn it into a language-conditioned policy *without adding a separate decoder? The gap is not just “use 3D,” but how to minimally adapt a general 3D backbone for closed-loop action.

[ ASCII logic topology ]

Problem: 2D latent spaces lose geometry for contact-rich tasks
   |
   +-- Assumption: A pretrained GFM already has the needed 3D priors
   |
   +-- Method: Split GFM, insert a causal future predictor,
   |          feed predicted tokens back through the same backbone
   |
   +-- Evidence: GAM beats larger baselines on 9 simulation + 3 real-robot tasks
   |
   +-- Conclusion: Minimal architectural change is enough to turn a GFM into a world-action model

The Increment

One sentence: Before this paper, using a geometric model for manipulation meant either training from scratch or bolting on a separate action head; after GAM, you can slice an off-the-shelf GFM in half, drop in a predictor, and get both future geometry and actions from the same pretrained weights.

Core Mechanism

GAM takes a pretrained GFM (e.g., a transformer trained on point clouds and multi-view images) and splits it at an intermediate layer. The shallow layers become an observation encoder: they ingest camera images, proprioception, and language tokens, producing a set of latent features that encode the current scene geometry.

At the split, a causal future predictor — a small transformer module — sees these features plus language and action history, and outputs *future latent tokens (say, tokens corresponding to the scene 0.2 seconds ahead). Those predicted tokens are then fed into the remaining deep layers of the GFM, which were originally trained to decode geometry (e.g., reconstruct point clouds). But now, because the future tokens carry temporal context, those layers naturally produce both the predicted future geometry and, through a tiny linear head after the last block, the robot action.

The entire pipeline is end-to-end differentiable, and only the predictor (≈5% of total parameters) is trained from scratch; the GFM weights are kept frozen or lightly fine-tuned.

[ ASCII diagram of method internals ]

                    +-- Language, proprio, action history --+
                    v                                       |
[ Camera images ] -> [ GFM shallow layers ] -> [ features ] -> [ Future Predictor (causal) ]  
                                                                      |
                                                                      v  (predicted future tokens)
                                                                      |
                                             [ GFM deep layers ] -> [ linear action head ] -> action
                                                                      |
                                                                   [ geometry decoder ] -> future point cloud

Load-bearing structural metaphor: think of this as a factory assembly line where a single robot arm (the GFM) already knows how to build chairs from parts.

• The shallow layers are the conveyor belt sensors: they identify raw materials (observations) and label them with tags (language, proprioception).
• The mid-line is a planning station (future predictor) that, given the current tagged parts and the order (language command), sketches the next state of the assembly (future tokens).
• The deep layers are the arm itself: it takes the plan and executes the necessary movements (action), while simultaneously updating the part inventory (geometric scene prediction).

No extra specialized machinery is needed — the arm already knows how to grab, twist, and insert; you only need a better forecast of what the assembly looks like next.

Key Concepts

• Geometric Foundation Model (GFM): A model pretrained on massive 3D data (e.g., point clouds, depth maps, multi-view images) to understand spatial layout, object shapes, and their mutual occlusion. Unlike a language model that “knows” words, a GFM “knows” that a cube sits on a table or that a handle protrudes. Example architectures include DPT, PointNet++ variants, or transformer-based point cloud encoders.
• Causal future predictor: A module that only looks at past and present information (no peeking at future frames). It is trained to autoregressively predict the next latent token sequence, conditioned on language. This enforces temporal consistency — the model cannot cheat by hallucinating a noncausal future.
• Shared substrate: Here it means that the same backbone (the GFM) does triple duty: perception (encoding observations), temporal prediction (via the injected module), and action decoding (through its final layers). This contrasts with typical designs where each function has its own network, leading to parameter bloat and limited cross-task transfer.

Framework Shift

Before (mainstream VLA approach):          After (GAM):

[Image] -> [2D VLM] -> [action head]       [Image+Lang] -> [GFM shallow -> predictor -> GFM deep] -> [action + geometry]
          (no explicit 3D)                                  (explicit 3D conditioning everywhere)

One sentence: From 2D latent space with separate action head to single 3D backbone with built-in temporal prediction, the core shift is repurposing a geometric model as the central nervous system rather than just a vision pre-processor.

Expert Assessment

Problem choice: Real gap. Contact-rich manipulation needs 3D geometry; 2D VLAs have been papering over this with big data. The paper correctly identifies that pretrained GFMs are underutilized in policy learning.

Method maturity: Cleverly minimal. Splitting a GFM is a simple idea, but the causal predictor insertion is non‑trivial (needs careful architecture matching). However, the paper doesn’t explore whether simpler insertions (e.g., an MLP instead of a transformer predictor) would work — possibly a missed ablation.

Experimental integrity: Baselines include state-of-the-art VLAs (OpenVLA, RT-2-X) and a video world model (UniPi). GAM wins on 9/9 sim tasks and 3/3 real tasks. The speed and memory comparisons are convincing. One red flag: the real-robot tasks are relatively simple (pushing, pick‑and‑place, screwing) — not yet long-horizon or deformable objects.

Writing quality: The abstract and introduction are crisp. Section 3 (method) is well‑structured. The related work is unusually thin — they only mention a few competitors and don’t discuss prior attempts to inject 3D into policies (e.g., PerAct, Act3D). Cutting that corner makes the novelty claim seem stronger than it is.

Verdict: Weak accept — The core idea is sound and the results are strong, but the paper oversells the “first to use 3D for manipulation” narrative (others have done it, just not with pre‑trained GFMs). Still, GAM is a neat engineering contribution that practitioners should pay attention to.

Takeaways

1. Take a pretrained GFM, not a VLM, if your task is contact-rich. The geometric priors transfer almost for free.
2. Splitting a transformer at a middle layer and inserting a causal predictor is a reusable pattern — you can do it to any generative model that produces spatial outputs (e.g., NeRF transformers, diffusion‑based scene generators).
3. The future geometry prediction acts as an implicit world model — you can use the predicted point cloud for downstream collision checking or grasp planning without extra training.