CamVLA: Letting Robot Policies Figure Out Where the Camera Is

Paper: 2607.05396 Authors: Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Shijian Lu, Gongjie Zhang, Ran Xu Categories: cs.CV, cs.AI, cs.LG, cs.RO

The Gap

Existing Vision-Language-Action (VLA) models have made impressive strides in learning manipulation policies from demonstrations, but they share a hidden fragility: they assume the camera stays where it was during training. The moment you reposition the camera — which happens constantly in real deployments — the policy breaks.

Prior work on view-robust manipulation (like ViewFormer, VAT-Mart, and various data-augmentation approaches) tried to patch this by explicitly feeding camera extrinsics into the policy. In other words: “tell the robot where the camera is, and it’ll figure out how to move.” But this creates a chicken-and-egg problem — you need precise calibration before the robot can do anything, which is exactly the kind of setup burden real-world deployment hates.

CamVLA’s insight flips the assumption: what if the policy doesn’t need to be told where the camera is, but instead figures it out from the visual input itself, and then uses that self-discovered geometry to generate correct actions?

Problem:  Camera repositions -> policy breaks
              |
              v
Prior fix: Feed camera extrinsics explicitly (fragile, requires calibration)
              |
              v
Gap: No calibration-free, depth-free, single-view VLA for arbitrary viewpoints
              |
              v
CamVLA: Predict both (a) camera-centric action AND (b) hand-eye matrix
              |
              v
Compose via deterministic geometry -> robot base-frame action
              |
              v
Evidence: Consistent success rate gains across unseen viewpoints (sim + real)

The Increment

One sentence: Before this paper, robustness to camera viewpoint changes required explicit calibration; after this paper, a VLA policy can infer camera geometry from a single RGB image and still produce correct robot-base actions.

Core Mechanism

CamVLA’s architecture has two prediction heads that work together through geometric composition. The first head predicts a camera-centric end-effector action — essentially, “how should the gripper move relative to what I’m seeing?” This is expressed in the local camera frame, which makes it inherently viewpoint-invariant: if the camera sees the object to the right of the gripper, the action is “move right in camera coordinates” regardless of where the camera physically sits.

The second head predicts a 6-DoF hand-eye matrix — the rigid transformation that relates the camera’s coordinate frame to the robot’s base frame. This is the geometric grounding step: “where am I looking from, relative to the robot?” By predicting this matrix rather than requiring it as input, CamVLA eliminates the need for calibration.

At inference, a deterministic geometric transformation composes these two predictions: the camera-centric action gets transformed through the hand-eye matrix into a robot base-frame action. This is pure linear algebra — no learning involved in the composition step — which means it’s exact and doesn’t introduce additional error.

Input: Single RGB image + task instruction
            |
            +-------+-------+
            |               |
            v               v
    [Action Head]    [Hand-Eye Head]
            |               |
            v               v
  Camera-centric       6-DoF hand-eye
  end-effector         matrix (T_cam->base)
  action (delta)              |
            |               |
            +-------+-------+
                    |
                    v
          [Geometric Composition]
           (deterministic math)
                    |
                    v
        Robot base-frame action
            |
            v
          Execute on robot

Here’s a structural metaphor that makes this click: imagine you’re giving driving directions to someone over the phone while watching them through a security camera. The old approach requires you to know exactly where the camera is mounted on the building (extrinsics) before you can translate “turn left on screen” into “turn left on the street.” CamVLA is like having a smart assistant who looks at the scene — the street signs, the sun’s position, the building layout — and infers the camera’s location and orientation, then automatically translates your screen-relative directions into street-relative directions. You just say “go left on screen” and the system figures out the rest.

The two heads map perfectly: the Action Head is you giving directions in camera-relative language (“go left on my screen”), and the Hand-Eye Head is the assistant figuring out the camera’s position in the real world. The geometric composition is the assistant doing the coordinate math. The key insight: both tasks are learnable from the same visual input, so you never need to explicitly calibrate.

Key Concepts

• Camera-Centric Action Representation: Instead of predicting “move the gripper 5cm forward in the robot’s frame,” the model predicts “move the gripper 5cm to the right in the camera’s frame.” Why does this matter? Because the same physical motion looks different from different cameras, but the camera-centric representation stays consistent — “right on screen” always means the same direction relative to what you’re seeing. Think of it like giving directions relative to landmarks you can see (“turn left at the gas station”) rather than absolute compass directions (“head north-northwest”). The former works even if you don’t know where you are.

• Hand-Eye Matrix (6-DoF): This is a 4x4 transformation matrix that encodes the full position and orientation relationship between two coordinate frames — in this case, the camera and the robot base. “Hand-eye” comes from the classical robotics problem of relating a camera (“eye”) to a robot end-effector (“hand”). In CamVLA, the model predicts this matrix from visual input alone, effectively answering: “Given what I see and what I know about robot arms, where must this camera be?”

• Deterministic Geometric Composition: The final action is computed by matrix multiplication — not another neural network. The camera-centric action gets multiplied by the inverse of the hand-eye matrix to produce the base-frame action. This is crucial because it means the composition step is exact: if both heads predict correctly, the output is guaranteed to be geometrically correct. No compounding learned error.

Framework Shift

Before (mainstream):                 After (CamVLA):

  RGB image                          RGB image
      |                                  |
      v                                  +----------+
  [Policy Net]                             |          |
      |                                  v          v
      v                            [Action Head] [Hand-Eye Head]
  Base-frame action                        |          |
  (camera must be fixed                    v          v
   or extrinsics given)            cam-centric    T_cam->base
                                       action         |
                                           |          |
                                           +----+-----+
                                                |
                                                v
                                          [Geometry]
                                          (exact math)
                                                |
                                                v
                                        Base-frame action
                                       (camera can be anywhere)

From “tell the robot where the camera is” to “let the robot figure it out,” the core shift is moving camera geometry from an input requirement to a learned intermediate prediction.

Expert Assessment

Problem choice: This is a genuine and practically important gap. Camera repositioning is not an edge case — it’s the default in real deployments. Prior work either ignored this, used heavy data augmentation, or required explicit extrinsics. The paper sits at a natural intersection of VLA scaling and robotics deployment, and the problem will only become more pressing as these models get deployed outside labs.

Method maturity: The core insight — decompose into camera-centric action + hand-eye matrix, compose geometrically — is clean and elegant. It’s closer to “clever reframing” than “brute force scaling.” One concern: the hand-eye prediction is a high-dimensional regression problem (6-DoF), and the paper doesn’t deeply analyze failure modes when this prediction is noisy. A simpler baseline the authors might have considered: explicit visual odometry or marker-based calibration as a preprocessing step. The paper would be stronger if it showed why the learned approach beats these engineering shortcuts.

Experimental integrity: The sim-to-real evaluation is reassuring — testing in both simulation and real robot data is the right call. The baselines include reasonable comparisons (standard VLA, VLA with augmentation, VLA with extrinsics). One flag: the paper evaluates on “unseen viewpoints” but the degree of viewpoint variation matters enormously. Moving a camera 10cm is very different from rotating it 90 degrees. The paper could be more granular here. The numbers are solid for moderate viewpoint changes; extreme cases are less clear.

Writing quality: The abstract and introduction are well-crafted — they nail the problem framing. The method section could be tighter; the geometric composition derivation, while correct, buries the intuition under matrix notation. The experiments section would benefit from a dedicated failure analysis — when does CamVLA break, and what does the hand-eye head predict when it fails? This would elevate the paper from “it works” to “we understand when and why it works.”

Verdict: weak accept — Clean problem framing and elegant decomposition, but the failure mode analysis is thin and extreme-viewpoint robustness needs deeper probing.

Takeaways

Three things worth stealing:

1. The decomposition pattern: When a policy needs to handle variable input conditions, decompose into (a) a condition-invariant task representation and (b) a condition estimation module, then compose them with deterministic operations. This transfers to any domain where you have nuisance variables — lighting changes, sensor noise, domain shifts. The key: the composition should be exact math, not another learned function.

2. Camera-centric vs. base-centric actions: If your robot policy struggles with viewpoint generalization, try expressing actions in the sensor’s frame rather than the robot’s frame. This single reframing can buy you significant robustness without architectural changes.

3. “Learn the calibration” as a strategy: Instead of building calibration into the pipeline as a prerequisite, learn to predict the calibration parameters as an intermediate representation. This converts a deployment requirement into a training target — a much more scalable pattern.