NEO-VLA: Towards Native, Encoder-Free Vision-Language-Action Policies

Paper: 2611.06918
Authors: Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang, Tianqi Liu, Mingze Sun, Ziyi Wang, Zhongang Cai, Hanming Deng, Quan Wang, Lei Yang, Lewei Lu, Fangzhou Hong, Zhaoxi Chen, Dahua Lin, Ziwei Liu (NTU S-Lab, SenseTime Research, Ropedia)
Categories: cs.CV, cs.RO, cs.AI, cs.LG

The Gap

The modern Vision-Language-Action (VLA) foundation model paradigm—championed by OpenVLA, Octo, and $\pi_0$—is built upon an implicit architectural assumption: raw visual perception is first compressed by a frozen, separately pretrained vision encoder (e.g., CLIP, SigLIP, or DINOv2), and policy reasoning begins only after this visual endpoint.

However, robotic manipulation differs fundamentally from internet-scale image captioning or classification:

1. The Representation Mismatch: Vision encoders trained on web images are optimized for semantic invariance (e.g., recognizing that a “mug” is a mug regardless of lighting, precise angle, or camera distance). Conversely, dexterous manipulation hinges precisely on fine-grained spatial signals: metric depth, edge contact geometry, micro-slips, and millimeter-level end-effector alignments.
2. The Final-Layer Abstraction Trap: Pretrained vision encoders progressively discard high-frequency spatial and geometric details in their deepest layers to maximize semantic abstraction. When a VLA model connects to this final layer, the geometric cues required for physical control have already been erased.
3. Architectural Rigidity: Encoder-based VLAs treat visual features as an immutable prefix. Scaling the language backbone or tuning the action head cannot recover high-resolution geometric signals lost upstream.

                      THE CONVENTIONAL VLA BOTTLENECK

  Raw Camera Pixels
        │
        ▼
  [ Pretrained Vision Encoder (CLIP / SigLIP) ]
  Layer 1 ──> Layer 6 ──> Layer 12 ──> Layer 24 (Final Layer)
  (Rich 3D Geometry)   (Loss of spatial details)    (Pure semantic abstraction)
                                                         │
                                                         ▼
                                               [ Language Backbone ]
                                                         │
                                                         ▼
                                               [ Discrete / Flow Action Head ]

  Problem: Physical manipulation requires metric 3D boundaries and depth,
           which were systematically discarded by the final encoder layer!

---

The Increment

One sentence: The authors introduce NEO-VLA, a structurally encoder-free Vision-Language-Action architecture where pixels are directly embedded into the language backbone via lightweight patch convolutions, allowing a flow-matching action expert to tap into instruction-conditioned visual representations across multiple computational depths where action-relevant signals naturally peak.

                  NEO-VLA NATIVE ENCODER-FREE ARCHITECTURE

  Multi-View Pixels      Camera Intrinsics & Extrinsics      Robot State & Language
  [ Previous & Current ]   [ Geometry MLPs ]                  [ State MLPs & Prompt ]
         │                         │                                  │
         └─────────────────────────┼──────────────────────────────────┘
                                   ▼
          ┌─────────────────────────────────────────────────────────┐
          │     Native Unified Backbone (NEO-2B, 40 Layers)         │
          │     Joint Self-Attention: Patches + Text + Geometry     │
          │                                                         │
          │  Layer 0-10:   Peaks in wrist-camera spatial geometry   │
          │  Layer 15-25:  Peaks in object-instruction grounding    │
          │  Layer 30-40:  High-level task progression semantics   │
          └─────────────────────────────────────────────────────────┘
                                   │  Per-Layer Cached K, V
                                   ▼
          ┌─────────────────────────────────────────────────────────┐
          │     Flow-Matching Action Expert (Narrow Transformer)    │
          │     Multi-Depth Cross-Attention Readout                 │
          └─────────────────────────────────────────────────────────┘
                                   │
                                   ▼
             Continuous Action Chunks (Arm: 26D, Body: 8D)

---

Core Framework & Mechanisms

NEO-VLA replaces the conventional multi-component vision-language-action pipeline with a unified, end-to-end framework:

1. Structurally Native Visual Interface

• Zero Pretrained Vision Encoder: NEO-VLA contains no external Vision Transformer (ViT), vector-quantized codebook, or autoencoder.
• Direct Patch Embedding: Raw images (up to 3 camera views per timestep: exterior, wrist, auxiliary) are dynamically tokenized using lightweight patch convolutional layers shared across views and time. Images preserve their native aspect ratios under a dynamic budget (up to 88 visual tokens per view).
• Geometric and Proprioceptive Prefix: Camera intrinsics, full 6-DoF poses, and robot joint states (Arm 0, Arm 1, mobile body) are mapped via lightweight MLPs and prepended alongside structured language instructions into a single unified sequence.

2. Action Readout Across Computational Depths

Where do action-relevant features actually live inside a foundation model? The authors performed layer-wise perturbation and swapping interventions:

• Spatial Geometry Peaks Early: Interventions on cached key-values reveal that end-effector spatial translation errors are most sensitive to layers 0–10. Masking wrist camera representations at layers 6–10 causes immediate manipulation failure.
• Instruction Grounding Peaks Mid-Network: Language conditioning sensitivity exhibits strong peaks at layer 0 and layer 17.
• The Multi-Depth Policy Map: Instead of binding solely to the final layer, NEO-VLA’s action expert reads cached Keys ($K$) and Values ($V$) from a distributed set of backbone layers across the computational hierarchy, granting the policy direct access to both low-level spatial geometry and high-level semantic intent.

3. Continuous Flow-Matching Action Expert

• The action expert is a narrow, randomly initialized Transformer trained via conditional flow matching.
• It receives projected noisy action tokens and flow time $\tau$, equipped with shared temporal rotary embeddings (t-RoPE) spanning execution horizons of 30 steps.
• The expert outputs a continuous velocity field covering 26-DoF dual-arm trajectories (joint angles, $\Delta$position, $\Delta$rotation, gripper states) and 8-DoF mobile base/torso coordinates.

---

Key Quantitative Results

NEO-VLA was pretrained across 9 public robot dataset families (comprising 2,231,583 episodes and 9,927 hours of robot manipulation) and systematically benchmarked on LIBERO, LIBERO-plus, and RoboTwin 2.0.

1. Backbone Comparison on LIBERO (Success Rate %)

Backbone Architecture	Model Scale	Adaptation Mode	LIBERO Spatial	LIBERO Object	LIBERO Goal	LIBERO Long	Average Success
Qwen3-VL (Encoder-based)	2B	Frozen Backbone	71.4%	89.8%	92.8%	64.0%	79.5%
Qwen3-VL (Encoder-based)	2B	Full Fine-Tuning	92.2%	96.6%	95.8%	78.6%	90.8%
Gemma-3 (Encoder-based)	4B	Frozen Backbone	80.2%	93.6%	92.6%	71.6%	84.5%
Gemma-3 (Encoder-based)	4B	Full Fine-Tuning	89.0%	97.6%	94.8%	82.8%	91.0%
NEO-VLA (Ours)	2B	Frozen Backbone	89.2%	95.8%	93.2%	85.4%	90.9%
NEO-VLA (Ours)	2B	Full Fine-Tuning	95.6%	97.8%	96.0%	92.0%	95.3%

Remarkable finding: With a completely frozen backbone (updating only the action expert), NEO-VLA (2B) achieves 90.9%, matching or exceeding fully fine-tuned 2B and 4B encoder-based models. Under full fine-tuning, NEO-VLA achieves 95.3% on LIBERO (and 97.4% at step 27.5k).

2. Robustness Under Environmental Shifts: LIBERO-plus

LIBERO-plus tests manipulation policies across 7 realistic perturbation axes (camera jitter, robot base displacement, paraphrased language, lighting changes, noisy background, sensor noise, and layout shifts):

Model	Model Scale	Open Weights	Camera	Robot Pose	Language	Light	Background	Noise	Layout	Total Score
$\pi_0$	3B	Yes	13.8%	6.0%	58.8%	85.0%	81.4%	79.0%	68.9%	53.6%
StarVLA	4B	Yes	52.5%	49.8%	88.5%	95.7%	95.7%	73.0%	76.9%	74.1%
ABot-M0	4B	Yes	60.4%	67.9%	86.4%	96.2%	91.6%	86.4%	82.6%	80.5%
Cosmos-Policy	2B	Yes	75.8%	63.3%	81.7%	96.5%	88.9%	92.7%	82.2%	82.2%
$\pi_{0.5}$	3B	Yes	78.4%	73.6%	80.8%	96.2%	94.1%	89.0%	84.5%	84.4%
Qwen-RobotManip	4B	No (Closed)	87.2%	75.5%	85.6%	96.6%	97.7%	97.7%	87.3%	89.0%
NEO-VLA (Ours)	2B	Yes	94.5%	72.6%	91.7%	96.2%	96.6%	96.9%	82.4%	89.7%

NEO-VLA achieves 94.5% on camera perturbations and 89.7% overall, outperforming all open-weight models and rivaling closed industrial solutions despite having only 2B parameters.

---

Practical Implications

1. Retiring Separate Vision Encoders in Robotics: The assumption that robotics requires frozen visual encoders pretrained on static image datasets is flawed. Native patch tokenization allows the network to preserve metric geometric structures, delivering higher control precision at lower parameter counts.
2. Action-Critical Signals Peak Early: Roboticists designing policy heads should not simply tap the final layer of a vision model. Cross-attending into early- and mid-level representations is necessary to preserve 3D boundaries and contact points.
3. Multimodal Co-Training is Mandatory: The paper demonstrates that training purely on action tokens degrades general visual comprehension and causes policy robustness to collapse under scene randomization (falling from 69.4% to 10.7% on RoboTwin C2R). Retaining general multimodal understanding data during policy tuning is essential for real-world generalization.

                     传统编码器驱动 VLA 架构的根本缺陷

  原始相机像素输入
        │
        ▼
  [ 独立预训练视觉编码器 (CLIP / SigLIP / DINOv2) ]
  第 1 层 ──> 第 6 层 ──> 第 12 层 ──> 第 24 层 (编码器最终输出)
  (丰富3D几何与边缘)   (高频细节逐渐丢失)   (高度抽象的分类语义)
                                                   │
                                                   ▼
                                         [ 语言基座模型 LLM ]
                                                   │
                                                   ▼
                                         [ 离散 / 连续动作生成头 ]

  核心矛盾：机械臂抓取所需的局部精确几何与深度信息，
           早在进入大模型之前就被编码器的最终层彻底过滤！

                     NEO-VLA 原生无编码器具身基座架构

  多视角相机像素输入      相机内外参几何矩阵         机器人本体状态与任务指令
  [ 历史与当前观测帧 ]   [ 轻量几何 MLPs ]           [ 状态 MLPs 与结构化 Prompt ]
         │                         │                                  │
         └─────────────────────────┼──────────────────────────────────┘
                                   ▼
          ┌─────────────────────────────────────────────────────────┐
          │     原生统一大模型基座 (NEO-2B，深度 40 层 Transformer) │
          │     图像块 + 文本 + 空间几何 联合注意力自演化计算       │
          │                                                         │
          │  第 0-10 层：腕部相机空间几何与边缘特征达到峰值         │
          │  第 15-25 层：物体与指令对齐的跨模态语义达到峰值        │
          │  第 30-40 层：高阶任务时序演进与因果规划                │
          └─────────────────────────────────────────────────────────┘
                                   │  各层缓存的 Key, Value
                                   ▼
          ┌─────────────────────────────────────────────────────────┐
          │     流匹配动作专家 (Flow-Matching Action Expert)        │
          │     多深度交叉注意力动态读取 (Multi-Depth Readout)      │
          └─────────────────────────────────────────────────────────┘
                                   │
                                   ▼
             输出高精连续动作轨迹 (双臂 26 维 + 移动底盘/躯干 8 维)