VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding

Paper: 2611.08449
Authors: Yu-Cheng Chou, Shihao Wang, Jin Huang, Chi-Pin Huang, Chiao-An Yang, Xianzhe Fan, Jing Wang, Zhiqi Li, Yuanguo Kuang, Ruijie Zheng, Johan Bjorck, Sifei Liu, Hang Su, Abhishek Badki, Lei Zhang, Alan Yuille, Linxi “Jim” Fan, Jan Kautz, Zhiding Yu (NVIDIA, Johns Hopkins University, The Hong Kong Polytechnic University, Purdue University, The University of Hong Kong)
Categories: cs.RO, cs.CV

The Gap

Vision-Language Models (VLMs) have emerged as foundational cognitive backbones for embodied intelligence in robotics and autonomous driving. Rather than reasoning solely in natural language, modern physical AI systems increasingly represent action intents via visual plans—such as 2D/3D end-effector trajectories, target bounding boxes, affordance points, and dense future 3D point tracks.

However, standard autoregressive (AR) VLMs treat spatial coordinates as sequences of discrete text or special coordinate tokens, generating them strictly one token at a time:

• A dual-arm 10-waypoint trajectory spanning 2D image plane and 3D metric coordinates requires 54 sequential token generations.
• Dense 3D motion forecasting across multiple future frames requires hundreds of coordinate tokens.
• On embedded robotics hardware (e.g., NVIDIA Jetson Thor), each 4B VLM forward step takes 35–39 ms. Autoregressive decoding alone absorbs 75% to 87% of the entire control loop budget, resulting in multi-second planning delays that cripple real-time reaction.

Prior speculative decoding frameworks (e.g., Medusa, EAGLE, DFlash) fail in spatial coordinate regimes:

1. Misaligned Verification: Conventional speculative decoding requires strict token-level agreement with the target VLM’s next-token distribution: $\tau = \min \{ j \in [1, \gamma] : a_j \neq \arg\max p_\theta(\cdot | x, a_{<j}) \}$ If a drafter predicts coordinate bin 499 while the target predicts 500 (a negligible 0.1% geometric difference), the draft is instantly rejected, causing the effective acceptance length to collapse to just 2.1–2.2 tokens per step.
2. Fixed-Window Splitting: Standard drafters use fixed-length windows (e.g., 4 or 16 tokens), which arbitrarily slice spatial primitives mid-coordinate or across waypoints, breaking geometric consistency.

                  THE BOTTLENECK IN AUTOREGRESSIVE VISUAL PLANNING

  User Image + Instruction: "Fold the shorts, predict 10 key trajectory points (2D+3D)"
  
  [Autoregressive VLM]: 54 sequential forward passes!
   Pass 1   Pass 2   Pass 3          Pass 53  Pass 54
  ┌──────┐ ┌──────┐ ┌──────┐        ┌──────┐ ┌──────┐
  │ u_1  │─│ v_1  │─│ u_2  │─ ... ─ │ z_9  │─│ z_10 │  ──> Latency: ~3,552 ms on Jetson Thor
  └──────┘ └──────┘ └──────┘        └──────┘ └──────┘      (Incompatible with real-time control)

  [Standard Speculative Decoding (DFlash)]:
  Draft 16 tokens ──> Target verification rejects at bin mismatch (e.g. 500 vs 499)!
                      Acceptance rate drops to ~2.2 tokens/step (Lat: ~1,828 ms)

---

The Increment

One sentence: NVIDIA and academic partners introduce VFlash, a generalist visual planning framework that identifies structured visual primitives (boxes, variable-length 2D/3D trajectories, dense motions) via declared schemas, drafts the entire primitive in a single parallel forward pass using a non-causal lightweight drafter, and directly accepts structurally valid predictions without requiring token-level agreement with the target VLM.

                     VFLASH: FLASH PRIMITIVE DECODING (FPD)

  Context: Image + "Fold the shorts"
       │
       ▼
  Target VLM: Emits primitive start token `<2D>` and tool-call schema length B(s) = 54
       │
       ▼
  [VFlash Lightweight Drafter]: Single non-causal forward pass over ALL 54 masked positions!
  ┌─────────────────────────────────────────────────────────────────────────────┐
  │ Parallel Prediction:                                                        │
  │ q_φ(a_{1:B(s)-1} | H_θ(x), a_0) = ∏_{j=1}^{B(s)-1} q_{φ,j}(a_j | H_θ, MASK)  │
  └─────────────────────────────────────────────────────────────────────────────┘
       │
       ▼
  [Structural Validator / Confidence Guard]: Validates format and coordinate bounds
       │
       ├──> Valid schema? ──> ACCEPT ENTIRE PRIMITIVE (All 54 tokens committed!)
       └──> Boundary check? ──> One target VLM forward pass to sync KV cache
                                Total latency: 703 ms (5.0x speedup; 80% reduction!)

---

Core Framework & Mechanisms

1. Flash Primitive Decoding (FPD) Formulation

VFlash models spatial generation through structured schemas $s$. A schema defines:

• The primitive type (bounding box, variable-length trajectory, dense point flow).
• The coordinate dimensionality (2D pixel coordinates $(u, v) \in [0, 1000]^2$, 3D metric coordinates $(x, y, z) \in \mathbb{R}^3$).
• The block length $B(s)$, declaring the total token budget upfront.

Instead of autoregressively factorizing tokens via $\prod_{j} p_\theta(a_j | x, a_{<j})$, VFlash freezes the backbone hidden representations $H_\theta(x)$ at the primitive start token $a_0$, and predicts all $B(s)-1$ coordinate positions conditionally independently in a single forward step: $q_\varphi(a_{1:B(s)-1} | H_\theta(x), a_0) = \prod_{j=1}^{B(s)-1} q_{\varphi, j}(a_j | H_\theta(x), a_0, \text{MASK}_{B(s)-1})$

2. Dual Alignment Principles

To overcome the instability of non-causal generation over long sequences, VFlash establishes two structural invariants:

1. Boundary Alignment: Each draft block corresponds exactly to one complete visual primitive. The tool call explicitly outputs the primitive length $P$, preventing the drafter from truncating entities mid-trajectory.
2. Position Alignment: Every token index $j$ within the block corresponds to a deterministic geometric semantic role (waypoint index, spatial axis $x/y/z$, or XML delimiter tag). Legal coordinate vocabularies are enforced directly on the output logits via zero-cost mask constraints.

                      PRIMITIVE TOKEN LAYOUT SCHEMAS

  1. Bounding Box (Length = 6):
     <box>  u_1   v_1   u_2   v_2  </>

  2. 2D + 3D Trajectory (Length = 2 + 2P + 3P + 2 = 5P + 4 tokens):
     <2D> (u  v) x P  </>  <3D> (x  y  z) x P  </>

  3. Dense 3D Point Motion (Frame-interpolated):
     <Dense> t_1  t_2  [ k (s d_1 d_2 d_3 d_4 d_5) x 3 ] x P  </>

3. Training and Direct Acceptance Criterion

The drafter $\varphi$ is trained with uniform position weights over ground-truth annotated primitives, rather than matching the target VLM’s autoregressive sequence: $\mathcal{L}_{\text{prim}} = - \sum_{j=1}^{B(s)-1} \log q_{\varphi, j}(a_j^* | H_\theta(x), a_0, \text{MASK}_{B(s)-1})$

At inference time, VFlash replaces token-level rejection with structural validation: $\tau_{\text{dir}} = \begin{cases} B(s), & \text{if } \mathcal{A}_s(\hat{a}) = 1 \\ 1 + r, & \text{otherwise} \end{cases}$ where $\mathcal{A}_s$ checks grammatical schema validity and bounding coordinates. If valid, all $B(s)$ tokens are accepted in one shot. The amortized decoding latency per coordinate is reduced to: $L_{\text{dir}} = \frac{t_{\text{par}} + t_{\text{target}}}{B(s)}$ where $t_{\text{par}}$ is the drafter forward pass and $t_{\text{target}}$ is a single target VLM pass used to update the KV cache.

---

Benchmark Results & Empirical Findings

1. Open-Loop 2D & 3D Robot Trajectory Accuracy

Crucially, single-pass primitive drafting not only accelerates inference, but actually improves geometric accuracy over autoregressive generation by eliminating error accumulation along the trajectory chain.

Method	Target Params	ShareRobot-Bench DFD $\downarrow$	ShareRobot-Bench HD $\downarrow$	ShareRobot-Bench RMSE $\downarrow$	VABench-V DFD $\downarrow$	VABench-V HD $\downarrow$	VABench-V RMSE $\downarrow$
Qwen3-VL (2025)	4B	0.381	0.329	0.220	0.279	0.253	0.204
RoboBrain-2.0 (2025)	7B	0.167	0.158	0.125	0.329	0.260	0.224
LoHo-Manip (2026b)	4B	0.231	0.206	0.156	0.212	0.182	0.147
Baseline (AR)	4B	0.202	0.175	0.142	0.130	0.125	0.093
VFlash (Ours)	4B	0.194	0.162	0.133	0.126	0.116	0.089

Note: DFD = Discrete Fréchet Distance; HD = Hausdorff Distance; RMSE = Root Mean Square Error.

2. Closed-Loop Robotics Manipulation (MolmoSpaces & LIBERO)

• On MolmoSpaces, a policy conditioned on VFlash visual plans achieves 70.2% average success across four distinct task suites, compared to 62.6% for the baseline autoregressive VLM plan.
• On LIBERO-PRO (out-of-distribution physical perturbations), VFlash improves Spatial, Object, and Goal task completion rates by +11.1, +9.4, and +6.9 percentage points over the unconditioned baseline.

3. End-to-End Autonomous Driving (NAVSIM & Waymo)

Conditioning a lightweight diffusion action head on VFlash plans achieves state-of-the-art closed-loop driving metrics:

Method	Driving Benchmark	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	PDMS $\uparrow$ / RFS $\uparrow$
DiffusionDrive (2025)	NAVSIM	98.2	96.2	82.2	94.7	88.1
SpanVLA (2026c) [RL]	NAVSIM	99.1	97.1	86.3	95.2	90.3
Qwen-Drive-1.0-RL	NAVSIM	98.6	98.2	84.8	95.9	90.7
VFlash-RL (Ours)	NAVSIM	98.3	97.8	85.9	95.4	90.7
MindVLA-U1 (2026) [RL]	Waymo (5s)	ADE: 1.09	—	—	—	7.87
Qwen-Drive-1.0-RL	Waymo (5s)	ADE: 1.19	—	—	—	7.91
VFlash-RL (Ours)	Waymo (5s)	ADE: 1.22	—	—	—	7.86

4. End-to-End Inference Latency Profiling

Latency measurements across embedded edge hardware and server accelerators (Batch Size = 1):

System Benchmark	Platform	Autoregressive (AR)	DFlash (Speculative)	VFlash (FPD)	Total Speedup	Latency Reduction
MolmoSpaces	Jetson Thor	3,552.7 ms	1,828.4 ms	703.3 ms	5.05x	-80.2%
MolmoSpaces	DGX Spark	3,356.5 ms	1,809.2 ms	649.9 ms	5.16x	-80.6%
MolmoSpaces	NVIDIA H100	516.1 ms	340.0 ms	142.3 ms	3.63x	-72.4%
NAVSIM Driving	Jetson Thor	1,842.5 ms	1,609.5 ms	534.4 ms	3.45x	-71.0%
Dense 3D Flow	H100	~2,500 ms	~1,200 ms	~100 ms	~25x	-96.0%

On Jetson Thor, VFlash reduces pure decoding time for an 84-token driving trajectory from 1,420 ms down to 78 ms—an astounding 18.2x decoding speedup.

---

Practical Implications

1. System-1 Latency for High-Level Planners: By collapsing multi-token coordinate generation into a single draft forward pass, physical AI architectures can execute high-level VLM re-planning at 1.4–2.0 Hz directly on embedded robotics platforms (Jetson Thor), transforming slow deliberative planners into reactive real-time controllers.
2. Decoupling Geometric Fidelity from Token Alignment: The paper provides a crucial insight for embodied post-training: speculative drafting of spatial coordinates should optimize for ground-truth geometric distance, not token-probability equivalence with an autoregressive base model.
3. Plug-and-Play VLA Acceleration: VFlash requires no modifications to the base VLM’s weights, training only a shallow non-causal drafting head with standard cross-entropy loss over spatial datasets.

                    自回归视觉规划在端侧设备的严重瓶颈

  输入场景图像 + 任务指令："折叠短裤，输出双臂未来 5 秒内 10 个关键轨迹点 (2D+3D)"
  
  【传统自回归解码】：必须进行 54 次完全串行的前向传播！
   Step 1   Step 2   Step 3          Step 53  Step 54
  ┌──────┐ ┌──────┐ ┌──────┐        ┌──────┐ ┌──────┐
  │ u_1  │─│ v_1  │─│ u_2  │─ ... ─ │ z_9  │─│ z_10 │  ──> Jetson Thor 端到端耗时: 3,552 ms
  └──────┘ └──────┘ └──────┘        └──────┘ └──────┘      (无法满足高频动态交互需求)

  【常规推测解码 (DFlash)】：
  推测 16 个 token ──> 坐标数值轻微波动触发严格拒绝 (499 vs 500) ──> 接受长度仅 2.2 tokens
                      耗时仍高达 1,828 ms，无法彻底解决核心瓶颈

                     VFLASH：闪速图元解码（FPD）架构

  上下文环境：RGB 输入 + "折叠牛仔裤"
       │
       ▼
  目标 VLM：触发图元起始 Token `<2D>`，通过工具调用明确图元长度 B(s) = 54
       │
       ▼
  【VFlash 轻量非因果草稿头】：单次并行前向推导全部 54 个 Mask 位置！
  ┌─────────────────────────────────────────────────────────────────────────────┐
  │ 条件独立并行概率联合建模：                                                      │
  │ q_φ(a_{1:B(s)-1} | H_θ(x), a_0) = ∏_{j=1}^{B(s)-1} q_{φ,j}(a_j | H_θ, MASK)  │
  └─────────────────────────────────────────────────────────────────────────────┘
       │
       ▼
  【结构校验器 / 置信度门控】：检测坐标取值区间与几何合法性
       │
       ├──> 格式合法？ ──> 直接接受全部 54 个 Token！（无需目标模型逐字比对）
       └──> 目标 VLM 仅需执行 1 次前向计算以更新 KV 缓存并提供后续隐状态
            Jetson Thor 端到端延迟降至 703 ms（提速 5.05 倍，时延削减 80%！）

                      三类核心空间图元的排列模式表

  1. 目标包围盒 (长度固定 = 6 tokens):
     <box>  u_1   v_1   u_2   v_2  </>

  2. 2D + 3D 机械臂空间轨迹 (长度 = 5P + 4 tokens, P 为航路点数):
     <2D> (u  v) x P  </>  <3D> (x  y  z) x P  </>

  3. 稠密 3D 场景流预测 (跨帧运动序列):
     <Dense> t_1  t_2  [ k (s d_1 d_2 d_3 d_4 d_5) x 3 ] x P  </>