DreamX: Repurposing Video Diffusion Models as Universal Latent Action Learners

Paper: 2611.05377
Authors: Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang, Seungryong Kim (KAIST, NVIDIA)
Categories: cs.CV, cs.RO, cs.AI, cs.LG

The Gap

Scaling generalist robot manipulation policies—such as Vision-Language-Action (VLA) models and World Action Models (WAMs)—is bottlenecked by the extreme scarcity of robot teleoperation data. Collecting real-world robot demonstrations is economically prohibitive, slow, and inherently fragmented: Franka arms, UR5 manipulators, ALOHA bimanual setups, and humanoid robots all possess fundamentally different joint configurations, kinematics, and action control spaces.

First-person human videos (such as Ego4D, EgoDex, and EgoVerse) offer an inexhaustible, highly diverse stream of manipulation skills. However, human videos lack robot action labels, creating a severe embodiment and domain gap.

Prior Latent Action Models (LAMs, such as Genie, LAPA, and MoTo) attempted to bypass this mismatch by learning discrete latent actions from adjacent frame transitions ($o_t \to o_{t+1}$). Yet, existing approaches suffer from three fatal flaws:

1. Appearance Entanglement: Because latent action encoders are trained from scratch on pixel reconstruction losses over isolated frame pairs, the learned representations inevitably entangle irrelevant domain visual features—such as human arm skin tones, camera viewpoints, background textures, and table colors—with the actual physical motion.
2. Short-Horizon Myopia: Encoders operating solely over two consecutive frames fail to capture temporal continuity, causal dynamics, and multi-step manipulation intent.
3. Cross-Embodiment Failure: Latent actions learned on human demonstrations fail to zero-shot transfer onto real robot hardware without extensive, expensive paired alignment data.

                  THE CROSS-EMBODIMENT DATA BOTTLENECK

  Abundant Internet Data:                Physical Robot Deployments:
  [Human Egocentric Videos]               [Franka / UR5 / ALOHA / Humanoids]
  - Massive diversity & volume            - Extremely expensive teleoperation
  - NO robot action labels                - Fragmented joint & control spaces
  - Human arm morphology                  - Custom gripper physics
             │                                       │
             ▼                                       ▼
  Prior LAMs (Genie, LAPA):
  Short-frame encoder ($o_t \to o_{t+1}$) ──> Entangles background, viewpoint & skin!
  Zero-shot transfer to robot fails catastrophically.

---

The Increment

One sentence: KAIST and NVIDIA researchers present DreamX, which repurposes a pretrained video diffusion foundation model (Cosmos-Predict 2.5 2B) as a universal latent action learner—compressing spatiotemporal video representations by 300x (from 1,200 tokens per frame down to just 4 compact tokens) while leveraging generative priors to disentangle pure motion dynamics from visual appearance for zero-shot cross-embodiment transfer.

                      DREAMX UNIVERSAL LATENT ACTION FRAMEWORK

  Source Video (Human or Robot)
  [ V_0, V_1, ..., V_T ]
          │
          ▼  + Learnable Queries q_g
  ┌────────────────────────────────────────────────────────┐
  │   Pretrained Video Diffusion Encoder (Cosmos 2.5)      │
  │   Spatiotemporal Causal Cross-Attention Masking        │
  └────────────────────────────────────────────────────────┘
          │
          ▼  300x Compression: 1,200 tokens/frame ──> 4 tokens!
  Compact, Embodiment-Agnostic Latent Actions: Q_src
          │
          ├───────────────────────────────────┬───────────────────────────────────┐
          ▼                                   ▼                                   ▼
    Video Transfer                     Action Transfer                      Policy Learning
  Initial Target Frame V_0^tgt        Target Proprioception p_0^tgt       Compact Latent Policy
          │                                   │                                   │
          ▼                                   ▼                                   ▼
  [ Video Diffusion Decoder ]         [ Video Diffusion Decoder ]         [ DiT Action Head ]
          │                                   │                                   │
  Generated Target Video              Executable Robot Joint Trajectory   Low-Compute Real Rollout
  (Zero-Shot Motion Transfer)         (Zero-Shot Cross-Embodiment)        (80%+ FLOPs Savings)

---

Core Framework & Mechanisms

DreamX rearchitects the pretraining and inference pipeline of generative video diffusion models into a unified representation learner for physical dynamics:

1. Latent Action Encoding via Learnable Spatiotemporal Queries

Instead of training a separate convolutional or transformer encoder from scratch, DreamX directly taps into the internal spatiotemporal representations of a pretrained video diffusion backbone (Cosmos-Predict 2.5 2B):

• For each latent video frame group $V_g \in \mathbb{R}^{H \times W \times D}$ (consisting of 1,200 video tokens), DreamX appends a compact set of $K = 4$ learnable query tokens $q_g$.
• The query tokens pass through the backbone’s cross-attention and self-attention layers alongside the video tokens.
• Frame-Causal Masking: Queries $q_g$ attend only to current and past video tokens $V_{0:g}$ in a strictly causal manner, while query-to-query attention is bidirectional across the sequence.
• The encoder outputs the compact latent action sequence: $Q = \mathcal{E}_\phi(q; V, \ell)$ where $\ell$ denotes text task instructions. This achieves an unprecedented 300x compression ratio ($1,200 \to 4$ tokens).

2. Generative Decoding Disentangles Motion from Appearance

Why does DreamX avoid entangling background appearance and embodiment specifics?

• The decoder $\mathcal{D}_\theta$ is conditioned strictly on the initial observation frame $V_0$ and the compact latent queries $Q$, without text guidance.
• Because $V_0$ already provides the scene’s appearance, camera angle, and static morphology, and the pretrained video diffusion model already possesses a world-model prior over natural scene physics, the latent action queries $Q$ are forced solely to carry the residual motion dynamics needed to reconstruct future frames $\hat{V}_{1:T}$.
• An asymmetric attention mask prevents decoder video and action tokens from attending to each other, forcing all dynamic supervision through the latent action bottleneck.

3. Dual Cross-Embodiment Transfer

Once pre-trained on diverse robot demonstrations (Bridge, DROID, AgiBot, MolmoAct2) and human videos (EgoDex, EgoVerse), the extracted latent representation $Q_{\text{src}}$ functions as a universal action lingua franca:

1. Video Transfer: Combining $Q_{\text{src}}$ with a target initial frame $V_0^{\text{tgt}}$ yields a target video reproducing the source motion under a completely different embodiment and scene: $\hat{V}^{\text{tgt}} = \mathcal{D}_\theta(Q_{\text{src}}, V_0^{\text{tgt}})$
2. Zero-Shot Action Transfer: Combining $Q_{\text{src}}$ with target robot proprioceptive state $p_0^{\text{tgt}}$ decodes the human demonstration directly into continuous, executable robot joint actions without any paired demonstration data: $\hat{A}^{\text{tgt}} = \mathcal{D}_\theta(Q_{\text{src}}, p_0^{\text{tgt}})$

4. Policy Learning in Compact Latent Space

Rather than executing compute-heavy video diffusion rollouts at inference time, DreamX trains a lightweight latent policy $\pi(Z, A \mid V_0, \ell, p_0)$:

• Unlabeled human videos supervise latent code prediction ($Z$) via $\mathcal{L}_{\text{latent}}$.
• Robot demonstrations supervise low-level action prediction ($A$) via a dedicated Diffusion Transformer (DiT) head.
• This decoupled policy architecture cuts inference compute by over 80% compared to full video-diffusion policies.

---

Key Quantitative Results

The authors benchmarked DreamX on the comprehensive LARY-Bench (Latent Action Representation Benchmark) across action regression and classification, as well as real-world manipulation tasks.

1. LARY-Bench Representation Quality

Method	Latent Input Type	Action Regression MSE (CALVIN) $\downarrow$	Action Regression MSE (VLABench) $\downarrow$	Action Regression MSE (AgiBot) $\downarrow$	Average MSE $\downarrow$	Atomic Robot Accuracy $\uparrow$	Composite Human Accuracy $\uparrow$	Average Accuracy $\uparrow$
LAPA (Ye et al., 2025)	2 Frames	0.96	0.95	1.00	0.97	22.27%	14.61%	20.17%
UniVLA (Bu et al., 2025)	2 Frames	0.82	0.74	0.97	0.87	18.62%	19.08%	18.75%
LAPA-DINOv3	2 Frames	0.50	0.25	0.84	0.60	56.27%	64.19%	49.17%
MoTo (Chen et al., 2025)	Full Video	0.43	0.13	0.68	0.44	41.57%	55.11%	51.01%
VideoWorld2 (Ren et al., 2026)	Full Video	0.72	0.41	0.85	0.70	14.50%	30.24%	27.37%
DiLA (Zhang et al., 2026)	Full Video	0.43	0.09	0.52	0.38	58.51%	65.47%	62.25%
DreamX (Ours)	Full Video (Cosmos 2.5)	0.15	0.05	0.22	0.17	68.03%	72.72%	69.67%

2. Real-World Manipulation Performance

• Action Regression Error: DreamX slashes normalized action MSE from 0.38 (DiLA) down to 0.17—a 55.3% error reduction over previous state-of-the-art and an 82.5% error reduction over LAPA.
• Real-World Drawer Closing & Pick-and-Place: On real-robot evaluations using zero-shot transfer from human video demonstrations, DreamX achieved an average subgoal success rate of 96.7%.
• Extreme Latent Compression: Compressing 1,200 tokens down to 4 tokens maintained 98.4% of the motion transfer fidelity while reducing policy rollout latency by $5.2\times$.

---

Practical Implications

1. Unlocking Web-Scale Human Video for Robotics: DreamX eliminates the need to collect billions of dollars in teleoperated robot data. By learning a shared latent action space invariant to visual appearance and embodiment, any first-person video from YouTube or everyday wearable cameras can directly train robotic manipulation policies.
2. Solving Embodiment Fragmentation: Robotics has suffered from incompatible action spaces across different hardware vendors. DreamX’s 4-token latent action acts as a universal intermediate interface that any robot arm can execute via a local proprioceptive decoder.
3. High-Throughput World Action Models: By compressing diffusion tokens by 300x, DreamX enables real-time, low-latency world model planning on edge compute hardware, overcoming the prohibitive inference latency of standard diffusion rollouts.

                    具身智能跨形态数据壁垒的核心症结

  海量互联网数据源：                     物理机器人硬件部署：
  [第一人称人类交互视频]                   [Franka / UR5 / ALOHA / 人形机器人]
  - 极高技能多样性与海量规模               - 极度高昂的遥操作采集成本
  - 没有任何机器人动作控制标签             - 割裂的关节空间与控制维度
  - 人类手臂与生物特征                     - 各异的末端夹爪物理特性
             │                                       │
             ▼                                       ▼
  传统 LAM 隐式动作方案 (Genie, LAPA):
  双帧编码器 ($o_t \to o_{t+1}$) ──> 严重纠缠背景光影、视角与皮肤特征！
  零样本向真实机器人迁移时彻底崩溃失效。

                     DREAMX 通用隐式动作学习与迁移全景架构

  输入视频源（人类日常视频 或 机器人示教视频）
  [ V_0, V_1, ..., V_T ]
          │
          ▼  + 可学习查询向量 q_g
  ┌────────────────────────────────────────────────────────┐
  │   预训练视频扩散编码器 (Cosmos-Predict 2.5 2B)         │
  │   时空因果交叉注意力掩码机制 (Spatiotemporal Masking)   │
  └────────────────────────────────────────────────────────┘
          │
          ▼  300 倍极致压缩：单帧 1,200 Tokens ──> 仅 4 个隐式动作 Token！
  紧凑、跨具身通用的纯净隐式动作：Q_src
          │
          ├───────────────────────────────────┬───────────────────────────────────┐
          ▼                                   ▼                                   ▼
      视频迁移 (Video Transfer)            动作迁移 (Action Transfer)          策略学习 (Policy Learning)
  目标域初始观察帧 V_0^tgt            目标机器人当前本体状态 p_0^tgt      紧凑隐式动作策略
          │                                   │                                   │
          ▼                                   ▼                                   ▼
  [ 视频扩散解码器 D_theta ]          [ 视频扩散解码器 D_theta ]          [ 专用 DiT 动作预测头 ]
          │                                   │                                   │
  生成目标场景视频 (保动作换场景)     直接输出可执行的机械臂关节动作轨迹  低算力部署与高效物理执行
  (零样本任意域动作风格迁移)          (零样本跨具身驱动物理机器人)        (降低策略推理算力 80%+)