Hierarchical World Models with Implicit Dynamics (ID-WM)

Paper: 2610.42146
Authors: Gaoyue Zhou, Yvonne Wu, Zichen Jeff Cui, Nicolas Ballas, Mido Assran, Lerrel Pinto, Yann LeCun (NYU Courant Institute, Meta FAIR, AMI Labs)
Categories: cs.RO, cs.AI, cs.LG

The Gap

World models (WMs)—neural networks that predict environment transitions given current states and actions—represent a foundational pillar of embodied AI. By enabling test-time reasoning via Model Predictive Control (MPC), world models can synthesize completely novel trajectories for unseen test goals without requiring task-specific policy retraining.

However, planning with world models in complex, multi-step environments faces two catastrophic bottlenecks:

1. Exponential Search Complexity:
   • The action-space search budget expands exponentially with the planning horizon ($|\mathcal{A}|^H$). Searching thousands of continuous action sequences over hundreds of steps quickly becomes intractable.
2. The Compounding Error & “Wormhole” Trap:
   • Small autoregressive prediction errors compound at each temporal step.
   • Crucially, test-time optimization is fundamentally adversarial: an optimizer seeking high-reward trajectories will naturally gravitate toward regions where the learned world model underestimates real-world constraints—discovering hallucinated transitions, teleportations, or physically impossible “wormholes” through obstacles.

Previous hierarchical world models attempted to fix this using pre-trained skills, macro-actions (frame-skipping), or value functions. However, macro-actions sacrifice fine-grained dexterity, while value functions require dense reward coverage that fails to generalize out-of-distribution (OOD).

                 THE WORLD MODEL PLANNING DILEMMA

  Single-Level World Model MPC (Raw Actions):
  s_0 ──(a_0)──> s_1 ──(a_1)──> s_2 ──...──> s_T
  - 100-step horizon = 100 autoregressive steps
  - Compounding error explodes: s_T diverges into unphysical hallucinations!
  - Optimizer exploits model error -> finds "wormholes" through obstacles.

  Rigid Macro-Action Hierarchies:
  s_0 ──────[Skill 1: Run]──────> s_10 ──────[Skill 2: Jump]──────> s_20
  - Inflexible: Cannot adjust millimeter-level gripper orientations.
  - Brittle: Fails immediately when encountering unseen obstacle geometries.

---

The Increment

One sentence: ID-WM (Implicit Dynamics World Models) resolves the long-horizon planning crisis by decoupling state-space topological reachability from local control: a diffusion-based high-level generator samples physically feasible neighboring subgoals from a well-behaved Gaussian latent prior, while a low-level world model operates as an inverse dynamics solver to deduce executable motor actions.

                               ID-WM ARCHITECTURE

  High-Level Planning (State-Space Manifold Reachability):
  Gaussian Noise z ~ N(0, I) ──┐
  Current State s_t ───────────┼──> [f_hi: Diffusion Model] ──> Subgoal s_{t+Δ}
  Temporal Horizon Δ ──────────┘       (Implicit Dynamics)
                                                │
                                                ▼
  Low-Level Execution (Inverse Dynamics Grounding):
  Subgoal Pair (s_t, s_{t+Δ}) ─┐
                               ├──> [f_lo: Inverse Dynamics] ──> Actions [a_t, ..., a_{t+Δ-1}]
  Local Physics Constraints ───┘
                                                │
                                                ▼
                                    [Environment Execution]
                                 s_t ──a_t──> s_{t+1} ──...──> s_{t+Δ}

---

The Anatomy

1. High-Level Model: Implicit Dynamics via Latent Diffusion

Instead of searching over jagged, high-dimensional sequences of motor commands, ID-WM formulates the high-level world model as a neighboring state generator: $s_{t+\Delta} = f_{\text{hi}}(s_t, z, \Delta)$

• $s_t$: Current environment state.
• $\Delta$: Desired temporal abstraction step size (e.g., 10 to 30 timesteps).
• $z \sim \mathcal{N}(0, I)$: A standard Gaussian latent noise vector.

By training $f_{\text{hi}}$ as a conditional diffusion model conditioned on $(s_t, \Delta)$, the unit Gaussian distribution $\mathcal{N}(0, I)$ is mapped smoothly onto the manifold of physically reachable states within $\Delta$ steps.

Why this eliminates wormholes: The latent variable $z$ serves as an implicit high-level action. Because $z$ is drawn from a smooth, continuous Gaussian prior, test-time optimizers (like Cross-Entropy Method / CEM) sample within a strictly well-behaved distribution $\mathcal{N}(0, I)$. The planner cannot construct arbitrary out-of-distribution hallucinations because the diffusion model projects every sample back to the valid data manifold.

2. Low-Level Model: Inverse Dynamics Action Solver

Once a sequence of abstract subgoals $\{s_0, s_\Delta, s_{2\Delta}, \dots, s_T\}$ is synthesized by the high-level planner, the low-level model extracts the actual motor commands: $(a_t, \dots, a_{t+\Delta-1}) = f_{\text{lo}}(s_t, s_{t+\Delta})$

Because $s_{t+\Delta}$ is guaranteed to be within the local reachability radius $\Delta$, the low-level problem reduces to a deterministic local trajectory tracking problem. The inverse dynamics solver solves for executable actions without needing to know global behavioral goals, eliminating error accumulation over long horizons.

3. Hierarchical Test-Time Planning

At inference time, given an arbitrary zero-shot goal $s_g$:

1. High-Level Trajectory Optimization: The planner searches for a sequence of latent noise vectors $(z_0, z_1, \dots, z_{K-1})$ that minimizes the distance between predicted final state $\hat{s}_{K\Delta}$ and goal $s_g$: $\mathcal{J}(z_{0:K-1}) = \|\hat{s}_{K\Delta} - s_g\|_2$ Optimization is performed using CEM over the Gaussian prior.
2. Interleaved Execution: The low-level controller executes actions for the first subgoal segment, observes the true environment state, and replans if deviations occur.

---

Quantitative Evaluation

The authors evaluate ID-WM on three complex benchmark suites: Medium Maze, Wall Navigation, and PushT Manipulation (under fixed and out-of-distribution random target configurations).

1. Long-Horizon Planning Benchmark Results

Environment	Evaluation Setting	Metric	HIQL (Offline RL)	GCBC (Goal-Cond.)	Diffuser (Diffusion Traj.)	H-Diffuser (Hierarchical)	Single-Level WM (MPC)	ID-WM (Ours)
Medium Maze	Arbitrary Goals	Success Rate	0.83	0.80	0.80	0.92	0.97	0.99 ± 0.01
Wall Maze	Detour Navigation	Success Rate	0.62	1.00	0.07	0.10	0.88	1.00 ± 0.00
PushT	Fixed Goal	Success Rate	0.45	1.00	0.00	0.01	0.19	1.00 ± 0.00
	Fixed Goal	Coverage	0.50	0.85	0.10	0.35	0.34	0.85 ± 0.01
PushT (OOD)	Random Goals	Success Rate	0.02	0.21	0.00	0.01	0.21	0.77 ± 0.01
	Random Goals	Coverage	0.22	0.31	0.05	0.20	0.27	0.58 ± 0.01
PushT-Aug	Random Goals	Success Rate	0.32	0.15	0.01	0.27	0.10	1.00 ± 0.01
	Random Goals	Coverage	0.59	0.25	0.14	0.72	0.17	0.77 ± 0.01

Key Findings:

1. OOD Generalization: On PushT with random unseen goals, standard policy methods (GCBC, HIQL) collapse to 2%–21% success. Single-level world models achieve only 21%. ID-WM achieves 77% success, jumping to a perfect 100% success with data augmentation.
2. Massive World Model Gain: Compared to standard single-level MPC world models, ID-WM achieves a 135% performance gain, proving that separating topological subgoal discovery from low-level torque tracking is essential to overcome compounding errors.
3. Outperforming Trajectory Diffusers: Trajectory-level diffusion planners (Diffuser and H-Diffuser) struggle to model fine-grained multi-modal object-agent contact, scoring under 27% on PushT-Aug, whereas ID-WM synthesizes smooth physical contact plans.

---

Key Takeaways

1. Decouple Reachability from Control: A world model should not attempt to predict every microsecond motor joint position over a 10-second task. The high-level model needs only to answer: “Where on the state manifold can I safely reach next?”
2. Gaussian Noise as Implicit High-Level Actions: Using standard Gaussian latents $z$ as implicit high-level actions bounds test-time search to a well-behaved distribution, permanently closing the “wormhole” exploit that plagues classical world model MPC.
3. True Zero-Shot Behavior Synthesis: By breaking long horizons into locally feasible segments, ID-WM stitches together novel trajectories that were never demonstrated in training, realizing Yann LeCun’s vision of autonomous planning via learned world dynamics.

                    传统世界模型长程规划的两难困境

  单层原生动作世界模型 (Single-Level WM with MPC):
  s_0 ──(a_0)──> s_1 ──(a_1)──> s_2 ──...──> s_T
  - 100 步长程任务 = 100 步自回归滚雪球
  - 预测误差雪崩：末端状态 s_T 彻底偏离物理真实流形
  - 规划器利用模型漏洞穿墙，产生“时空虫洞”幻觉！

  刚性技能/宏动作分层 (Rigid Macro-Actions):
  s_0 ──────[预设技能 1: 奔跑]──────> s_10 ──────[预设技能 2: 跳跃]──────> s_20
  - 丧失连续微操精度，无法应对毫米级灵巧操作
  - 泛化极差：遇到未见过的障碍物几何形态直接卡死

                              ID-WM 整体系统架构

  高层规划模块（状态空间流形可达性）：
  标准高斯隐向量 z ~ N(0, I) ──┐
  当前物理状态 s_t ─────────────┼──> [f_hi: 条件扩散生成器] ──> 子目标状态 s_{t+Δ}
  时间抽象步长 Δ ───────────────┘        (隐式动力学表征)
                                                │
                                                ▼
  底层执行控制（逆动力学物理对齐）：
  子目标对 (s_t, s_{t+Δ}) ─────┐
                               ├──> [f_lo: 逆动力学求解器] ──> 动作序列 [a_t, ..., a_{t+Δ-1}]
  局部物理执行约束 ────────────┘
                                                │
                                                ▼
                                    [实机/仿真环境精确执行]
                                 s_t ──a_t──> s_{t+1} ──...──> s_{t+Δ}