UniAction: Unified Action Conditioning for World Models

Paper: 2610.32977
Authors: Bowen Xue, Zizhang Li, Hong-Xing Yu, Jiajun Wu
Categories: cs.CV

The Gap

Video world models predict future visual frames conditioned on past visual contexts and incoming control actions. To serve as scalable simulators for embodied robotics, autonomous driving, and interactive gaming, these models must be steerable by actions.

However, existing action-conditioned world models suffer from a fundamental architectural fragmentation:

1. Heterogeneous Action Modalities: Real-world actions span completely disparate mathematical spaces:
   • Camera Control: 6-DoF extrinsic matrices $[R | t]$, field of view, and optical center trajectories.
   • Robotic Manipulation: Gripper 3D positions, Euler angles/quaternions, joint torques, and binary gripper states.
   • Hand Manipulation: 21-keypoint kinematic skeletons, finger articulation angles.
   • Physical Interactions: Continuous directional force vectors applied at arbitrary 3D surface contact points.
2. Siloed, Single-Action Architectures: Prior models (such as DriveDreamer, RoboDreamer, or CameraCtrl) design specialized conditioning modules for a single action type. A camera-conditioned model cannot process gripper motions, and a robot policy simulator cannot accept physical pushes or novel view camera trajectories.
3. Data Fragmentation and Zero Cross-Transfer: Action-annotated video datasets are exceptionally rare and expensive to gather. Forcing each action type into an isolated model divides this scarce data pool, preventing world models from benefiting from unified scaling laws across physical domains.

   THE HETEROGENEOUS ACTION SILO DILEMMA IN WORLD MODELS

   Disparate Action Sources:
   [Camera Trajectories]    [Robot Gripper Pos]    [Hand Keypoints]    [Physical Force Pushes]
   (SE(3) Transformation)   (End-Effector Pose)    (21-Joint Tree)     (Contact Point + Vector)
            |                        |                    |                       |
            v                        v                    v                       v
   [CameraCtrl VDM]        [RoboWorld VDM]       [HandGen VDM]        [PhysSim VDM]
   - Model A                - Model B             - Model C            - Model D
   - High camera parallax   - Strong gripper manipulation - Hand kinematics  - Force dynamics
   - Fractured Data         - Fractured Data      - Fractured Data     - Fractured Data

   CRITICAL FAILURE:
   None of these models can simulate a scenario where a camera moves WHILE a robot hand pushes an object!
   Data scaling is completely fragmented across incompatible silos.

The Increment

One sentence: UniAction introduces a universal action-conditioning representation that decomposes arbitrary physical control signals into sparse sets of discrete-semantic and continuous-spatiotemporal action elements mapped directly into the video model’s native coordinate space, enabling a single unified model to outperform action-specific expert models.

Core Mechanism

UniAction establishes a unified coordinate and attention formulation that handles any physical control signal:

1. Sparse Action Elements Formulation:
   • Rather than encoding entire action vectors via domain-specific MLPs, UniAction represents any action $A$ as a set of $M$ discrete action elements: $A = \left\{ (\text{id}_m, \mathbf{c}_m) \right\}_{m=1}^M$
   • $\text{id}_m \in \mathcal{V}_{\text{action}}$: A discrete semantic identity token designating what the element is (e.g., camera_center, left_finger_tip, push_force_origin).
   • $\mathbf{c}_m = (x_m, y_m, z_m, t_m)$: A continuous spatiotemporal coordinate anchoring where and when the action element operates in the physical 3D world over time.
2. Native Positional Space Grounding:
   • Instead of projecting actions through disparate auxiliary encoders, UniAction embeds the continuous coordinate $\mathbf{c}_m$ directly into the pretrained video diffusion model’s native 3D/4D positional embedding space (e.g., 3D RoPE or learned spatiotemporal grids).
   • This places action elements into the exact same metric coordinate frame as the latent video tokens, allowing self-attention layers to seamlessly correlate visual patch features with spatial action targets.
3. Cardinality-Calibrated Cross-Attention:
   • Different action modalities possess wildly different element counts: a global camera trajectory or force push has 1 to 2 elements per frame, whereas two articulated hands have 42 elements.
   • Standard cross-attention softmax exhibits severe cardinality bias, where high-element actions dominate attention weights while low-element actions are ignored.
   • UniAction introduces cardinality-calibrated attention, normalizing the cross-attention logits by the modality element count $M$, ensuring invariant conditioning strength regardless of action granularity.
4. Joint Cross-Modality Synergy:
   • Trained across four representative action forms: 3D force vectors, camera control, human hand poses, and robotic arm trajectories.
   • A single jointly-trained UniAction model consistently beats four separate action-specific expert baselines on action-following precision (FVD, PSNR, LPIPS), confirming that cross-action joint training creates positive physical transfer.

   UNIACTION UNIFIED CONDITIONING PIPELINE

   Heterogeneous Action Inputs:
   - Camera Pose (R, t)     ===> Action Elements: [(id_cam, c_cam)]
   - Robot Arm Gripper (x,y,z)===> Action Elements: [(id_grip, c_grip)]
   - Point Contact Force (F)===> Action Elements: [(id_force, c_force)]
                                    |
                                    v
   +---------------------------------------------------------------+
   | Coordinate Mapping into Native Video Spatiotemporal Space     |
   | Embed c_m into 3D RoPE / Spatiotemporal Positional Encodings  |
   +---------------------------------------------------------------+
                                    |
                                    v
   +---------------------------------------------------------------+
   | Cardinality-Calibrated Cross-Attention Layer                  |
   | Softmax normalization balances 1-element force pushes with    |
   | 42-element dexterous hand poses without signal saturation     |
   +---------------------------------------------------------------+
                                    |
                                    v
   +---------------------------------------------------------------+
   | Pretrained Video Diffusion Transformer (DiT Backbone)         |
   | Latent video tokens seamlessly attend to grounded action keys |
   +---------------------------------------------------------------+
                                    |
                                    v
   Physically Coherent Future Video Simulation (Joint Multi-Action!)

Key Concepts

• Sparse Action Elements: A modular representation that reduces arbitrary continuous control signals into sets of semantic IDs and spatiotemporal coordinates.
• Native Positional Grounding: Aligning action tokens directly into the existing spatiotemporal coordinate system of a vision model, avoiding separate metric translation heads.
• Cardinality-Calibrated Attention: An attention scaling technique that prevents cross-attention weights from being skewed by the varying number of discrete tokens in different action modalities.

Framework Shift

Before (Fragmented Action Silos):
  Architecture: Custom MLP / Conv heads designed for each action type.
  Isolation: Camera models cannot handle grippers; robot models cannot handle forces.
  Data Waste: Action-labeled video datasets split into tiny incompatible sub-pools.

After (UniAction Universal Conditioning):
  Architecture: Unified action elements mapped to native video spatiotemporal space.
  Compositionality: Simultaneously conditions on camera trajectories AND robot gripper actions.
  Scaling Synergy: Single jointly trained model outperforms isolated domain experts.

Expert Review

Topic Value: Groundbreaking. Action conditioning is the prerequisite for converting passive video generators into active, interactive world simulators.

Method Maturity: Representing actions as spatiotemporally grounded semantic elements is conceptually clean, mathematically sound, and directly compatible with modern Diffusion Transformers (DiT).

Experimental Rigor: Tested across four diverse action classes spanning micro-contact forces, multi-joint hand poses, robot grippers, and macro camera trajectories. The cardinality-calibration ablation proves essential.

Writing Quality: Clear mathematical formulations, insightful visualizations, and thorough benchmarking on action-following adherence.

Overall Verdict: strong accept — A landmark architecture for general-purpose, action-conditioned world models.

Takeaways

• Do not engineer separate network architectures for different control signals; all physical actions can be decomposed into spatiotemporally grounded semantic elements.
• Always calibrate cross-attention when conditioning on modalities with variable token cardinalities to avoid attention dilution or saturation.
• Jointly training world models across diverse action modalities produces positive transfer, beating models trained solely on narrow single-action datasets.