ARDY: Real-Time Motion Generation That Actually Listens to You

Paper: 2607.08741 Authors: Kaifeng Zhao, Mathis Petrovich, Haotian Zhang, Tingwu Wang, Siyu Tang, Davis Rempe Categories: cs.GR, cs.CV, cs.LG, cs.RO

The Gap

Imagine you’re building a game where an NPC needs to respond to a player’s spoken command — “run to the door, grab the sword, then crouch.” You need the motion to be high-quality, controllable via text and spatial constraints, and you need it now, frame by frame, not after 30 seconds of batch processing.

Current research sits on two sides of a canyon. On one side: offline diffusion models like MDM, MotionDiffuse, and T2M-GPT generate gorgeous, semantically rich motions controllable by text and kinematic constraints — but they operate in batch mode, generating entire sequences at once. They’re too slow for real-time interaction. On the other side: online/streaming methods (autoregressive tokenizers, recurrent models) can produce motions frame-by-frame, but they sacrifice either controllability or quality. Their fixed context windows mean they “forget” what happened 5 seconds ago, and they struggle with complex semantic instructions or long-horizon goals like following a path while respecting keyframe poses.

The logical path from gap to conclusion:

[Problem]
    |
    v
Existing methods force a tradeoff:
high quality + control <--> real-time speed
    |
    v
[Assumption]
    |
    v
Hybrid representation + streaming diffusion
can decouple this tradeoff
    |
    v
[Method]
    |
    v
ARDY: explicit root + latent body
+ two-stage autoregressive denoiser
+ variable history context
    |
    v
[Evidence]
    |
    v
HumanML3D + Bones Rigplay benchmarks
show competitive quality + real-time speed
+ strong constraint adherence
    |
    v
[Conclusion]
    |
    v
The tradeoff is not fundamental;
the architecture was wrong

The Increment

One sentence: Before this paper, you had to choose between “beautiful and controllable but offline” or “fast but dumb” — ARDY gives you “beautiful, controllable, and streaming” in one framework.

Core Mechanism

ARDY’s architecture has three interlocking ideas. First, a hybrid representation splits human motion into two channels: the root trajectory (global position and orientation, handled explicitly with precise numerical features) and the body pose (local joint rotations, compressed into a learned latent embedding via a VAE). This split is the foundation — it lets the model be surgically precise about *where the character goes while leaving the how of body articulation to a compact, efficient learned space.

Second, a two-stage autoregressive transformer denoiser operates in a streaming fashion. For each new time chunk, Stage 1 predicts the root trajectory autoregressively (conditioned on text, history, and kinematic constraints). Stage 2 takes that root trajectory and denoises body pose latents via a diffusion process. Both stages share an autoregressive backbone that attends over a variable-length history of previous chunks, not a fixed window — this is what gives ARDY its long-horizon memory.

Third, the training strategy uses ground-truth kinematic constraints sampled directly from the data (e.g., “the left hand should be at position X at frame 30”) rather than requiring separate constraint estimation. This means the model learns to obey constraints as a native capability, not as an afterthought bolted on via guidance or loss functions.

Text + Constraints + History
            |
            v
  +--------------------+
  | Autoregressive     |
  | Transformer        |
  | (variable context) |
  +--------------------+
            |
     +------+------+
     |             |
     v             v
[Stage 1]     [Stage 2]
Root Pred.    Body Denoise
(explicit)    (diffusion)
     |             |
     v             v
  Root Traj    Body Latents
     |             |
     +------+------+
            |
            v
     Full Motion
     Output Chunk

Structural Metaphor

Think of ARDY as a puppet show with a director whispering in real-time.

The puppeteer’s left hand controls the puppet’s overall position on stage — where it walks, which direction it faces. This is the root trajectory: explicit, precise, numerically exact. If the director says “move to the left side of the stage by beat 8,” the left hand executes that literally.

The puppeteer’s right hand controls the puppet’s body language — how the arms swing, how the torso leans, the subtle shifts in posture. This is the body pose latent: learned, compressed, expressive. The right hand doesn’t care about exact stage coordinates; it cares about making the movement look natural and semantically appropriate.

The director is the autoregressive transformer. It whispers new instructions at each beat (“now look surprised, now reach for the cup”), and it *remembers the entire story so far — not just the last 5 seconds. Unlike a forgetful director with a fixed attention span (prior methods with limited context windows), this director has a notebook that grows with the performance.

The script sitting on the music stand is the text prompt and kinematic constraints. During rehearsals (training), the director practices with real scripts from real shows (ground-truth constraints from motion capture data), so when it’s showtime (inference), following new instructions is second nature.

Key Concepts

• Hybrid Representation: Most motion generation models represent the entire body as one blob — either all explicit joint positions or all latent codes. ARDY splits it: root (explicit) + body (latent). Why? Because root trajectory has strong spatial semantics (it’s literally “where the person stands”) and benefits from precise numerical representation. Body pose is high-dimensional and redundant — a latent space compresses it efficiently. This split is like having a GPS for the destination and a stylist for the outfit: different problems need different tools.

• Autoregressive Diffusion: Autoregressive models predict the next token conditioned on all previous tokens (like GPT). Diffusion models start from noise and iteratively denoise. ARDY combines them: it generates motion chunks autoregressively (chunk N depends on chunks 1 through N-1), and within each chunk, body poses are refined via diffusion denoising. It’s like writing a novel sentence by sentence (autoregressive), but each sentence is drafted, revised, and polished multiple times before committing (diffusion).

• Variable History Context: Previous streaming methods use a fixed-size context window — say, the last 2 seconds of motion. ARDY’s transformer attends over *all previous chunks, with the context growing as generation proceeds. This means the model can reference a gesture from 10 seconds ago if it’s relevant to the current motion. Think of it as the difference between a goldfish memory and an elephant’s — both can swim, but one remembers where the food was hidden last week.

Framework Shift

Before (mainstream approach):         After (this paper):
                                     
[Text]                               [Text + Kinematic Constraints]
    |                                    |
    v                                    v
+----------+                       +------------------+
| Single   |                       | Hybrid: Root     |
| Latent   |                       | (explicit) +     |
| Rep.     |                       | Body (latent)    |
+----------+                       +------------------+
    |                                    |
    v                                    v
+----------+                       +------------------+
| Batch    |                       | Autoregressive   |
| Diffusion|                       | Streaming        |
| (offline)|                       | Diffusion (real- |
|          |                       | time, variable   |
+----------+                       | context)         |
    |                              +------------------+
    v                                    |
[Full Sequence]                          v
    |                              [Chunk-by-Chunk]
    v                                    |
[Render]                                 v
                                   [Live Render]

From batch-to-chunk-to-chunk offline generation to streaming autoregressive diffusion with hybrid representation, the core shift is treating trajectory control and body expressiveness as fundamentally different problems requiring fundamentally different representations.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The offline-vs-online tradeoff in motion generation has been a genuine pain point for anyone building interactive applications (games, VR, robotics teleoperation). The paper sits at the right intersection of a maturing field — diffusion models for motion are now well-understood enough that someone needed to ask “okay, but can we actually *use this in real-time?” This is an engineering-adjacent contribution, but a necessary one.

Method maturity: The hybrid representation insight is clever and well-motivated — splitting root from body is physically natural and technically effective. The two-stage autoregressive transformer is less novel; it’s a reasonable architecture but not a breakthrough insight. Variable history context is the kind of thing that feels obvious in retrospect but required careful implementation. I’d say this is 60% good engineering, 40% genuine architectural insight. Simpler approaches (e.g., just scaling up a single-stage autoregressive model) probably exist but would sacrifice the root precision that makes constraint adherence work.

Experimental integrity: The dual-benchmark evaluation (HumanML3D for standard comparison + Bones Rigplay for high-fidelity interactive testing) is solid and shows the authors are serious about the interactive use case, not just benchmark gaming. The interactive demo with mouse/keyboard control is a strong credibility signal — you can’t fake that. However, I’d like to see more ablation on the hybrid representation specifically: how much does the explicit root vs. a fully latent approach actually matter for constraint adherence? The numbers are good but the causal story could be tighter.

Writing quality: The paper reads well and the method is explained clearly. The supplementary video does heavy lifting — which is appropriate for a motion generation paper but means the prose alone undersells the contribution. Section 4 (experiments) could be sharper; the metrics discussion is a bit listy and would benefit from a “what surprised us” narrative. The related work section is comprehensive but could better articulate *why previous approaches specifically fail at the streaming+control combination, not just that they do.

Verdict: weak accept — The hybrid representation is a genuinely useful design principle, and the real-time interactive demo elevates this above incremental work, though the core transformer architecture is competent engineering rather than a conceptual leap.

Takeaways

1. Split representation by physical semantics, not by convenience. The explicit root / latent body split is a transferable idea: whenever your data has components with fundamentally different structure (e.g., global position vs. local articulation, or high-level intent vs. low-level execution), represent them differently. This applies to robotics, animation, even text generation (plan tokens vs. surface tokens).

2. Ground-truth conditioning during training beats post-hoc guidance. Rather than training a model and then coaxing it to follow constraints via classifier-free guidance or loss penalties, ARDY samples constraints directly from training data. If you can generate synthetic conditioning signals from your dataset, train with them natively — it’s more robust and faster at inference.

3. Variable context is underexplored in streaming models. Most streaming methods default to fixed windows for simplicity. ARDY’s growing-context approach is a reminder that attention can handle variable lengths gracefully, and the memory benefits are substantial for long-horizon coherence. This pattern transfers to any sequential generation task (music, dialogue, procedural content).