Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

Paper: 2603.23149 Authors: Massimiliano Pappa, Luca Romani, Valentino Sacco, Alessio Palma, Stéphane Lathuilière, Fabio Galasso, Xavier Alameda-Pineda, Indro Spinelli Categories: cs.AI

Abstract

Deploying safety-critical agents requires anticipating the consequences of actions before execution. While world models offer proactive foresight, current visual simulation approaches incur prohibitive latencies — often exceeding several seconds per step. DILLO (DIstiLLed Language-ActiOn World Model) challenges the assumption that visual processing is necessary for failure prevention. By showing that a policy’s latent state combined with planned actions already encodes sufficient information to anticipate outcomes, DILLO shifts the paradigm from “simulate-then-act” to “describe-then-act.” Trained via cross-modal distillation from a VLM teacher, DILLO achieves 14x speedup over baselines while improving episode success rate by up to 15 percentage points.

Key Contributions

• Latent Sufficiency Hypothesis: Demonstrates that a policy’s latent state is a sufficient statistic for predicting failure-critical outcomes, making visual simulation redundant
• Cross-modal distillation: VLM teacher annotates offline trajectories; LLM student learns to predict semantic outcomes from latent states alone
• 14x speedup: Text-only inference path bypasses heavy visual generation entirely
• Dual output: Natural language behavior preview + binary positive/negative verdict for autonomous rejection sampling
• Consumer hardware compatible: Runs on consumer GPUs unlike visual world models requiring enterprise hardware

The Foresight-Latency Dilemma

Current safety-critical agent approaches face a stark tradeoff:

Approach	Pros	Cons
Post-hoc analysis	Fast, cheap	Cannot prevent failures — only diagnoses after
Visual simulation	Can anticipate risks	~4 seconds per decision on RTX A6000 48GB
DILLO	Anticipates risks	Fast enough for real-time control

Visual world models generate high-dimensional future states (images/videos), but this is overkill for the binary question: “Will this action succeed or fail?”

The Latent Sufficiency Hypothesis

A policy’s internal latent representation is explicitly trained to retain task-critical features: object geometry, relative distances, contact dynamics. If this representation already contains the information needed to predict whether an action will succeed, why pay the cost of re-rendering pixels?

DILLO empirically validates this hypothesis: latent states + action plans → accurate outcome predictions, no pixels needed.

Architecture

Training (Offline)

1. VLM Teacher (privileged): Observes simulation environment, annotates trajectories with:
   • Natural language descriptions of what happens
   • Binary success/failure verdicts
2. LLM Student (DILLO): Learns to predict these annotations from:
   • Policy’s compact latent state (z_t)
   • Candidate action chunk (a_{t:t+k})

Inference (Online)

DILLO generates two outputs per candidate action:

1. Behavior preview (natural language): “The gripper goes left and forward, starting to approach the ball”
2. Binary verdict: Positive (advances task) or Negative (causes stagnation/failure)

Negative actions are rejected via rejection sampling; descriptions provide interpretability for human supervisors.

Results

Environment	Success Rate Improvement
MetaWorld	Up to +15 pp
LIBERO	+9.3 pp average
Speedup	14x over visual baselines

Takeaways

• Visual simulation is redundant for failure prevention — policy latent states contain sufficient information
• Cross-modal distillation (VLM → LLM) creates a fast text-only inference path from expensive visual annotations
• The “describe-then-act” paradigm provides both autonomous safety (rejection sampling) and human interpretability (natural language previews)
• Makes proactive agent steering feasible on consumer hardware, unlike visual world models requiring enterprise GPUs