SARAH: Spatially Aware Real-time Agentic Humans

Paper: 2602.18432 Authors: Evonne Ng, Siwei Zhang, Zhang Chen, Michael Zollhoefer, Alexander Richard Categories: cs.CV

Abstract

SARAH introduces the first real-time, fully causal approach for generating spatially-aware conversational motion in embodied agents. Unlike existing methods that only align gestures with speech, SARAH enables agents to naturally orient toward users, respond to their movements, and maintain appropriate gaze patterns. The system processes user position and dyadic audio to produce full-body motion at over 300 FPS—3x faster than non-causal baselines—making it deployable on streaming VR headsets. The architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. A novel gaze scoring mechanism with classifier-free guidance allows users to adjust eye contact intensity at inference time while the model learns natural spatial alignment from data.

Key Contributions

• First real-time, fully causal method for spatially-aware conversational motion generation deployable on VR headsets
• Causal transformer-based VAE architecture with interleaved latent tokens enabling streaming inference at 300+ FPS
• Flow matching model conditioned on both user trajectory and audio for spatially-grounded full-body motion
• Gaze scoring mechanism with classifier-free guidance that decouples learning from control, allowing runtime adjustment of eye contact intensity
• State-of-the-art motion quality on Embody 3D dataset while being 3x faster than non-causal baselines
• Validation on live VR system demonstrating practical deployment of spatially-aware conversational agents

Technical Architecture

The SARAH architecture addresses the fundamental challenge of generating spatially-aware motion in real-time through a two-stage approach. The first stage employs a causal transformer-based Variational Autoencoder (VAE) that processes motion sequences with interleaved latent tokens. This design choice is critical for streaming inference: by interleaving latent representations throughout the sequence rather than using a single global latent, the model can generate motion causally without requiring future context.

The second stage uses a flow matching model that takes the VAE’s latent space and conditions it on both audio features and user trajectory. Flow matching provides a more stable training objective compared to traditional diffusion models while maintaining high-quality generation. The conditioning on user trajectory $\mathbf{u}_t$ ensures that the agent’s motion remains spatially grounded—the agent naturally turns toward the user, adjusts body orientation based on relative position, and maintains appropriate interpersonal distance.

The gaze mechanism deserves special attention. Rather than directly predicting gaze from the model, SARAH introduces a scoring function that evaluates how well generated motion aligns with desired gaze patterns. This score is then used with classifier-free guidance during inference: $\mathbf{m}_t = \mathbf{m}_t^{\text{uncond}} + w \cdot (\mathbf{m}_t^{\text{cond}} - \mathbf{m}_t^{\text{uncond}})$, where $w$ controls gaze intensity. This decoupling means the model learns natural spatial behavior from data, but users can dial eye contact up or down based on cultural preferences or application requirements.

Performance and Results

SARAH achieves remarkable performance metrics that make it practical for real-world deployment. The system runs at over 300 FPS on standard VR hardware, which is 3x faster than comparable non-causal baselines. This speed advantage comes from the causal architecture—by never requiring future frames, the model eliminates the latency inherent in bidirectional approaches.

On the Embody 3D dataset, SARAH achieves state-of-the-art motion quality across multiple metrics. The generated motions exhibit natural spatial dynamics: agents smoothly rotate to face users as they move, maintain appropriate gaze patterns during conversation, and produce gestures that are both speech-aligned and spatially coherent. The flow matching formulation contributes to motion smoothness, avoiding the jittery artifacts common in autoregressive approaches.

The gaze scoring mechanism proves particularly effective. By learning from natural conversation data, the model captures subtle patterns like looking away during thought, returning gaze during emphasis, and adjusting eye contact based on interpersonal distance. The classifier-free guidance allows these patterns to be amplified or dampened at inference time without retraining, providing flexibility for different cultural contexts or application scenarios.

Implications for Embodied AI

SARAH represents a significant step toward truly interactive embodied agents. Previous work in conversational motion generation focused primarily on speech-driven gestures, treating the agent as essentially stationary. This limitation made agents feel disconnected from their spatial environment—they would gesture naturally but fail to acknowledge a user walking around them or maintain appropriate social distance.

By incorporating spatial awareness as a first-class concern, SARAH enables more natural human-agent interaction. In VR telepresence, this means remote participants can be represented by agents that respond naturally to others’ movements. In digital human applications, virtual assistants can maintain appropriate social protocols around personal space and eye contact. The real-time performance makes these applications practical on consumer hardware rather than requiring server-side processing.

The decoupled gaze control mechanism has broader implications for controllable generation. Rather than baking control parameters into the model architecture, SARAH demonstrates that scoring functions with classifier-free guidance can provide runtime control while preserving the model’s learned natural behavior. This pattern could extend to other aspects of motion control—energy level, formality, or emotional expression—without requiring separate models for each variation.

Takeaways

1. SARAH achieves the first real-time, fully causal spatially-aware conversational motion generation, running at 300+ FPS on VR hardware—3x faster than non-causal methods.

2. The causal transformer-based VAE with interleaved latent tokens enables streaming inference without future context, critical for low-latency interactive applications.

3. Flow matching conditioned on user trajectory and audio produces spatially-grounded full-body motion that naturally orients toward users and responds to their movement.

4. The gaze scoring mechanism with classifier-free guidance decouples learning from control, allowing users to adjust eye contact intensity at inference time while the model learns natural patterns from data.

5. Validation on live VR systems demonstrates practical deployment, bringing spatially-aware conversational agents from research to real-world applications in telepresence and digital humans.