Relax: Asynchronous RL Training for Omni-Modal AI at Scale

Paper: 2604.11554 Authors: Liujie Zhang, Benzhe Ning, Rui Yang, Xiaoyan Yu, Jiaxing Li, Lumeng Wu, Jia Liu, Minghao Li, Weihang Chen, Weiqi Hu Categories: cs.CL

The Gap

RL post-training (RLHF, PPO, etc.) has unlocked reasoning in LLMs, but existing systems like veRL and OpenRLHF were built for text-only, single-turn scenarios. Now models handle images, audio, video, and multi-turn agentic workflows. Three walls appeared: (1) Heterogeneous data flows — text pipelines can’t handle variable-sized images or streaming audio without hacky retrofits. (2) Operational fragility — tightly coupled components mean one GPU failure kills the entire training run. (3) Staleness-throughput tradeoff — strict on-policy training is slow; going fully async risks divergence. Prior systems force you to pick speed or stability, not both.

Problem: Omni-modal RL at scale
    |
    v
Assumption: Service decoupling + async execution can maintain convergence
    |
    v
Method: Three-layer architecture (omni-native / fault-isolated / async queue)
    |
    v
Evidence: 2× speedup on 30B model, stable convergence across modalities
    |
    v
Conclusion: Async RL works if you design the right staleness control

The Increment

One sentence: Before Relax, you chose between slow on-policy training or risky async speedups; after Relax, a single staleness knob smoothly trades off speed and convergence while natively handling video, audio, and text in one system.

Core Mechanism

Relax splits RL training into independent services: Rollout (generates trajectories), Trainer (updates policy), Critic (scores actions), and Reference (provides KL penalty). Each service runs in its own process with isolated fault recovery. Between them sits TransferQueue, a data bus that decouples producers from consumers. When Rollout finishes a batch, it pushes to the queue and immediately starts the next batch — no waiting for Trainer to finish. Trainer pulls from the queue whenever ready, even if the data is a few steps stale.

The staleness parameter τ controls how old data can be. τ=0 is strict on-policy (wait for fresh data). τ=∞ is fully async (use whatever’s available). τ=2 means “use data up to 2 training steps old.” This turns the binary choice (sync vs async) into a continuous dial.

For multimodal data, Relax doesn’t bolt modalities onto a text pipeline. It treats each modality as a first-class citizen from data loading through inference. Images get their own preprocessing, audio streams through dedicated buffers, video frames are batched with temporal awareness. The parallelism strategy adapts per modality — text uses sequence parallelism, images use tensor parallelism, video uses a hybrid.

Rollout Service          TransferQueue           Trainer Service
[Generate batch 1] ---> [Queue: batch 1] ---> [Update policy w/ batch 1]
[Generate batch 2] ---> [Queue: batch 2]      [Still working on batch 1...]
[Generate batch 3] ---> [Queue: batch 3]      [Now pull batch 2, update]
     ^                        |                        |
     |                        v                        v
  (no wait)            (staleness buffer)        (async pull)

Think of Relax like a restaurant kitchen. Old RL systems (veRL) are like a single chef who must finish cooking one dish before starting the next — if the stove breaks, service stops. Relax is a brigade: one station preps ingredients (Rollout), another cooks (Trainer), another plates (Critic). Between stations are pass-through windows (TransferQueue). The prep station doesn’t wait for the cook to finish — it keeps prepping and slides dishes through the window. The cook grabs whatever’s ready, even if it sat for a minute. The staleness parameter is like “how long can a prepped dish sit before we toss it?” Fresh dishes (τ=0) mean slower service but perfect quality. Older dishes (τ=2) mean faster throughput with slight quality variance. The kitchen keeps running even if one station’s equipment fails — just swap in a backup.

Key Concepts

• Staleness in RL: In on-policy RL (like PPO), the policy must be updated with trajectories generated by the current policy. If you use old trajectories (generated by a past policy), that’s “stale data.” Traditional wisdom says staleness breaks convergence because the policy distribution shifts. Relax’s insight: small staleness (τ=1-2 steps) barely shifts the distribution, but massively improves throughput because services don’t block each other. It’s like training on yesterday’s data — not ideal, but close enough if “yesterday” was only 30 seconds ago in wall-clock time.

• Omni-native architecture: Most multimodal systems are text pipelines with image/audio bolted on. You tokenize images into patches, treat them like text, and hope for the best. This fails for video (too many tokens) and audio (streaming doesn’t fit batch processing). Omni-native means each modality has its own data path from disk to GPU, with modality-specific parallelism and memory management. Text gets sequence parallelism (split along sequence length), images get tensor parallelism (split the model), video gets pipeline parallelism (split across time). It’s not “make everything look like text” — it’s “give each modality what it needs.”

• Service-level fault isolation: In monolithic RL systems, all components (rollout, training, critic) run in one process. A single GPU OOM kills everything. Relax runs each component as a separate service with its own process and memory space. If Rollout crashes, Trainer keeps running on queued data while Rollout restarts. If a GPU dies, only that service’s replica fails — others continue. Recovery is local, not global. It’s the difference between a power outage shutting down your whole house vs. just one room’s circuit breaker tripping.

Framework Shift

Before (veRL, OpenRLHF):              After (Relax):

Rollout --> Trainer --> Critic        Rollout ----+
   |           |          |                        |
(tightly coupled, sync)                            v
   |           |          |                   TransferQueue
(one fails, all stop)                              |
                                                   v
                                              Trainer ----+
                                                          |
                                                          v
                                                      Critic
                                              (decoupled, async)
                                              (isolated failures)

From synchronous monolith to asynchronous microservices, the core shift is trading coordination overhead for independent scalability.

Expert Assessment

Problem choice: Real gap. As models go multimodal and agentic, existing RL infrastructure genuinely breaks. The paper targets a pain point that will only grow (video RL, long-horizon agents). Not manufactured — this is where the field is heading.

Method maturity: Solid engineering, not a research breakthrough. The async RL idea isn’t new (A3C did it in 2016), but applying it to modern RLHF with proper staleness control is valuable. The omni-native architecture is mostly plumbing — necessary plumbing, but not conceptually novel. The service decoupling is borrowed from distributed systems. What’s clever is the integration: making these three ideas work together at scale.

Experimental integrity: Baselines are fair (veRL is the current standard). The 2× speedup on 30B models is believable and well-documented. The convergence curves look clean — all methods reach similar final reward, which validates the staleness approach. One concern: experiments are mostly on Qwen models from the same institution. Would love to see results on Llama or other architectures. The R3 MoE comparison (1.9% overhead vs 32% in veRL) is striking but only shown on one setup.

Writing quality: The paper front-loads architecture diagrams before motivating why each piece exists. Section 3 (architecture) should come after Section 4 (challenges). The related work is thin — doesn’t engage with distributed RL literature (Impala, Seed RL) or explain why their approaches don’t transfer. The experimental section is strong but could use ablations on staleness values (only τ=0 and τ=2 are tested, what about τ=1, 3, 5?).

Verdict: Weak accept — this is a well-executed systems paper that solves a real problem, but the novelty is in integration rather than new ideas. The open-source release adds value. It’ll be cited by practitioners, less by theorists.

Takeaways

Staleness as a hyperparameter: Don’t treat sync vs async as a binary choice. Expose staleness as a tunable parameter and sweep it. Small staleness (1-2 steps) often gives 80% of async speedup with 95% of sync stability.

Service boundaries for fault tolerance: If your training pipeline has distinct stages (data generation, model update, evaluation), run them as separate processes with a queue between. One crash doesn’t kill the whole job. This applies beyond RL — any long-running ML training benefits.

Modality-specific parallelism: Stop forcing all modalities through the same pipeline. Text, images, and video have different memory/compute profiles. Design separate data paths and choose parallelism strategies per modality. The performance gains are non-trivial (2× in this paper).

TransferQueue pattern: A bounded queue between producer/consumer services is a simple but powerful decoupling mechanism. The producer never blocks (unless queue is full), the consumer pulls when ready. Add a staleness check (drop data older than τ steps) and you have async execution with bounded divergence. Steal this pattern for any multi-stage pipeline.