Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards

Paper: 2610.18853
Authors: Yuante Li, Junwei Quan, Kai Qiu, Kate Zhang, Yicheng Tao
Categories: cs.AI, cs.LG

The Gap

Reinforcement learning with verifiable rewards (RLVR)—the post-training engine powering reasoning frontiers like DeepSeek-R1 and OpenAI o1—is bounded by a brutal economic constraint: sampling long chain-of-thought rollout tokens from large models is astronomically expensive. To amortize generation costs across cluster nodes, practitioners dramatically scale up the rollout batch size $B$, pairing it with the classical square-root learning-rate scaling rule:

$\eta \propto \sqrt{B}$

In idealized single-step rollouts, this rule appears to preserve per-sample optimization progress. However, in practical multi-step optimization runs (e.g., standard GRPO or PPO over multiple optimizer epochs), training curves exhibit an exasperating paradox:

1. The Progress Penalty: When batch size is scaled up, models require significantly more total environment samples to reach the same math or code reasoning benchmark threshold.
2. The Scheduled Collapse: Despite learning slower per sample, large-batch runs do not survive longer—they crash and experience policy collapse at nearly the exact same cumulative sample count as smaller batch runs.

Why do larger rollout batches simultaneously learn slower yet fail on schedule?

   THE MULTI-STEP RLVR SCALING PARADOX

   Rollout Batch B scaled by 16x (with lr * sqrt(16) = lr * 4):
   
   Expectation: Same per-sample progress; linear wall-clock speedup.
   
   Reality (The Two-Clock Divergence):
   +-----------------------------------------------------------+
   | Clock 1 (The Progress Clock - Optimization Geometry)      |
   | - Takes fewer, wider steps into high-curvature zones      |
   | - Second-order Taylor terms destroy progress efficiency   |
   | ===> Requires 2x MORE samples to hit target benchmark!   |
   +-----------------------------------------------------------+
   | Clock 2 (The Failure Clock - Policy Drift Dynamics)       |
   | - Square-root lr preserves cumulative first-order drift   |
   | - Cumulative policy drift hits the cliff at sample N*     |
   | ===> COLLAPSES ON THE EXACT SAME SCHEDULE!               |
   +-----------------------------------------------------------+
   Net Result: Policy crashes BEFORE reaching target accuracy!

The Increment

One sentence: By decomposing multi-step RLVR optimization into a curvature-penalized Progress Clock and a drift-invariant Failure Clock, the authors reveal why square-root scaling breaks down beyond a critical batch size and introduce a two-stage batch schedule that achieves both maximum sample efficiency and robust training stability.

Core Mechanism

The authors model the policy gradient landscape under multi-step off-policy updates using a local quadratic expansion:

1. The Progress Clock & Curvature Loss:
   • The first-order parameter drift per sample scales with $\eta / \sqrt{B} \approx \text{const}$.
   • However, the second-order curvature penalty scales with $\eta^2 \approx B$.
   • Because a batch size of $B$ takes $1/B$ as many optimizer steps, each individual step is substantially longer. The quadratic curvature penalty eats up the objective gain, establishing a learning-rate-dependent critical batch size ($B^*$).
   • Training with $B > B^*$ suffers from severe diminishing returns in sample efficiency.
2. The Failure Clock & Drift Invariance:
   • Policy collapse in RLVR (entropy collapse, reward hacking, or ratio explosion) is governed by cumulative parameter drift away from the reference checkpoint $\pi_0$.
   • Because the square-root rule holds expected first-order drift per sample approximately constant, the policy reaches the critical drift instability threshold at an invariant cumulative sample count, regardless of batch size.
3. Controlled Empirical Evidence on GRPO:
   • Validated on Qwen2.5-Math-1.5B and 7B across a $16\times$ batch sweep.
   • PPO/GRPO ratio clipping cushions intermediate steps and delays collapse, but completely fails to close the curvature-driven progress gap.
4. The Two-Stage RLVR Schedule:
   • To reconcile sample efficiency and stability, the authors derive a clean operational schedule:
     • Stage 1 (Rapid Ascent): Start with $B < B^*$ (small batch, conservative lr) where curvature loss is minimal, driving rapid early accuracy gains.
     • Stage 2 (Stability Transit): Switch to a larger batch size before approaching the critical drift threshold of the Failure Clock, stabilizing the policy through asymptotic convergence.

   THE DUAL DYNAMICS OF MULTI-STEP RLVR

   Optimization Space
           ^
           |     High-Curvature Valley
           |           \   /
           |            \_/  <-- Large Batch Step (Loses to curvature!)
           |             *
           |           /   \
           |          *     *  <-- Small Batch Steps (Follows curvature smoothly)
           |
           +---------------------------------------------> Sample Count
                                                |
                                                v
                                    [Failure Clock Deadline]
                               Policy Drift Hits Catastrophic Cliff

Key Concepts

• Progress Clock: The physical rate of metric improvement per consumed rollout sample, dictated by the interplay between first-order gradients and second-order Hessian curvature.
• Failure Clock: The invariant sample horizon at which cumulative parameter drift breaches the policy’s stable exploration basin, inducing entropy collapse.
• Critical Batch Size ($B^*$): The operational ceiling for rollout batch size beyond which second-order curvature losses outpace first-order gradient improvements.

Framework Shift

Before (Naive Square-Root Batch Scaling in RLVR):
  Assumption: eta = eta_0 * sqrt(B) maintains constant learning dynamics per sample.
  Observation: Massive GPU clusters hit policy collapse before reaching benchmark targets; sample efficiency degrades.
  Explanation: "Random training instability" or "insufficient clipping."

After (The Two-Clock Theory & Staged Schedule):
  Diagnosis: Square-root rule preserves drift (matching the Failure Clock) but penalizes curvature (lagging the Progress Clock).
  Prescription:
    1. Quantify critical batch B* for target architecture.
    2. Stage 1: Train below B* for maximum per-sample progress.
    3. Stage 2: Scale batch up before drift-induced failure horizon to cruise safely into convergence.

Expert Assessment

Problem choice: Hits the theoretical core of post-training reasoning models. Multi-step RLVR is notoriously finicky to tune, and huge engineering compute is wasted on unstable large-batch runs.

Method maturity: Formulates a clean analytical quadratic surrogate model that directly matches real empirical GRPO training curves across multiple model scales (1.5B to 7B).

Experimental integrity: Rigorous $16\times$ batch sweeps on mathematical reasoning, testing across multiple model families, general-purpose checkpoints, and varied clipping thresholds.

Writing quality: Highly lucid, with elegant physical analogies (“two independent clocks ticking at different rates”) that ground abstract optimization theory.

Verdict: strong accept — An essential scaling paper that transforms the black art of RLVR hyperparameter tuning into predictable engineering science.

Takeaways

• Do not assume square-root learning rate scaling maintains sample efficiency in multi-step RLVR; large batches bleed progress to loss curvature.
• PPO/GRPO ratio clipping can delay policy collapse, but it cannot fix the fundamental curvature penalty of oversized batches.
• Implement a two-stage batch schedule: explore and ascend with sub-critical batch sizes, then expand batch size to safely navigate late-stage convergence.