RELEX: Extrapolating RLVR Training via Rank-1 Trajectories

Paper: 2605.21468 Authors: Zhepei Wei, Xinyu Zhu, Wei-Lin Chen, Chengsong Huang, Jiaxin Huang, Yu Meng Categories: cs.LG, cs.CL

The Gap

Reinforcement learning with verifiable rewards (RLVR) has become the standard for improving reasoning in LLMs—think of methods like GRPO or REINFORCE applied to math and coding tasks. The problem: RLVR is expensive. Training runs consume thousands of GPU hours, and practitioners have no choice but to run the full trajectory. Prior work treats RLVR as a black box optimization process where you need every step to reach the final performance.

But what if the weight trajectory isn’t random noise? What if there’s hidden structure we’re ignoring? This paper asks: can we predict where RLVR training will end up by observing only the beginning?

Problem: RLVR training is expensive (1000s of steps)
   |
   v
Observation: Weight trajectories might have low-rank structure
   |
   v
Hypothesis: Rank-1 subspace captures most performance gains
   |
   v
Method: Fit rank-1 direction + linear magnitude from short window
   |
   v
Evidence: 15% observation window -> full performance at 10-20x extrapolation
   |
   v
Conclusion: RLVR geometry is predictable; most training is redundant

The Increment

One sentence: Before this paper, you needed 1000 RLVR steps to get 1000-step performance; after this paper, you need 50 steps plus 30 seconds of linear algebra.

Core Mechanism

RELEX operates in three phases. First, run standard RLVR training for a short observation window (say, 50 steps out of 1000). Save checkpoints at regular intervals. Second, compute the weight deltas between consecutive checkpoints and perform SVD to extract the dominant direction—the rank-1 subspace that captures most variance. Third, fit a linear regression to predict how the magnitude of projection onto this direction grows with training steps. Now extrapolate: take the initial checkpoint, add the rank-1 direction scaled by the predicted magnitude at step 1000, and you have your extrapolated checkpoint.

Observation Window (steps 0-50):
  theta_0 -> theta_10 -> theta_20 -> ... -> theta_50
     |         |           |                    |
     +-------- compute deltas (delta_i) --------+
                          |
                          v
                    SVD on stacked deltas
                          |
                          v
                  Extract rank-1 direction (u)
                          |
                          v
            Project each delta onto u: alpha_i = delta_i · u
                          |
                          v
              Fit linear model: alpha = a*step + b
                          |
                          v
        Extrapolate to step 1000: theta_1000 = theta_0 + alpha_1000 * u

Think of RLVR training as a ship sailing across the ocean. Traditional thinking assumes you need to sail the entire route to reach the destination. RELEX observes that the ship is actually moving in a nearly straight line (rank-1 trajectory), and the speed is predictable (linear growth). So instead of sailing for 1000 nautical miles, you sail for 50 miles, measure your heading and speed, then teleport to where you would have been at mile 1000. The rank-1 direction is your compass bearing, the linear regression is your speedometer, and the extrapolation is the teleportation. The key insight: RLVR doesn’t zigzag—it commits to a direction early and maintains it.

Key Concepts

• Rank-1 trajectory: Imagine you’re tracking a parameter tensor with millions of dimensions. At each training step, the weights move in some direction in this high-dimensional space. A rank-1 trajectory means that despite the millions of possible directions, the weight updates are essentially moving along a single line. Mathematically, if you stack all the weight deltas into a matrix and perform SVD, the first singular value dominates—it captures 80-90% of the variance. Concretely, if your model has 1.5B parameters, the update at step 100 can be approximated as: delta_100 ≈ alpha_100 ** u, where u is a fixed 1.5B-dimensional vector (the rank-1 direction) and alpha_100 is a scalar (the magnitude). This is surprising because stochastic optimization typically explores many directions, but RLVR converges to a single dominant mode almost immediately.

• Denoising effect: RLVR training involves stochastic gradients—each batch introduces noise. If you naively extrapolate the raw weight trajectory, you’re amplifying this noise. RELEX’s rank-1 projection acts as a filter: it keeps the signal (the dominant direction of improvement) and discards the noise (random fluctuations orthogonal to this direction). Think of it like averaging out the jitter in a hand-drawn line to reveal the underlying straight path. The paper shows that without this projection, extrapolation fails—performance degrades beyond the observation window. With projection, extrapolation works even at 20x the observed steps.

Framework Shift

Before (standard RLVR):              After (RELEX):

Training loop (1000 steps):          Training loop (50 steps):
  for step in 1..1000:                 for step in 1..50:
    sample batch                         sample batch
    compute reward                       compute reward
    update weights                       update weights
    [expensive]                          [cheap]
         |                                    |
         v                                    v
  Final checkpoint                      Extract rank-1 subspace
  (full cost)                           Fit linear model
                                        Extrapolate to step 1000
                                        (negligible cost)
                                             |
                                             v
                                        Final checkpoint
                                        (same performance)

From exhaustive exploration to predictive geometry: the core shift is treating RLVR not as a stochastic search but as a deterministic trajectory in a low-dimensional subspace.

Expert Assessment

Problem choice: This is a real gap. RLVR training costs are a bottleneck for anyone working on reasoning models, and the field has largely accepted this as unavoidable. The observation that weight trajectories have exploitable structure is non-obvious and practically valuable. It sits at the intersection of optimization geometry and efficient training—a sweet spot.

Method maturity: RELEX is refreshingly simple. No neural networks, no hyperparameter tuning beyond choosing the observation window length. The rank-1 assumption is validated empirically across three models and multiple benchmarks. However, the paper doesn’t deeply explore failure modes—when does the rank-1 assumption break? What happens with different RLVR algorithms (PPO vs GRPO)? The ablation on rank-2 and rank-3 is brief; more analysis of when higher ranks are necessary would strengthen the claims.

Experimental integrity: Baselines are fair—they compare against full RLVR training, not strawmen. The results are consistent across models (1.5B to 8B parameters) and tasks (math, coding, general reasoning). The 10-20x extrapolation claim is backed by actual numbers. One concern: all experiments use Qwen models. Does this generalize to Llama, Mistral, or other architectures? The paper doesn’t test cross-architecture robustness.

Writing quality: The paper is well-structured and the core idea is communicated clearly. The weakness is in the related work section—it doesn’t sufficiently connect to the broader literature on low-rank adaptation (LoRA) and optimization trajectory analysis. The discussion of why RLVR produces rank-1 trajectories is speculative; a deeper theoretical analysis would elevate the contribution from empirical observation to principled understanding.

Verdict: weak accept — Solid empirical contribution with immediate practical value, but lacks theoretical depth and cross-architecture validation.

Takeaways

If you’re training models with any form of iterative optimization (not just RLVR), check if your weight trajectories are low-rank. The technique is simple: save checkpoints, compute deltas, run SVD, look at the singular value spectrum. If the first singular value dominates, you can likely extrapolate. This applies beyond LLMs—think fine-tuning vision models, continual learning, or even hyperparameter optimization trajectories.

The denoising insight is transferable: when extrapolating any noisy iterative process, projecting onto the dominant subspace before extrapolation can prevent noise amplification. This is a general principle for time-series prediction in high-dimensional spaces.

For practitioners: if you’re running expensive RLVR training, implement RELEX as a checkpoint. Run 10-15% of your planned steps, fit the rank-1 model, extrapolate, and validate. If it works, you’ve saved 85% of your compute. If it doesn’t, you’ve lost 15%—a favorable bet.