Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewards

Paper: 2602.11128 Authors: Reinhard Heckel, Mahdi Soltanolkotabi, Christos Thramboulidis Categories: cs.LG

Abstract

This paper introduces asymmetric prompt weighting for reinforcement learning with verifiable rewards, challenging the conventional wisdom of focusing on prompts with intermediate success probabilities. The authors demonstrate that assigning higher weights to prompts with low or zero empirical success probability significantly accelerates training in from-scratch RL scenarios. Through theoretical analysis, they characterize optimal prompt weights that minimize convergence time under fixed update budgets, showing that in low-success regimes where informative responses are rare, asymmetric weighting becomes optimal by prioritizing difficult prompts.

Key Contributions

• Introduction of asymmetric prompt weighting that upweights low-success probability prompts, contrasting with existing methods like GRPO, DAPO, and RLOO
• Theoretical characterization of optimal prompt weights that minimize time to reach target accuracy under fixed update budgets
• Empirical demonstration that asymmetric weighting particularly benefits from-scratch RL (R1-Zero style) over post-SFT RL
• Analysis showing that in low-success regimes, where response cost dominates, asymmetric weights accelerate effective-time convergence

Methodology and Approach

The core innovation lies in rethinking how policy optimization algorithms weight gradients based on prompt difficulty. Traditional algorithms like GRPO (Group Relative Policy Optimization), DAPO, and RLOO focus on “ambiguous prompts” with intermediate success probabilities, effectively downweighting both very easy and very hard prompts. The rationale is that easy prompts provide little learning signal while hard prompts may be too noisy.

This paper challenges that assumption by proposing asymmetric weighting schemes that assign higher weights to prompts with low success rates. The key insight is that in early training stages or from-scratch scenarios, the model traverses a wide accuracy range where difficult prompts contain the most valuable learning signal. The authors formalize this through an optimization framework that considers:

• The fixed budget of model updates available
• The cost of generating responses for different prompts
• The information gain from successful responses on difficult prompts

Their theoretical analysis reveals that when informative responses are rare (low-success regime), the optimal weighting strategy becomes asymmetric, prioritizing prompts where the model currently struggles.

Experimental Results and Analysis

The empirical evaluation reveals a clear dichotomy between training scenarios:

From-Scratch RL (R1-Zero style): Asymmetric weighting shows substantial benefits. When training starts from a randomly initialized or minimally capable model, the wide accuracy range means the model encounters many low-success prompts. By upweighting these difficult cases, the training process more efficiently allocates compute to the most informative examples, accelerating convergence to target accuracy levels.

Post-SFT RL: The benefits are less pronounced. When starting from a supervised fine-tuned model that already achieves high accuracy, most prompts have relatively high success probabilities. The training regime doesn’t spend much time in the low-success zone where asymmetric weighting provides advantages.

This finding has important practical implications: the choice of prompt weighting strategy should depend on the initial model capability and training paradigm.

Theoretical Insights

The paper provides a rigorous theoretical framework for understanding prompt weighting. Under a fixed update budget $B$, the authors characterize weights $w(p)$ as a function of success probability $p$ that minimize expected time to reach target accuracy $p^*$ from initial accuracy $p_0$.

In the low-success regime where $p \ll 1$, the optimal weights satisfy:

$w(p) \propto \frac{1}{p \cdot c(p)}$

where $c(p)$ is the cost of generating responses for prompts with success probability $p$. This inverse relationship with success probability creates the asymmetric weighting pattern.

The key intuition: when successful responses are rare, each success on a difficult prompt provides disproportionate value. The cost of generating many failed attempts is justified by the high information content of eventual successes. This contrasts with high-success regimes where the marginal value of additional successes diminishes.

Implications for LLM Training

This work has several important implications for large language model post-training:

Training Paradigm Selection: The effectiveness of asymmetric weighting suggests that from-scratch RL approaches like R1-Zero may benefit from different optimization strategies than post-SFT RL. Training pipelines should adapt their prompt weighting based on initial model capability.

Compute Allocation: The theoretical framework provides principled guidance for allocating compute budget across prompts of varying difficulty, potentially improving training efficiency in resource-constrained settings.

Curriculum Learning Connections: Asymmetric weighting naturally implements a form of difficulty-based curriculum, but driven by empirical success rates rather than predefined difficulty metrics.

Reasoning Tasks: For reasoning tasks with verifiable rewards (mathematical problem-solving, code generation), where success is binary and informative, asymmetric weighting may be particularly valuable during early training phases.

Takeaways

1. Asymmetric prompt weighting that prioritizes low-success prompts accelerates from-scratch RL training by focusing compute on the most informative examples
2. The benefit of asymmetric weighting diminishes in post-SFT scenarios where models start at high accuracy, suggesting training strategy should adapt to initial capability
3. Theoretical analysis reveals that in low-success regimes, optimal weights are inversely proportional to success probability, justifying the asymmetric approach
4. The framework provides principled guidance for compute allocation in RL with verifiable rewards, particularly relevant for reasoning tasks
5. Training paradigm choice (from-scratch vs post-SFT) should inform prompt weighting strategy, with asymmetric weighting favored for wider accuracy traversal scenarios