When Wrong Rewards Help: Why Imperfect Feedback Can Accelerate RL Training

Paper: 2604.25872 Authors: Shuning Shang, Hubert Strauss, Stanley Wei, Sanjeev Arora, Noam Razin Categories: cs.LG, cs.AI, stat.ML

The Gap

RLHF for language models relies on proxy rewards (like reward models trained on human preferences) because true rewards are unavailable. The field evaluates these proxies using ranking accuracy: how often does the proxy rank outputs in the same order as ground truth? This treats all reward errors equally — any deviation from ground truth is considered harmful.

But this is wrong. When you run policy gradient, not all outputs matter equally. The algorithm increases probability mass on high-reward outputs. If your proxy reward is wrong about outputs the policy rarely visits, who cares? If it’s wrong about outputs the policy is about to exploit, that’s a problem. Standard metrics ignore this interaction between reward errors and policy dynamics.

Problem: Proxy rewards have errors
         |
         v
Current approach: Measure ranking accuracy
         |                (treats all errors equally)
         v
Gap: Errors matter differently depending on
     which outputs the policy is moving toward
         |
         v
This paper: Categorize errors by their effect
            on policy gradient dynamics
         |
         v
Evidence: Theory + experiments show some errors
          are benign/beneficial
         |
         v
Conclusion: New metrics that weight errors by
            their impact on learning

The Increment

One sentence: Before this paper, reward errors were uniformly bad; after, we distinguish harmful errors (on outputs the policy exploits) from benign/beneficial ones (on outputs the policy avoids or escapes).

Core Mechanism

The method starts by analyzing what policy gradient actually does. During training, the algorithm samples outputs from the current policy, evaluates them with the proxy reward, and increases probability on high-reward outputs. The key insight: only outputs that (1) have non-negligible probability under the current policy AND (2) receive high proxy reward will attract probability mass.

The authors categorize reward errors into three types based on their effect on ground truth reward improvement. Harmful errors occur when the proxy assigns high reward to outputs with low ground truth reward that the policy would otherwise avoid — these pull the policy toward bad outputs. Benign errors happen on outputs the policy already avoids or will avoid regardless — these don’t change the policy’s trajectory. Beneficial errors occur when the proxy underrates mediocre outputs, preventing the policy from stalling on local optima and pushing it to explore further.

Policy gradient flow:

Current policy π_t
    |
    | sample outputs
    v
Outputs {y1, y2, ...}
    |
    | evaluate with proxy reward r^
    v
Weighted by: π_t(y) * r^(y)
    |
    | gradient step
    v
Updated policy π_{t+1}
    |
    v
Probability shifts toward high-r^ outputs

Error impact depends on:
- Is π_t(y) significant? (Does policy visit this output?)
- Is r^(y) high? (Does proxy encourage this output?)
- What's the true reward R(y)?

Think of policy gradient as water flowing downhill on a landscape shaped by the proxy reward. The current policy is a probability distribution — a cloud of water droplets spread across the landscape. Policy gradient pushes water toward valleys (high proxy reward regions). Now imagine the proxy reward landscape is a distorted version of the true reward landscape.

If there’s a fake valley (high proxy reward, low true reward) where no water currently sits, it doesn’t matter — the water won’t flow there from its current position. That’s a benign error. If there’s a fake valley where water is already pooling, that’s harmful — the water gets trapped in the wrong place. But if there’s a fake hill (low proxy reward) where water is pooling at a mediocre true reward, that’s beneficial — it forces the water to keep flowing instead of settling for “good enough.” The landscape’s distortions only matter where the water actually flows.

Key Concepts

• Probability mass concentration: Policy gradient doesn’t uniformly improve all outputs. It concentrates probability on outputs that are both (1) reachable from the current policy (non-negligible π(y)) and (2) encouraged by the reward (high r(y)). This creates a “selection effect” — only errors on outputs satisfying both conditions affect learning. An output with 0.0001% probability won’t attract mass even if the reward is wrong about it, because the policy rarely samples it. An output with 10% probability but low reward won’t attract mass either, because the gradient pushes away from it. The danger zone is outputs with moderate probability AND high proxy reward but low true reward.

• Beneficial errors as escape mechanisms: Standard RL theory assumes more accurate rewards are always better. But when the true reward landscape has plateaus (many outputs with similar mediocre reward), an accurate proxy can trap the policy. If the proxy slightly underrates these mediocre outputs, it creates artificial gradient that pushes the policy to keep exploring. This is beneficial because the policy escapes local optima it would otherwise settle into. The key condition: the underrated outputs must have mediocre true reward, and there must exist better outputs the policy can reach. If you underrate the globally optimal output, that’s just harmful.

• Ranking accuracy vs. learning-weighted accuracy: Ranking accuracy asks: “Do you rank output A above output B correctly?” It treats all pairs equally. But policy gradient cares more about pairs where both outputs have significant probability under the current policy. Learning-weighted accuracy weights each pair by how much it affects the policy update. A pair (A, B) matters if the policy might choose between them during training. If both have negligible probability, getting their ranking wrong is harmless. This explains why reward models with similar ranking accuracy can produce very different RLHF outcomes — they differ on the pairs that matter for learning.

Framework Shift

Before (standard view):              After (this paper):

Reward error = bad                   Reward error = context-dependent
                                     
Ground truth reward R                Ground truth reward R
Proxy reward r^                      Proxy reward r^
                                     
Evaluate: |R - r^|                   Evaluate: Effect on policy
(absolute deviation)                 gradient trajectory
                                     
All errors weighted equally          Errors weighted by:
                                     - Current policy π(y)
                                     - Proxy reward r^(y)
                                     - True reward R(y)
                                     
Goal: Minimize ranking errors        Goal: Minimize harmful errors,
                                     tolerate/exploit benign/beneficial

[One sentence: From treating all reward errors as uniformly harmful to recognizing that error impact depends on the policy’s current state and learning dynamics.]

Expert Assessment

Problem choice: This is a real gap. RLHF practitioners have noticed that reward model accuracy doesn’t predict downstream performance well, but the field lacked a principled explanation. The problem sits at a productive intersection — it’s theoretically interesting (challenges standard RL assumptions) and practically relevant (RLHF is everywhere). The framing as “errors can be beneficial” is slightly provocative but earned.

Method maturity: The theoretical analysis is solid but not groundbreaking — it’s a careful application of policy gradient theory to the proxy reward setting. The categorization (harmful/benign/beneficial) is the main conceptual contribution, and it’s genuinely useful. The proposed metrics (learning-weighted accuracy) are straightforward once you accept the framework. No major technical innovations, but the insight is valuable. One weakness: the beneficial error story relies on the policy being able to escape mediocre regions, which assumes sufficient exploration. In practice, policies often get stuck anyway.

Experimental integrity: The experiments are honest but limited. The RLHF experiments (Section 5.1) show the new metrics correlate better with downstream performance than ranking accuracy, but the improvement is modest and dataset-dependent. The synthetic experiments (Section 5.2) cleanly demonstrate the theory but use simplified settings. The paper acknowledges that “gaps remain in robustly evaluating reward models” — this is refreshingly honest. Baselines are fair. The main concern: the experiments don’t test whether practitioners can actually use these insights to design better reward functions, only that the theory explains observed phenomena.

Writing quality: The paper is well-structured and clearly written. The theory section (Section 3) is dense but necessary. The practical implications (Section 5) feel somewhat disconnected from the theory — the metrics are proposed but not deeply validated. The related work section is thorough. If I were revising, I’d expand Section 5.2 to show concrete reward design strategies that exploit beneficial errors, not just demonstrate they exist. The current version proves the concept but leaves practitioners without clear guidance on “now what?”

Verdict: weak accept — Solid theoretical contribution that reframes how we think about proxy rewards, with honest experimental validation showing modest practical gains. The insight is valuable even if the immediate applications are limited.

Takeaways

For RLHF practitioners: When evaluating reward models, don’t just look at overall ranking accuracy. Compute accuracy on the subset of output pairs your policy is likely to encounter (high probability under current policy). This is more predictive of downstream performance. If you have access to the policy during reward model training, you can weight training data by policy probability to focus the reward model on decision-relevant regions.

For reward design: If your policy is stuck on mediocre outputs, consider intentionally underrating them in your reward function to force exploration. This is risky (you might push the policy toward worse outputs), but in domains with verifiable rewards where you can detect and correct mistakes, it’s a viable strategy. The key condition: you need confidence that better outputs exist and are reachable.

For RL theory: The interaction between reward function and initial policy matters more than we typically acknowledge. Two reward functions with the same ranking accuracy can produce wildly different learning trajectories depending on where the policy starts. This suggests we need policy-conditional notions of reward quality, not just static metrics.

Transferable technique: The “learning-weighted evaluation” idea applies beyond rewards. Anytime you’re evaluating a proxy signal (value function, advantage estimate, uncertainty estimate), weight your evaluation by how much that signal affects the learning algorithm’s decisions. Errors on rarely-visited states don’t matter; errors on decision boundaries do.