RRC: Turning LLM Judges Into Usable RL Rewards by Ranking, Not Scoring

Paper: 2608.06310 Authors: Chenglong Wang, Ziming Zhu, Yifu Huo, Bei Li, Qiaozhi He, Yan Ding, Xiaoyang Hao, Yuxin Gao, Tianhua Zhou, Xiaojia Chang Categories: cs.LG, cs.CL

The Gap

The reward modeling world has been migrating from discriminative RMs (a Bradley-Terry head bolted onto a backbone, outputs one float) to generative RMs (an LLM that reasons in text and then judges). On RewardBench-style leaderboards, GenRMs win: they generalize better, they explain themselves, and you can scale them with test-time compute.

Then people plug them into RL and the gains largely evaporate. The standard workarounds are all attempts to squeeze a scalar out of a model that was never trained to produce one:

• Prompted scoring: “Rate this response 1-10.” The output is discretized, mode-collapsed onto 7/8/9, and badly calibrated across prompts.
• Token-probability / logit rewards: read P("Yes") or the logit gap for “is this good?” A continuous number, but it measures the judge’s linguistic confidence, not response quality, and drifts with prompt phrasing.
• Pairwise-vs-baseline: compare each sample against a single fixed reference and take the binary win. Cheap, but it’s a 1-bit reward — and once the policy beats the reference on everything, the signal dies.

The consequence bites hardest in group-relative RL (GRPO and friends), where the advantage is (R_i - mean) / std over the sampled group. If the judge hands back [8, 8, 8, 8], the advantage is exactly zero and the batch contributes nothing. The paper’s framing: this is not a GenRM capability problem, it’s an interface mismatch. GenRMs are comparators; RL algorithms consume scalars; everyone has been building the adapter badly.

[Problem]
 GenRM ranks well          but      RL consumes scalars
 (comparative by design)            (PPO / GRPO / RLOO)
        |                                   |
        +-----------------+-----------------+
                          v
                  [Observed symptom]
   "rate 1-10" / logit reward  ==>  coarse, tied, un-calibrated
                                    ==>  flat advantages, dead batches
                          |
                          v
                    [Assumption]
   the *ordering* a judge produces is the reliable part;
   the *number* it prints is a lossy guess at that ordering
                          |
                          v
                   [Method: RRC]
   +---------------------------+----------------------------+
   | self-competitive ranking  | anchor-guided ranking      |
   | order the G sampled       | order each sample against  |
   | responses among themselves| K stratified references    |
   +---------------------------+----------------------------+
                          |
                          v
                    [Evidence]
   open-ended chat + reasoning benchmarks;
   consistent gains over scalar / logit / single-baseline
                          |
                          v
                   [Conclusion]
   the usable RL signal from a GenRM is relative order.
   ask for comparisons, derive the scalar yourself.

The Increment

One sentence: Before, you asked the LLM judge for a score and inherited all of its calibration noise; after, you only ask it the one question it’s actually good at — “which of these two is better?” — and construct the scalar yourself from the resulting ranking.

Core Mechanism

RRC sits exactly where the reward function sits in a GRPO loop, and changes nothing else. For a prompt x, the policy samples G responses (you’re already doing this). RRC then converts those G responses into G scalars using only pairwise judgments.

Self-competitive ranking runs the group against itself. Every pair (y_i, y_j) goes to the GenRM, which returns a winner. Each response accumulates a win count w_i in [0, G-1], and the reward is the normalized win rate w_i / (G-1). This is a Copeland score / round-robin standing. Its virtues: zero extra data, guaranteed spread (win counts of a strict ordering are 0,1,...,G-1, so the group std is never zero), and robustness to judge miscalibration because only the sign of each comparison matters. Its vice: O(G^2) judge calls per prompt, which at G=8 and a 7B+ judge is a real bill. It’s also purely *within-group — a great response in a group of great responses gets the same reward as a mediocre one in a mediocre group, so there’s no cross-prompt notion of “good.”

Anchor-guided ranking fixes both. You keep a small set of K reference responses per prompt (or per prompt family), stratified by quality: a bad one, a mediocre one, a decent one, a strong one. Each sampled response is compared against the anchors and slotted into the ladder, giving a rank position in [0, K]; the reward is that position normalized. Cost drops to O(G**K) comparisons, and if you exploit the anchors’ known ordering with a binary-search insertion it’s O(G log K). More importantly, because the anchors are shared and stable, the resulting scalars are comparable across prompts and across training steps — the anchor ladder is an external ruler rather than a relative one. The two strategies are presented as complementary: self-competition gives fine-grained separation among near-identical samples, anchors give absolute grounding and scalability.

prompt x
   |
   v
policy pi -- sample G --> [ y_1  y_2  y_3 ... y_G ]
                                  |
         +------------------------+------------------------+
         |                                                 |
         v  (A) self-competitive                            v  (B) anchor-guided
   for all pairs (i,j):                        anchors  A = [ a_1 < a_2 < ... < a_K ]
     GenRM(x, y_i, y_j) -> winner              (fixed, quality-stratified)
         |                                                 |
         v                                                 v
   win count  w_i  in  [0, G-1]                 GenRM(x, y_i, a_k) -> win / lose
         |                                      insert y_i into the ladder
         |                                      rank_i  in  [0, K]
         v                                                 |
   r_i = w_i / (G-1)                                       v
   cost: O(G^2) judge calls                     r_i = rank_i / K
         |                                      cost: O(G*K), or O(G log K)
         +-------------------+-----------------------------+
                             v
                    reward   R_i = f(r_i)
                             |
                             v
              GRPO advantage  A_i = (R_i - mean) / std
                             |
                             v
                      policy gradient step

The metaphor: a chess club with a rating ladder. Your GenRM is a referee who can adjudicate a single game flawlessly but has no idea how to assign an Elo number — ask her “how strong is this player, 1 to 10?” and you get a shrug and “uh, 8,” for everyone. That’s prompted scalar scoring, and it’s why RL stalls: every player gets an 8, so nobody gets promoted or demoted.

Self-competitive ranking is the club round-robin: the eight players who showed up today all play each other, and you rank them by wins. The referee only ever answers questions she’s competent to answer, and you always get a clean 1st-through-8th standing. But a round-robin among eight is 28 games (O(G^2)), and “1st place today” tells you nothing about how today’s field compares to last week’s — the champion of a weak field and the champion of a strong field both go home with a trophy.

Anchor-guided ranking is the calibration ladder: the club keeps four house bots of known strength — beginner, club-level, expert, master. A new player plays down the ladder until they lose, and their rung is their rating. Four games instead of 28 (two if you binary-search: try the expert first), and the rung means the same thing this week as last week, because the bots don’t change. Which is also the failure mode you should worry about: once your policy routinely beats the master bot, everyone tops out at rung 4, ties return, and advantages flatten again. The bots need periodic upgrading — the paper’s anchor set has to be refreshed or re-stratified as the policy improves, or you re-derive the exact pathology RRC was built to avoid.

Key Concepts

• Calibration vs. discrimination: These are different skills and models are good at very different ratios of them. Discrimination is “can you tell A from B” — a judge who reliably says the better answer is better has high discrimination. Calibration is “does the number you emit mean the same thing every time” — a judge whose “7” always denotes the same quality level, across topics and prompt formats, is calibrated. LLM judges are strong discriminators and terrible calibrators: ask about a poem and a proof and the “8” means wildly different things. RRC’s whole design is a refusal to depend on calibration. It builds the calibration itself, out of an ordering, the way a thermometer is built by fixing two known points rather than by trusting the mercury’s self-report.

• Advantage collapse from tied rewards: In GRPO you don’t use the raw reward; you z-normalize it within the group of G samples for one prompt. This is elegant — no value network needed — but it has a cliff: if all G rewards are equal, the numerator is zero for every sample and the entire prompt contributes literally no gradient. A discretized 1-10 judge that outputs “8” four times out of four has silently deleted that training example. Concretely, with G=8 and a judge that only ever emits 7, 8, or 9, you’re squeezing eight samples into three buckets and burning most of your comparison information. A ranking, by construction, cannot tie everything: a strict order over G items always has nonzero variance.

• Ordinal vs. cardinal reward: RRC deliberately throws away magnitude. Rank 3-of-8 says “two responses beat you” but not “and they beat you by a mile.” That’s the trade: you gain robustness (immune to the judge’s scale drift) and lose the ability to express “this one is catastrophically bad.” For open-ended chat, where quality is genuinely a soft ordering, this is a great trade. For verifiable reasoning, where correctness is a hard binary, forcing an ordinal ranking onto a set of eight wrong answers manufactures a gradient toward “the most plausible-looking wrong answer” — a real risk the ranking framing doesn’t automatically avoid.

Framework Shift

Before (mainstream approach):          After (this paper):

    x, y                                   x, y_i  vs  y_j
      |                                    x, y_i  vs  a_k
      v                                          |
 +----------------+                              v
 | GenRM prompted |                     +------------------+
 | "rate 1 to 10" |                     | GenRM asked ONLY |
 +----------------+                     | "which is better"|
      |                                 +------------------+
      v                                          |
    "8"    <-- one token,                        v
      |         all the noise             ordering over the batch
      v                                          |
  R = 0.8                                        v
      |                                   R = position in order
      v                                          |
 [8, 8, 8, 8]  ==>  std = 0                      v
      |                                   [0, .33, .67, 1.0]
      v                                          |
  dead batch                                     v
                                          dense, tie-free signal

 the judge does the                       the judge does comparison,
 quantization (badly)                     RRC does the quantization

One sentence: From *asking the judge for a number to asking the judge for an order and deriving the number yourself, the core shift is relocating quantization out of the model’s prompt and into the reward-construction algorithm, where you control it.

Expert Assessment

Problem choice: A real gap, and a well-timed one. The GenRM-vs-DiscRM migration is genuinely happening, the “why doesn’t my LLM-judge RL run work” complaint is widespread, and the diagnosis — comparators forced through a scalar interface — is crisp and correct. It’s not a deep gap; it’s the kind of thing that becomes obvious the moment someone names it, which is a compliment about the framing more than the difficulty. Trajectory-wise this sits in the practical plumbing layer of RLHF, alongside advantage normalization tricks and reward-hacking mitigations: high adoption potential, modest conceptual depth.

Method maturity: Clever-cheap rather than clever-deep, and the two halves are not equally novel. Win-rate-as-reward is old news — it’s essentially Copeland scoring, and pairwise-preference-driven optimization has a whole lineage (Nash-MD, SPPO, self-play preference methods, and plain pairwise RMs in RLHF). The genuinely useful contribution is the stratified anchor set: it’s the piece that makes ranking rewards O(G**K) instead of O(G^2) and, more importantly, cross-prompt comparable. That’s the part I’d steal.

Two things I’d want interrogated harder. First, anchor staleness: a fixed ladder saturates as the policy improves, reintroducing ties. Any serious deployment needs a refresh schedule, and the choice of when/how to refresh is itself a hyperparameter with reward-hacking implications (a fixed anchor set is a fixed target to game). Second, positional bias: pairwise LLM judges famously prefer whichever response appears first, and win counts amplify a systematic bias rather than averaging it out. Order randomization or dual-order averaging is mandatory here, and doubles the cost. Simpler baselines that deserve to be in the table: just z-normalizing or rank-transforming the existing scalar rewards within the group, which fixes tie-induced collapse for free without any extra judge calls. If RRC only beats *un-normalized scalar rewards, the result is much less interesting.

Experimental integrity: I can’t verify the numbers — the abstract reports “consistent gains” without magnitudes, and I haven’t run the code. What I’d check first: (a) is compute matched? RRC spends many more judge calls per update, so the honest comparison is against a scalar-reward baseline given the same total inference budget, not the same number of policy steps; (b) is the reported win over a *rank-transformed scalar baseline, or only over raw prompted scores; (c) how do the anchor responses get built, and could they be leaking evaluation-adjacent quality signal; (d) reasoning-benchmark gains are the ones I’d trust least, since ordinal rewards over sets of wrong answers are conceptually shaky. Claiming both open-ended chat and reasoning wins from one reward-construction change is the kind of breadth that usually hides a narrow sweet spot.

Writing quality: Based on the abstract, the framing paragraph does its job well — the “comparative vs. scalar mismatch” sentence is the whole paper in one line, and that’s good writing. The section that would most elevate the work is a rigorous treatment of the anchor set: where anchors come from, how many you need, what stratification quality buys you, and what happens over long training runs as the policy outgrows them. That’s the load-bearing novelty and it’s the easiest thing to under-specify. A cost table (judge FLOPs per policy update, for each reward-construction method) would also close the most obvious reviewer objection before it’s raised.

Verdict: weak accept — the diagnosis is sharp and the anchor mechanism is a genuinely useful, immediately reusable trick, but the win-rate half is largely rediscovery and the paper lives or dies on anchor-staleness and compute-matched baselines that the abstract doesn’t promise to address.

Takeaways

• Never ask a model for a scalar when you can ask it for a comparison. This generalizes far beyond reward modeling: LLM-based eval, retrieval reranking, data filtering, curriculum ordering. Judges are near-useless calibrators and solid discriminators; design your pipeline so you only ever consume the part they’re good at, and do the quantization in code.
• The stratified anchor set is the portable trick. Keeping a handful of fixed, quality-graded reference items and locating new items relative to them turns any comparator into a cheap, cross-batch-comparable scorer at O(log K) calls with binary-search insertion. Use it for eval harnesses (a stable ruler that doesn’t drift as you change judge prompts), for data curation, for any “score this” problem where absolute scores are unstable but comparisons are reliable.
• Watch for advantage collapse as a first-class failure mode. If you run GRPO-style group normalization, instrument the fraction of prompts per batch where reward std is ~0. That single metric will tell you whether your reward function is discretizing away your gradient — and it’s a diagnostic worth logging regardless of whether you adopt RRC.
• When you replace a reward function, budget-match your baseline. Ranking rewards buy signal quality with judge inference. If your scalar baseline could have used the same FLOPs for a bigger judge or more samples, that’s the comparison that matters.
• Ordinal rewards are the right default for soft-quality tasks and a hazard for verifiable ones. If correctness is binary, don’t manufacture a ranking among wrong answers; gate ranking behind a correctness check, or you’re optimizing for plausible-looking failure.