User Feedback Is a Signal LLM Judges Cannot See

Paper: 2609.02859 Authors: Shachar Don-Yehiya, Leshem Choshen, Omri Abend Categories: cs.CL

The Gap

Naturally occurring user feedback is an attractive learning signal — it is abundant, free, and grounded in real use. Recent work concluded it is inherently noisy and difficult to leverage, and the field largely moved on.

The paper argues that conclusion is wrong, and that it was reached through a systematic bias in the evaluation paradigm rather than through a property of the feedback. That is a strong claim about how the field evaluates rather than about what the data contains, and it predicts something specific: the feedback is informative, but the measurement keeps registering it as uninformative.

   USER FEEDBACK: ATTRACTIVE, THEN ABANDONED

   attractive signal: ABUNDANT, FREE, GROUNDED IN REAL USE
        |
        v
   recent work: "inherently noisy and difficult to leverage"
        |
        v
   the field largely moved on
        |
        v
   [THIS PAPER'S OBJECTION]
     that conclusion is WRONG, and it was reached through a
     SYSTEMATIC BIAS IN THE EVALUATION PARADIGM, not through
     a property of the feedback
        |
        v
   [PREDICTION] the feedback IS informative; the MEASUREMENT
                keeps registering it as uninformative

The Increment

One sentence: Before this paper, user feedback was judged too noisy to use; after it, revisions made with feedback resolve targeted issues at significantly higher rates, and the reason the field misjudged this is that LLM judges systematically prefer the uncorrected baseline.

Core Mechanism

The design isolates the usefulness of feedback by controlling the ground truth.

Synthetic data with a definitive ground truth provides the controlled measurement: when you know what the right revision is, you can ask whether feedback steers toward it. Naturalistic data then checks that the finding survives real-world scenarios — the pairing is the right structure, since synthetic-only would leave open that the effect is an artefact of controlled inputs.

The comparison is straightforward and well posed: compare model revisions generated with and without access to feedback across both settings. The result is that feedback-informed revisions resolve targeted issues at significantly higher rates than baseline revisions. So the signal is actionable — the earlier conclusion was not.

Then the diagnosis, which is the paper’s real contribution. When a model successfully fixes an issue exclusively due to feedback, LLM judges frequently fail to identify the genuinely corrected response, systematically preferring inferior baseline outputs instead.

Unpack that sentence, because each clause carries weight. The failure is conditional: it appears when the fix is attributable to feedback — which is exactly the class of cases that any evaluation of feedback must be sensitive to. The judges fail to identify the corrected response, so the improvement is present but unrecognised. And the preference is systematic and in the wrong direction — not random error, but a bias toward the worse output. An evaluation built on such judges will report that feedback does not help, because the cases where it did help are precisely the cases the judge marks down.

That is a mechanism for a published negative result, and it is a specific and checkable one. It also has an uncomfortable implication beyond this paper: any conclusion resting on LLM-judge comparison inherits a bias whose direction depends on what the intervention does, and if the intervention improves outputs in ways the judge systematically disfavours, the evaluation will invert the sign of the finding.

   DESIGN: ISOLATE FEEDBACK'S USEFULNESS

   SYNTHETIC data with a DEFINITIVE GROUND TRUTH
     -> controlled: you know the right revision
   NATURALISTIC data
     -> checks the finding survives real-world scenarios
        (synthetic-only leaves open an artefact explanation)
        |
        v
   COMPARISON: revisions generated WITH vs WITHOUT feedback
        |
        v
   RESULT: feedback-informed revisions resolve targeted
           issues at SIGNIFICANTLY HIGHER rates
           -> the signal is ACTIONABLE
           -> the earlier conclusion was not
        |
        v
   DIAGNOSIS: WHY THE FIELD MISJUDGED IT
     when a fix is due SOLELY to feedback, LLM judges
     FAIL TO IDENTIFY the genuinely corrected response
     and SYSTEMATICALLY PREFER inferior baseline outputs
        |
        +-- the failure is CONDITIONAL: it appears exactly
        |   when the fix is attributable to feedback --
        |   the class any feedback evaluation must detect
        +-- the improvement is PRESENT but UNRECOGNISED
        +-- the bias is SYSTEMATIC and in the WRONG DIRECTION,
            not random error
        |
        v
   -> an evaluation built on such judges reports that
      feedback does not help, because the cases where it
      DID help are the cases the judge marks down

Think of it as a referee who cannot see the goal that came from the coaching. A team works with a coach on a specific weakness and, in the next match, scores by executing exactly the taught movement. The referee — who was not watching the training — sees an unusual play that does not match the team’s usual style and rules it out. The coaching worked; the measurement scored it as a mistake. And notice the failure is not random: it appears specifically for the goals that came from coaching, so the better the coaching is at changing behaviour, the more goals it gets disallowed.

Key Concepts

• Evaluation-paradigm bias as the explanation: the negative reputation of user feedback was produced by the measurement, not by the data. It relocates the problem from data quality to evaluation design, which changes what one would do about it.
• Conditional judge failure: the judges fail specifically when a fix is due to feedback. Because the failure coincides with the intervention’s effect, no amount of averaging over cases removes it — the bias aligns with the signal.
• Systematic rather than random error: the judges prefer the inferior baseline. Random judge noise would attenuate a measured effect; a directed bias can invert its sign, which is what happened here.
• Paired synthetic and naturalistic evaluation: ground truth for control, real data for validity. The pairing is what lets the paper claim the effect is real rather than an artefact of controlled inputs.

Framework Shift

Before (judge comparison as the measure):
  generate revisions with and without feedback
  -> score both with an LLM judge
  -> judges prefer the baseline when the fix came from feedback
  -> conclusion: "user feedback is too noisy to leverage"

After (isolating feedback from the judge):
  synthetic data with definitive ground truth
  + naturalistic data for validity
  -> feedback-informed revisions resolve targeted issues
     at significantly higher rates
  -> the judge's failure is CONDITIONAL and DIRECTIONAL:
     it rejects exactly the feedback-attributable fixes

From concluding that a signal is too noisy, to discovering that the measuring instrument is blind precisely where the signal acts, the core shift is that a judge-based evaluation can invert the sign of a finding when the intervention improves outputs in ways the judge disfavours.

Expert Assessment

Problem choice: Excellent, and unusually valuable for a paper whose headline is positive. The contribution is not only “feedback works” but a demonstration that a published negative conclusion about a widely available signal was an artefact of the tooling — which is the kind of result that should prompt re-examination of other judge-based negatives.

Method maturity: The design is disciplined: synthetic data for ground truth, naturalistic data for validity, and a with-versus-without comparison that isolates the variable. The diagnosis is the sophisticated part: identifying that the judge failure is conditional on the intervention’s effect is what makes the mechanism specific, and it explains why the field’s measurements were consistently wrong rather than randomly noisy.

Experimental integrity: The paper does the harder thing by looking for the failure’s mechanism rather than only reporting a better result, and locating the bias in the judge is a falsifiable claim about a specific comparison. The main caveat is the scope of the naturalistic validation: synthetic ground truth gives crisp measurement, and real-world feedback is messier, so the strength of the finding in uncurated production settings is suggested rather than fully established. The paper is also, in effect, reporting a defect in a tool the field depends on, which makes its reproducibility particularly important.

Writing quality: The abstract’s structure is exemplary: reputation, challenge, controlled comparison, result, and then the diagnosis of why the field misjudged it. Because the practical implication is a warning about judge-based evaluation, a short passage on when to be suspicious of a judge comparison — specifically, when the intervention could improve quality along a dimension the judge does not reward — would generalise the lesson beyond feedback.

Verdict: strong accept — it reopens a signal the field had set aside, and explains the misjudgement with a specific, directional bias in the measurement instruments the field relies on.

Takeaways

• Check whether a judge can see your intervention. If the intervention improves outputs along a dimension the judge does not reward, the evaluation can invert the sign of the finding.
• Prefer a directional bias over noise as the explanation for a null result. Random judge error attenuates effects; a directed bias can hide them entirely, and only the conditional analysis reveals which you have.
• Pair synthetic ground truth with naturalistic validation. Definitive labels give control; real data rules out the artefact explanation.
• Re-examine judge-based negatives before accepting them. A conclusion that a readily available signal is unusable is worth testing against the possibility that the measurement, not the signal, failed.