When AI Judges Disagree With Themselves: Diagnosing LLM Evaluation Reliability

Paper: 2604.15302 Authors: Manan Gupta, Dhruv Kumar Categories: cs.AI, cs.CL, cs.LG

The Gap

LLM-as-judge frameworks are now standard for evaluating text generation—cheaper and faster than human annotation. Existing work validates these judges by checking aggregate metrics: does the average score correlate with human judgment? But this hides a critical question: can we trust the judge’s verdict on this specific document?

Prior approaches assume that low aggregate error means reliable per-instance judgments. They report overall accuracy or correlation, then deploy the judge everywhere. The gap: aggregate reliability doesn’t guarantee instance-level reliability. A judge might be right on average but wildly inconsistent on individual cases—like a doctor with 95% accuracy who randomly flips diagnoses for the same patient.

Problem: LLM judges used without per-instance reliability checks
    |
    v
Assumption: Low aggregate error → reliable individual judgments
    |
    v
Method: (1) Transitivity analysis (catch logical contradictions)
        (2) Conformal prediction sets (quantify uncertainty per doc)
    |
    v
Evidence: 33-67% of docs have transitivity violations
          Set width correlates with human disagreement (r=0.576)
    |
    v
Conclusion: Aggregate metrics mask document-level unreliability
            Prediction set width = practical uncertainty indicator

The Increment

One sentence: Before this paper, we knew LLM judges worked on average; after, we can identify which specific documents they’re confused about.

Core Mechanism

The paper introduces two complementary diagnostics. First, transitivity analysis: if a judge says summary A > B and B > C, it should say A > C. Violations reveal logical inconsistency. The authors count directed 3-cycles (A > B > C > A) across all summary triplets for each document.

Second, split conformal prediction: instead of outputting a single score (e.g., “relevance = 4”), output a *set of plausible scores (e.g., $\{3, 4, 5\}$). The set is constructed by calibrating on held-out data to guarantee coverage—if the true score is 4, it must appear in the set at least (1-α)% of the time. Narrow sets mean high confidence; wide sets mean uncertainty.

Input: Document + Summary
    |
    +---> Transitivity Check:
    |         Judge compares all summary pairs
    |         Count cycles: A>B>C>A
    |         Output: violation rate per doc
    |
    +---> Conformal Prediction:
              Judge scores summary (1-5 Likert)
              Calibrate threshold on validation set
              Output: prediction set {s1, s2, ...}
              Width = |set| = uncertainty measure

Think of the judge as a wine critic rating bottles. Transitivity analysis is like catching them saying “Merlot A beats B, B beats C, but C beats A”—logically impossible if they have consistent taste. Conformal prediction is like asking “how confident are you?” Instead of saying “this wine is 4 stars,” they say “I’m sure it’s between 3 and 5 stars.” A narrow range (3-4) means they’re confident; a wide range (1-5) means they’re guessing. The width of their confidence interval tells you when to trust their rating.

Key Concepts

• Transitivity violations: If you prefer apples to oranges and oranges to bananas, you should prefer apples to bananas. A violation means your preferences form a loop—you’d trade apples for oranges, oranges for bananas, bananas back to apples, ending up where you started but somehow “better off.” In ranking summaries, this reveals the judge doesn’t have a coherent internal quality scale. The paper finds 33-67% of documents trigger at least one such loop, even though the aggregate violation rate looks low (0.8-4.1%) because most triplets are consistent.

• Conformal prediction sets: Traditional prediction gives a point estimate (“this summary scores 4”). Conformal prediction gives a set with a guarantee: “the true score is in this set with 90% probability” (if α=0.1). You build it by ranking calibration examples by their nonconformity score (how weird they are), then including all labels up to the (1-α) quantile. The set width becomes a calibrated uncertainty measure—wide sets mean the model is confused, narrow sets mean it’s confident. Crucially, this works for *any black-box model without retraining.

• Cross-judge agreement on difficulty: The paper shows that when one judge outputs a wide prediction set for a document, other judges also output wide sets (correlation r=0.32-0.38). This means set width captures something real about the document—it’s genuinely hard to evaluate—rather than random noise from a specific judge. It’s like multiple teachers independently finding the same essay difficult to grade; the difficulty is in the essay, not the teachers.

Framework Shift

Before (mainstream approach):        After (this paper):

  LLM Judge                            LLM Judge
      |                                    |
      v                                    +---> Transitivity Check
  Single Score                             |         (logical consistency)
      |                                    |
      v                                    +---> Conformal Set
  Aggregate Metrics                        |         (per-doc uncertainty)
  (correlation, accuracy)                  |
      |                                    v
      v                                Per-Instance Diagnostics
  Deploy Everywhere                    (know when to trust)

From “trust the average” to “diagnose each case,” the core shift is treating reliability as a per-instance property rather than a system-level statistic.

Expert Assessment

Problem choice: Real gap. The field has been deploying LLM judges at scale based on aggregate validation, which is methodologically lazy. This paper asks the obvious-in-hindsight question: “but is it reliable *this time?” The timing is right—LLM-as-judge is mature enough that people are hitting edge cases in production.

Method maturity: Transitivity analysis is straightforward but effective—it’s just counting cycles in a preference graph. Conformal prediction is borrowed from the uncertainty quantification literature, not invented here, but the application is novel. The combination is clever: one diagnostic catches logical failures, the other quantifies uncertainty. No simpler approach would give both.

Experimental integrity: Baselines are fair—they compare four different LLM judges (GPT-4, Claude, etc.) across four criteria. The SummEval dataset is standard. The correlation between set width and human disagreement (r=0.576) is strong and well-validated. One weakness: they don’t test whether filtering out high-uncertainty instances actually improves downstream task performance. The paper diagnoses but doesn’t prescribe.

Writing quality: The abstract is dense—too many numbers upfront. Section 3 (method) is clear, but Section 4 (results) buries the lead: the cross-judge agreement finding (which validates that set width measures real difficulty) appears late. Rewriting the results section to lead with “set width is not noise, here’s proof” would strengthen the narrative.

Verdict: weak accept — Solid diagnostic contribution with immediate practical value, but stops short of showing how to act on the diagnostics.

Takeaways

Practitioners can steal two things immediately:

1. Conformal prediction sets as uncertainty flags: Wrap any LLM judge in split conformal prediction (implementation is ~50 lines). Use set width to route: narrow sets → auto-accept, wide sets → human review. This is cheaper than reviewing everything and more reliable than trusting all outputs.

2. Transitivity as a sanity check: Before deploying a judge, run it on a small sample and count 3-cycles. If >50% of documents have violations, your judge is incoherent—don’t trust it even if aggregate metrics look good. This catches failures that correlation scores miss.

The broader lesson: aggregate metrics are necessary but insufficient. Always ask “can I trust this specific prediction?” and build diagnostics that answer per-instance.