The Judge That Knows When to Google — And When to Shut Up

Paper: 2608.17994 Authors: Sher Badshah, Ali Emami, Hassan Sajjad Categories: cs.CL

The Gap

LLM-as-a-judge grew up in the subjective world. MT-Bench, reward modeling, G-Eval — all of them score helpfulness, harmlessness, style, where there is no single right answer and approximate agreement with humans is good enough. Objective factual evaluation is a different animal: “Who voices Jack Kahuna Laguna in SpongeBob?” has exactly one right answer, and a reference-free judge either knows it or is bluffing.

Two failure modes follow, and both are silent. The judge may simply lack the knowledge — recent events, rare entities, specialized domains — and cannot tell a correct answer from a plausible fabrication. Or the judge has the knowledge and hallucinates anyway, producing a confident wrong verdict. Neither failure announces itself. The paper’s own qualitative table has a judge asserting that Steve Carell voiced the character and calling it “a well-documented fact.” The gold answer is Johnny Depp.

The field has three partial answers on the table, and each stops one step short. Tool-augmented judges (SAFE, SAGE) hand the judge a search engine, which fixes knowledge gaps but retrieves unconditionally — paying search cost on every instance, including the ones the judge already knew cold — and still offers no formal statement about how often the final verdicts are wrong. Uncertainty quantification (predictive entropy, self-consistency sampling) produces a score that correlates with correctness, but the threshold you apply to it is a hyperparameter someone tuned; correlation plus a hand-picked cutoff is not a guarantee. Risk-controlled selective prediction (COIN, SConU) does supply finite-sample guarantees, but it was built to filter examinee answers in QA, not to control errors in the meta-evaluation layer; the two closest works that do target judging — Trust or Escalate, SCOPE — guarantee agreement with humans on subjective pairwise preference, which is a different quantity than factual correctness. And every one of these treats abstention as the only escape hatch, which throws away exactly the instances where a two-second web lookup would have settled the matter.

[PROBLEM] reference-free judging of factual QA:
          judge lacks knowledge OR hallucinates,
          and both failures are silent
     |
     v
[PARTIAL FIXES AND WHERE THEY STOP]
   tool use     ==> fixes knowledge, retrieves always,
                    no error guarantee
   UQ + cutoff  ==> flags bad verdicts, cutoff is a
                    hyperparameter, no guarantee
   risk control ==> real guarantee, but built for
                    examinee answers / subjective
                    preference, abstain-only
     |
     v
[ASSUMPTION] the uncertainty score separates correct
             from incorrect verdicts well enough that
             a calibrated cutoff is useful; and the
             calibration set is i.i.d. with the
             deployment distribution
     |
     v
[METHOD] two modes, two thresholds, one joint
         calibration.  U1 <= t1 -> accept parametric
         verdict.  U1 > t1 and U2 <= t2 -> accept the
         retrieval-augmented verdict.  else abstain.
         Clopper-Pearson UCB on FDR among accepted.
     |
     +-- [LEMMA 1] routing is a deterministic function
     |             of the input, so selected failures
     |             stay i.i.d. Bernoulli and the
     |             single-threshold proof carries over
     |
     v
[EVIDENCE] 32 configs (2 candidates x 4 judges x
           4 datasets), 100 splits each.  FDR at or
           below alpha everywhere.  Coverage at
           alpha = .20 on NQ-Open: .07 -> .82;
           HotpotQA: .40 -> .85
     |
     v
[CONCLUSION] uncertainty should trigger *escalation*
             first and abstention only second, and the
             whole policy still admits one certificate

The Increment

One sentence: Before, an LLM judge on factual tasks either answered everything (with an unknown error rate) or abstained on anything it wasn’t sure about (with a guarantee, but throwing away most of the data); after, uncertainty routes an instance to web evidence before it routes it to the bin, and the whole three-way policy still carries one finite-sample bound on the error rate among the verdicts you keep.

Core Mechanism

Start with the single-mode base case. The judge J reads a question and a candidate answer and emits a binary verdict plus an uncertainty score U. The score here is predictive entropy computed at the verdict token — you take the log-probabilities of the True and False tokens from the same forward pass that produced the verdict and compute the binary entropy, so uncertainty costs zero extra inference. A held-out calibration set of N instances carries ground-truth labels. For any candidate threshold t you can count how many calibration instances would be accepted (U <= t) and how many of those the judge got wrong; that ratio is the empirical FDR. The trick is that you do not trust the empirical ratio — you feed the counts into a one-sided Clopper-Pearson upper confidence bound and pick the largest t whose bound stays under the user’s risk level alpha. That is a real guarantee: with probability at least 1 - delta over the calibration draw, the true error rate among accepted verdicts is at most alpha.

Now the actual contribution. Instead of one threshold, there are two, and a second evaluation mode sits between confidence and the bin. Mode 1 is the judge with parametric knowledge only. Mode 2 queries a web search engine with the original question, concatenates the top-k results (title, snippet, URL; k = 3 by default) into an evidence passage, appends it to the prompt, and re-runs the judge. At test time: if U1 <= t1, ship the Mode 1 verdict and never touch the search API; if U1 > t1 but U2 <= t2, ship the Mode 2 verdict; otherwise abstain and flag for human review. Retrieval cost is paid only on the instances Mode 1 could not close.

The calibration is joint, not sequential. During calibration both modes are pre-computed for all N instances, and the selection size and error count are computed for a candidate pair (t1, t2) — an instance counts as selected if either arm accepts it, and its error is attributed to whichever arm actually produced the verdict. Then a grid search over the sorted unique uncertainty values picks the pair maximizing coverage subject to the Clopper-Pearson bound staying under alpha, in O(|T1| * |T2|) time with cumulative sums. The theoretical work is Lemma 1: because both U1 and U2 are deterministic functions of the input (x, y_hat), the routing decision introduces no new randomness, so the failure indicators over the selected subset are still i.i.d. Bernoulli and the single-threshold Clopper-Pearson argument transfers verbatim. No extra distributional assumptions, no conformal machinery beyond what the base case needed.

CALIBRATION (offline, needs labels)      DEPLOYMENT (online)

for each (x, y_hat, y*) in D_cal:              (x, y_hat)
    v1, U1 <- J(x, y_hat)                          |
    E     <- RETRIEVE(x)                           v
    v2, U2 <- J(x, y_hat, E)                 +-----------+
    c1, c2 <- match against y*               | Mode 1: J |  parametric
        |                                    +-----------+
        v                                          |
  grid over (t1, t2):                          U1 <= t1 ?
    m = # accepted by either arm             /            \
    w = # errors among them                yes             no
    UCB = BetaInv(1-delta; w+1, m-w)        |               |
    keep pair with max m s.t. UCB <= alpha  v               v
        |                              [ACCEPT v1]    +-----------+
        v                               zero search   | RETRIEVE  |
   (t1_hat, t2_hat)  ---------------->   cost         | top-k web |
                                                      +-----------+
                                                            |
                                                            v
                                                      +-----------+
                                                      | Mode 2: J |
                                                      | (x,y_hat, |
                                                      |     E)    |
                                                      +-----------+
                                                            |
                                                       U2 <= t2 ?
                                                      /          \
                                                    yes           no
                                                     |             |
                                                     v             v
                                                [ACCEPT v2]   [ABSTAIN]
                                                              human review

  guarantee:  Pr( error rate among all ACCEPTed <= alpha ) >= 1 - delta

The metaphor: airport screening with a published clearance standard. Every passenger is an instance. Walking through the metal detector is Mode 1 — instant, free, and it produces not a decision but a reading. The sensitivity dial on that detector is t1. Passengers whose reading sits below the dial walk straight to the gate; nobody wastes a hand search on them.

Everyone above the dial goes to secondary screening: the bag comes off the belt, gets opened, gets X-rayed. That is retrieval — expensive, slower, but it puts actual evidence on the table instead of relying on the officer’s memory of what a suspicious silhouette looks like. Secondary screening has its own dial, t2. Clear it and you fly. Fail it — the X-ray was inconclusive, the contents ambiguous — and you don’t get waved through on a hunch; you get escalated to a human supervisor. Three outcomes, two dials, and crucially the second look happens before the refusal, which is the whole point. A single-dial system has to refuse everyone who beeps.

Where do the dial settings come from? Not from an officer’s intuition. A regulator runs a batch of test bags with known contents through both stages and sets both dials together so that, with 95% confidence, at most alpha of everything cleared — through either lane — was cleared in error. That joint setting is why the two dials cannot be tuned independently: loosening the first dial pushes more marginal cases into the fast lane, which means the second dial must tighten to keep the overall clearance standard. And the certificate covers cleared passengers, not all passengers — a system that refuses everybody trivially satisfies it. Which is exactly why coverage is reported alongside FDR, and why the interesting number is how many people actually fly.

Key Concepts

• FDR among accepted, not accuracy overall: The quantity being controlled is E[W | U <= t] — the error rate conditional on the judge having chosen to answer. This is subtler than it looks, and the conditioning is where naive approaches break. Suppose your judge is 70% accurate overall. Nothing stops the 30% it gets wrong from being concentrated in exactly the instances it feels most confident about — that is what a systematic hallucination looks like. Controlling FDR-among-accepted means you’re making a promise about the selected subpopulation, which is a moving target: change the threshold and you change which population you’re making the promise about. And note what the promise buys you. It does not make the judge smarter. It converts an unknown error rate into a knob: you say “I will tolerate 10% wrong verdicts,” the framework tells you what fraction of your data it can evaluate at that price, and the rest goes to a human. That’s the entire product.

• The Clopper-Pearson upper bound, and why the empirical rate isn’t enough: Imagine you accept 10 calibration instances at some threshold and the judge got all 10 right. Empirical FDR: 0%. Would you ship a claim of “0% error” on that basis? Obviously not — 10 samples is nothing. Clopper-Pearson answers the question properly: what is the largest true error rate that would still make “0 errors out of 10” a non-freakish observation at the 5% level? The answer is about 26% — because a coin that’s wrong 26% of the time still comes up clean ten times in a row with 5% probability. So the bound reports 26%, not 0%, and if your alpha is 10% that threshold gets rejected. This is exact and finite-sample: no normal approximation, no asymptotics, valid at N = 10 as much as at N = 10,000. The price is conservatism, and conservatism spends coverage — which is precisely the currency the second mode exists to earn back.

• Why routing doesn’t break the proof (and what would): The worry is natural. Conformal-style guarantees are fragile under adaptivity — if you let the system look at outcomes and then decide how to handle an instance, the selected failures stop being i.i.d. and the bound dies. Lemma 1’s observation is that this system never does that. The routing decision depends only on U1 and the fixed thresholds, and U1 is a deterministic function of (x, y_hat). The judge’s computation is deterministic; the retrieval is treated as deterministic given the query. So “which lane did this instance take” is just another deterministic feature of the input, conditioning on selection preserves i.i.d.-ness, and the Bernoulli structure survives. What would break it: routing that depends on the ground-truth label, retrieval whose results shift between calibration and deployment (the paper tests this — see below), or a threshold search that’s certified as if it were a single pre-specified pair (the paper is honest that this one is a real gap).

Framework Shift

Before (mainstream):                 After (this paper):

  judge always answers                 U1 -- below t1 --> ACCEPT (free)
        |                               |
        v                               +-- above t1 --> RETRIEVE
   verdict, unknown error rate                              |
                                                            v
  OR, with selective prediction:                    U2 -- below t2 --> ACCEPT
                                                     |
  U -- below t --> ACCEPT                            +-- above t2 --> ABSTAIN
    |
    +-- above t --> ABSTAIN            one certificate covers BOTH
                    (data lost)        accept paths:
                                       Pr(FDR <= alpha) >= 1 - delta
  guarantee: yes
  coverage on hard sets: ~5%           coverage on hard sets: ~74-85%
  uncertainty means: give up           uncertainty means: go look it up

  OR, with tool augmentation:

  every instance --> RETRIEVE          retrieval spend is targeted:
        |                              ~45% of TriviaQA never searches
        v                              (Qwen3-14B judge, alpha = .20)
   verdict, unknown error rate
   search cost on 100% of data

From abstain-or-guess to escalate-then-abstain, the core shift is treating an uncertainty score as a routing signal with a budget rather than a binary trust switch — and showing that the extra branch costs nothing in the proof.

Expert Assessment

Problem choice: Real, well-scoped, and sitting at a genuinely useful intersection. The field has been sliding toward LLM judges as infrastructure — RLAIF reward signals, agent self-evaluation, automated benchmark scoring — while the reliability story stayed at “GPT-4 correlates with humans at 0.8.” For subjective tasks that’s arguably fine. For factual tasks it’s a load-bearing beam nobody stress-tested. The specific insight that abstention is a wasteful default when the failure is a knowledge gap rather than an ambiguity is a good one, and obvious only in retrospect. The trajectory is clear: this is the third or fourth paper from adjacent groups pulling conformal risk control into the judging layer, and the combination with tool use is the natural next node.

Method maturity: Honestly, the method is simple, and I mean that as praise. Two thresholds, a grid search, and a textbook binomial bound. Lemma 1 is close to trivial once stated — the whole content is “routing is a deterministic function of the input” — but the value of a trivial lemma is that it licenses the construction without new assumptions, and papers that skip stating it are the ones that quietly break. My real reservation is elsewhere: the framework needs a labeled calibration set with ground-truth answers for every (task, candidate model, judge model) triple, and any change to any of the three requires recalibration. That’s a heavy operational tax on a method whose selling point was reference-free evaluation. It doesn’t eliminate labeled data; it amortizes it, trading a fully-labeled test set for a labeled calibration set plus an abstention rate. That’s a real win, but it should be framed as such rather than as escaping references altogether.

Experimental integrity: The main claim — FDR at or below alpha across all 32 candidate/judge/dataset configurations, 100 splits each, with bands of 0.01 to 0.02 — is exactly what it should be, and a guarantee that holds for a Qwen3-4B judge as well as a 14B one is a meaningful demonstration that the mechanism doesn’t secretly depend on judge strength. The coverage gains are large and specific (NQ-Open with a Qwen3-8B judge: 7% to 82% at alpha = .20; HotpotQA with Llama-8B: 40% to 85%). Three things give me pause, all visible in their own appendices, which speaks well of them:

First, the empty-evidence control. Appendix D.3 runs the Mode 2 prompt with the evidence slot left blank. On HotpotQA at alpha = .20, direct Mode 1 gets 5% coverage, empty-evidence Mode 2 gets 57%, and the full retrieval framework gets 74%. So the majority of the headline gain on the hardest dataset comes from the second forward pass under a different prompt, not from the retrieved web pages. The paper draws the opposite conclusion from the Self-eval control (which barely helps) and I don’t think that’s the right read — Self-eval and Empty are different interventions, and Empty is the one that lands. This deserved to be in the main paper, not buried at D.3.

Second, the selection-vs-certification gap. The deployed thresholds are chosen by grid search over the calibration data to maximize coverage subject to the bound. Clopper-Pearson is valid for each fixed pair; certifying the selected pair needs a correction. The authors know this and provide a Bonferroni variant — and its cost is brutal: NQ-Open coverage at alpha = .20 drops from 0.82 to 0.05, HotpotQA at alpha = .15 from 0.49 to 0.00. They argue, correctly, that Bonferroni is absurdly loose here because adjacent grid thresholds select nearly identical instance sets, and that the true jointly-valid coverage sits much closer to the pointwise numbers. I believe them. But “the number we report doesn’t have the guarantee we proved, and the number that does is near zero” is an uncomfortable place for a paper whose title says provable. A data-splitting variant — select thresholds on one half, certify on the other — would cost a factor of two in calibration data and close the hole cleanly. Its absence is the paper’s biggest missed opportunity.

Third, what the ground truth actually is. The label e* comes from an admission function, and the one used for all main results is Qwen2.5-7B-Instruct judging semantic equivalence against references. So the certified quantity is “at most alpha of accepted verdicts disagree with a 7B model’s opinion of correctness.” Table 7 shows how much this matters: on NQ-Open at alpha = .15, coverage is 56% under the LLM evaluator and exactly 0% under both exact match and token F1. The framework is agnostic to the choice, and the FDR guarantee holds under all three — but the meaning of the guarantee swings enormously, and a reader skimming the abstract will not catch that.

Credit where due on two fronts. The retrieval drift experiment is genuinely good practice: they re-queried the same 2,000 questions three months later, found the top-3 URL sets had turned over substantially (mean Jaccard 0.30 on TriviaQA, 0.24 on HotpotQA), applied the old thresholds anyway, and found FDR still controlled and coverage stable. That’s the right way to probe the determinism assumption Lemma 1 leans on. And the observation that retrieval raises uncertainty on 14-29% of instances — concentrated on multi-hop questions where snippets are partially relevant or sources disagree — is the kind of finding most papers would leave out. It’s also the cleanest justification for why t2 needs to exist at all.

Writing quality: Clean, well-organized, appropriately terse. Section 3.2 is a stub — it announces “Uncertainty Quantification” and says only that better separation yields higher coverage, deferring the actual definition to the experiments. Given that predictive entropy at the verdict token is the entire uncertainty story in the main results, and that Appendix D.2 shows a LoRA probe lifting HotpotQA coverage from 0.04 to 0.44 at alpha = .10, the UQ choice is doing far more work than its half-page suggests. The section I’d rewrite is the results narrative: promote the empty-evidence baseline and the Bonferroni table out of the appendix and address them head-on. The paper is strong enough to survive its own caveats, and burying them makes it look more fragile than it is.

Verdict: weak accept — a clean, correct, genuinely useful construction with an honest empirical program, held back by a certification gap the authors document but don’t close, and by a headline gain that their own control suggests is substantially prompt-driven rather than evidence-driven.

Takeaways

• Escalate before you abstain. This is the transferable idea and it costs almost nothing to adopt. Any system with an abstention path — RAG with a “I don’t know” fallback, a classifier with a reject option, a triage router — is throwing away instances where a cheap second look would resolve the uncertainty. Add a middle tier with its own threshold and calibrate the two together, not sequentially.
• A deterministic router is free in the proof. If your routing decision is a function of the input alone (no peeking at labels, no adaptivity across instances), you can bolt arbitrarily many branches onto a selective-prediction system and the finite-sample guarantee carries over unchanged. That’s a useful design constraint to know about in advance, because it tells you exactly which kinds of cleverness are forbidden.
• Uncertainty-gated tool calls, priced. The cost story here is concrete: at alpha = .20 with a Qwen3-14B judge on TriviaQA, roughly 45% of instances never touch the search API. If you’re paying per search call in an agent loop, “retrieve only when the calibrated threshold says you’re not confident enough” is a directly implementable spend policy with a stated error budget, not a vibe.
• Watch for the empty-evidence control in your own ablations. Before you attribute a retrieval win to retrieval, run the retrieval prompt with the evidence slot blank. The gap between that and your full system is the actual value of the evidence. This paper’s own numbers show the gap is often smaller than it looks.
• Ask what your ground truth is before you trust a bound. A finite-sample guarantee is a statement about agreement with whatever produced your calibration labels. If those labels came from another model, the certificate is about model agreement, and the word “provable” is carrying assumptions that live outside the proof.