Clean Engineering, Unstable Measurement: A Model Name Is Not a Frozen Instrument

Paper: 2609.04198 Authors: Haoyaun Zhu, Jie Zhang Categories: cs.AI, cs.LG

The Gap

Language-model judges have become load-bearing infrastructure: they gate training data, score generations, and drive leaderboards. At that point a judge is not a heuristic — it is a measurement instrument, and instruments carry assumptions. The one this paper audits is rarely stated:

The same request, sent to the same model name, reads the same tomorrow.

That assumption is what makes a judge usable for gating. If it fails, then a gate frozen on today’s readouts is calibrated against a measurement that will not reproduce, and every downstream decision inherits an error nobody can see. The paper’s framing — measurement rather than modelling — is what makes the failure legible: it is not that the judge is a poor model, it is that the instrument does not hold still.

   LM JUDGES AS INFRASTRUCTURE

   they GATE TRAINING DATA, SCORE GENERATIONS,
   DRIVE LEADERBOARDS
        |
        v
   at that point the judge is not a heuristic --
   it is a MEASUREMENT INSTRUMENT
        |
        v
   instruments carry assumptions
   THE ONE AUDITED HERE, RARELY STATED:
     the same request, sent to the same MODEL NAME,
     reads the same TOMORROW
        |
        v
   [WHY IT MATTERS] a gate frozen on today's readouts is
   calibrated against a measurement that will not reproduce
     -> every downstream decision inherits an invisible error
   [FRAMING] measurement, not modelling:
     not "the judge is a poor model",
     but "the INSTRUMENT does not hold still"

The Increment

One sentence: Before this paper, the reproducibility of judge scores was assumed; after it, two preregistered campaigns that never got past validating their instrument show that same-day reruns disagree at 0.400 Spearman against a required 0.90, and that neither waiting nor changing provider repairs it.

Core Mechanism

The campaign design is the contribution as much as the result, and its key property is that every threshold was fixed in advance. That turns “the judge is unreliable” into a specific, falsifiable claim: the gate was set, the instrument missed it, and neither campaign got past validating its instrument — meaning the studies stopped at the first stage, before any substantive question could be asked.

The numbers are stark and worth reading in full. Across 52,988 audited request attempts:

• Same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.
• Byte-identical next-day replays agreed at 0.78 against a required 0.99.
• In both cases the execution record was at ceiling — so the infrastructure was working correctly. The instability is in the measurement, not in the plumbing.

Three mechanisms explain the gap, and each is a distinct failure of the instrument:

A label-to-meaning mapping that biased readouts as strongly as the signal. The mapping from a judge’s label to a number carries its own bias, of comparable magnitude to the effect being measured.

Candidate gaps seven orders of magnitude below the instrument’s own noise floor. This is the most damning one: the differences being ranked were smaller than the noise, so the ranking was noise with extra steps.

Byte-identical inputs returning different rankings — with exact-permutation readouts compounding that noise rather than averaging it out. Identical inputs, different outputs, is the cleanest possible demonstration that the model name is not a frozen instrument.

Two attempted repairs failed on the tested grid: neither metric substitution nor sampling repaired it. Then the preregistered follow-ups, which are what make the paper’s scope honest — each closes an intuitive escape:

• Waiting did not help — 0.805 versus 0.800, replicated over five further days. So the instability is not transient.
• Switching providers did not help — four providers share the floor, medians 0.74 to 0.88, and this was predicted by none of the metadata fields they expose. The problem is not vendor-specific and not detectable from the information vendors publish.
• Self-hosting on batch-invariant kernels helped only while the server was quiet — a partial, conditional fix that fails as soon as load arrives.
• On constructed errors with known gaps, the readout’s separation tracks error type, not size. So the instrument is sensitive to the wrong property: it responds to what kind of error rather than how large.

What the paper produces from this is deliberately practical: a three-level snapshot-identity ladder, eight design rules, and a reporting checklist. And the point that gives it teeth — a pilot at roughly 2% of the study’s call volume would have exposed both unreachable gates in advance. So the methodology the paper recommends is cheap, and its absence is what let two studies run to completion against an instrument that never worked.

The central prescription is the sentence the whole paper supports: on a shared endpoint, a model name is not a frozen instrument; a preregistered evaluation must measure its instrument before freezing any gate on it.

   TWO PREREGISTERED CAMPAIGNS
     every threshold FIXED IN ADVANCE
     NEITHER got past VALIDATING ITS INSTRUMENT
        <- studies stopped at the first stage, before any
           substantive question could be asked

   SCALE: 52,988 audited request attempts

     same-window repeat rankings: Spearman 0.400  (required 0.90)
     byte-identical next-day:     0.78             (required 0.99)
     AND in both cases the EXECUTION RECORD WAS AT CEILING
       -> the infrastructure worked; the MEASUREMENT did not

   THREE MECHANISMS
     [1] a LABEL-TO-MEANING MAPPING that biased readouts as
         strongly as the signal
     [2] candidate gaps SEVEN ORDERS OF MAGNITUDE BELOW the
         instrument's own noise floor
         -> the ranking was noise with extra steps
     [3] BYTE-IDENTICAL INPUTS returning different rankings,
         with exact-permutation readouts COMPOUNDING the noise
         <- the cleanest demonstration that a model name is
            not a frozen instrument

   REPAIRS THAT FAILED ON THE TESTED GRID
     metric substitution: no
     sampling:            no

And the preregistered follow-ups, each closing an intuitive escape:

   "wait and it will stabilise"
     -> 0.805 vs 0.800, replicated over FIVE further days: NO

   "it is this vendor"
     -> FOUR providers share the floor (medians 0.74-0.88),
        and NONE of their exposed metadata fields predicted it

   "self-host it"
     -> batch-invariant kernels helped ONLY WHILE THE SERVER
        WAS QUIET

   "then the metric is just insensitive to size"
     -> on constructed errors with KNOWN gaps, the readout
        separates by ERROR TYPE, not SIZE

   OUTPUT: a three-level snapshot-identity ladder,
           eight design rules, a reporting checklist
   AND: a pilot at ~2% of the call volume would have exposed
        BOTH unreachable gates in advance

Think of it as a ruler made of a material that changes length with the weather. You could use it to compare two objects measured back-to-back and get a consistent ordering — same-window agreement was 0.400 against a required 0.90, so even that was shaky. Measure again tomorrow and the ordering changes, because the ruler changed and everything else did not: that is the byte-identical-input result. The three mechanisms have analogues too. A scale whose tick labels are misaligned introduces bias as large as the weight difference you are trying to detect. Objects differing by a hair when the ruler’s own markings vary by a centimetre is the seven-orders-of-magnitude finding. And a ruler that responds to what kind of material you measure rather than how long it is answers a question you did not ask. Buying rulers from four suppliers does not help, because they were all cut on the same machine.

Key Concepts

• A model name as an instrument, not an identity: “the same name tomorrow” is a reliability assumption, and the paper shows it fails for shared endpoints. It is the reframing that makes prompt-level reproducibility a measurement concern.
• Preregistered thresholds as the instrument of the audit: fixing the gate in advance is what makes “unreliable” a falsifiable claim rather than an impression, and it is what allows the paper to report that both studies stopped rather than that they concluded something weak.
• A noise floor above the signal: candidate gaps seven orders of magnitude below the instrument’s noise. When this happens, a ranking is not a mediocre measurement; it is not a measurement.
• Failure to track size: separation follows error type rather than magnitude, so the readout is not monotone in the quantity of interest. A metric can be sensitive and still answer the wrong question.
• The cheap pilot: roughly 2% of the call volume would have exposed both failures. It converts the paper from a warning into a procedure.

Framework Shift

Before (a shared endpoint as a stable judge):
  send the same request to the same model name
  -> assume the readout reproduces
  -> freeze gates on those readouts
  -> rank models, filter training data, publish leaderboards

After (measuring the instrument first):
  preregister the thresholds, then audit reproducibility
  -> same-window 0.400 (need 0.90); next-day 0.78 (need 0.99)
  -> execution record at ceiling: the infrastructure is fine
  -> waiting, switching vendor, and self-hosting do not fix it
  -> run a 2% pilot to validate the instrument BEFORE any gate

From treating a model name as an instrument you can freeze, to validating the instrument’s reproducibility before gating anything on it, the core shift is that a shared endpoint makes the measurement — not just the model — an object that has to be checked.

Expert Assessment

Problem choice: Excellent, and overdue. Judges have moved from convenience to infrastructure, and the assumption that a model name reads consistently is exactly the kind of premise that becomes invisible once it is load-bearing. Auditing it with preregistration, rather than reporting an anecdote about flakiness, is what makes the result usable.

Method maturity: The preregistered design is the methodological core, and the decision to report that neither campaign got past instrument validation is the honest and more useful outcome — a paper could have been written around a weakened endpoint, and this one refuses. The follow-up campaign is well targeted: waiting, provider switching, self-hosting and metric sensitivity are precisely the four repairs a practitioner would attempt, and each is closed with evidence rather than argument. The three mechanisms are distinguishable and each implies a different mitigation, which is what makes the diagnosis actionable.

Experimental integrity: The strengths are the fixed thresholds, the 52,988-attempt scale, the ceiling execution record (which forecloses “your infrastructure was broken”), and the replicated waiting result. The scope statement is also appropriately tight: all results concern externally measured behaviour on shared serving infrastructure, so nothing here claims that self-hosted models with pinned weights and deterministic kernels are unstable — and the batch-invariant finding that it worked while the server was quiet is consistent with that boundary. The natural open question is whether the instability is specific to this benchmark task or general, which the paper leaves open.

Writing quality: The three mechanisms are named compactly and the follow-ups are organised as escapes foreclosed, which is a structure a reader can act on directly. Because the deliverable is procedural, the two-by-two of design rules and checklist are the right artifacts, and the 2% pilot figure is the single most useful number in the paper. A short worked example of the ladder applied to one judge endpoint would let a team adopt it the same afternoon.

Verdict: strong accept — it audits the premise that makes judge-based evaluation possible, finds it false on shared infrastructure, closes every intuitive repair with evidence, and hands over a cheap procedure for detecting the same problem elsewhere.

Takeaways

• Validate the instrument before freezing a gate on it. A preregistered reproducibility check at a fraction of the call volume comes before any substantive evaluation, not after.
• Check whether your effect exceeds the noise floor. If the differences you rank are orders of magnitude below the instrument’s own variation, you are ranking noise.
• Test that your metric tracks magnitude, not just category. A readout that separates by error type rather than error size answers a different question than the one you asked.
• Do not assume the fix is vendor-specific. Four providers shared the same floor, and none of their published metadata predicted it — so provider switching is not a remedy and is not informed by what vendors disclose.