Beyond Scores: How an LLM Judge Actually Decides

Paper: 2609.01604 Authors: Himil Vasava, Ming Jiang Categories: cs.CL, cs.LG

The Gap

LLM-based evaluators are used in two capacities that both demand trust: as scoring tools for reporting quality, and as automated training signals for improving generators. In the second role their errors do not merely misreport — they are optimised against.

And yet the internal procedure by which a judge assigns a rating was poorly understood. Correlation studies showing that a judge’s scores track human scores tell you the output is useful; they say nothing about the mechanism, and therefore nothing about when it will fail. A training signal whose failure modes are uncharacterised is a liability in a way a reporting metric is not.

   TWO ROLES FOR LLM JUDGES, BOTH DEMANDING TRUST

   [1] SCORING TOOL
         reports quality
   [2] AUTOMATED TRAINING SIGNAL
         its errors are not just misreported -- they are
         OPTIMISED AGAINST
        |
        v
   [GAP] the internal procedure behind a rating was
         poorly understood
        |
        v
   correlation with human scores only says the OUTPUT
   is useful:
     - nothing about the MECHANISM
     - therefore nothing about WHEN IT FAILS
        -> a training signal with uncharacterised failure
           modes is a liability, unlike a reporting metric

The Increment

One sentence: Before this paper, how an LLM judge produces a rating was unexamined; after it, causal tracing identifies a two-stage pipeline with a sharp crystallization layer, and a base-model control isolates the two mechanisms fine-tuning installs on top of an existing substrate.

Core Mechanism

The instrumentation is thorough, and each piece answers a different kind of question.

An eight-attack perturbation taxonomy across the Readability and Adequacy dimensions. The attacks are the experimental stimulus: rather than observing the judge on ordinary inputs, the design introduces controlled errors of known kinds.

A generation pipeline producing paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps. This is what makes the analysis attributable — the corruption is localised and quantified, so when a mechanism responds, you know what it responded to.

A four-experiment battery: causal tracing, logit-lens vocabulary projection, and attention-head knockout. Three different instruments, which matters because the claims are about where computation happens and how information flows.

Applied to Themis (Llama-3-8B) and Prometheus (Mistral-7B), the result is a structured, coherent, two-stage pipeline:

• Below layer 15, attention performs local error comparison and routes the result to the final input position. The judge does not integrate everything at once; local comparison happens in attention, and its output is routed — the mechanism is explicitly about moving a signal to a place, not just computing it.
• Above layer 15, an MLP cascade integrates the signal and writes the rating. Integration and writing are separate from comparison and routing.
• The decision crystallizes in the residual stream at a sharp late layer — L=26 on Themis, L=25 on Prometheus. A “sharp” crystallization layer is a strong claim: the rating becomes determined at a specific depth rather than gradually.

Then the element that elevates this from a description to a finding: a base-model control at the same scale (Llama-3-8B) reproduces the routing architecture and the crystallization but not the stage separation. Because the base model shares the substrate, this control isolates the two mechanisms that fine-tuning specifically installs:

1. Suppression of below-L15 MLP contribution at the last position.
2. A two-layer advance of the crystallization depth.

The framing this supports is the paper’s most interesting claim: fine-tuning sculpts an existing substrate rather than building the pipeline from scratch. The base model already had the routing and the crystallization; what the judge training added was a change to the division of labour between attention and MLP at one position, and a shift in when the decision hardens.

   INSTRUMENTATION
     eight-attack perturbation taxonomy
       (Readability + Adequacy dimensions)
     paired clean/corrupt summaries,
       controlled error intensity + token-level maps
     four-experiment battery:
       causal tracing | logit-lens projection | head knockout

   JUDGES STUDIED: Themis (Llama-3-8B), Prometheus (Mistral-7B)

   FINDING: A TWO-STAGE PIPELINE
     BELOW L15: attention does LOCAL ERROR COMPARISON
                and ROUTES the result to the final input position
     ABOVE L15: MLP CASCADE integrates the signal and
                WRITES the rating
     CRYSTALLIZATION: the decision hardens at a SHARP late layer
                L=26 (Themis) | L=25 (Prometheus)

   BASE-MODEL CONTROL (Llama-3-8B, same scale)
     reproduces: routing architecture + crystallization
     does NOT reproduce: stage separation
        -> ISOLATES WHAT FINE-TUNING INSTALLS
             [1] suppression of below-L15 MLP contribution
                 at the last position
             [2] a TWO-LAYER ADVANCE of the crystallization depth
        -> CONCLUSION: fine-tuning SCULPTS AN EXISTING SUBSTRATE
           rather than building the pipeline from scratch

Think of it as opening up a calibrated instrument and finding it was assembled from parts already in the box. You expect a bespoke measuring device to be built specially. Instead you find the housing, the wiring and the readout were already there, and the calibration step added exactly two things: it disconnected one component and moved the point at which the reading locks. That is a much more specific account than “the model learned to judge”, and it is falsifiable — the base model is the control that shows the parts were pre-existing.

Key Concepts

• Judge mechanism as a two-stage pipeline: local error comparison and routing in attention below layer 15, integration and rating in an MLP cascade above it. The separation into stages is the structural finding, and it distinguishes an engineered pipeline from a generic scoring behaviour.
• Routing to the final input position: the signal is not merely computed but moved to a specific location. Naming routing as a distinct step is what makes the below-L15 stage comparable to mechanisms like visual retrieval heads.
• Sharp crystallization: the rating becoming determined at a specific late layer (L=26, L=25) rather than accumulating gradually. It gives the pipeline a definite endpoint within the network.
• The base-model control as the isolator: same scale, same family, reproducing the architecture but not the stage separation. Without it, the finding would be a description of a judge; with it, it becomes a decomposition of what training adds.
• Sculpting rather than building: fine-tuning suppressing one contribution and advancing one transition depth. It reframes fine-tuning as a small edit to an existing computation rather than the construction of a new one.

Framework Shift

Before (judges as black-box scorers):
  a judge correlates with human scores
  -> its output is trusted, its procedure unknown
  -> failure modes uncharacterised
  -> used as a training signal anyway

After (judge as a traceable two-stage pipeline):
  below L15: attention compares locally, routes to final position
  above L15: MLP cascade integrates and writes the rating
  crystallization is SHARP (L=26 / L=25)
  base-model control: fine-tuning suppressed one contribution
    and advanced the crystallization depth by two layers
  -> fine-tuning sculpts an existing substrate

From trusting a judge’s scores because they correlate with human ratings, to knowing where in the network the comparison happens and where the decision hardens, the core shift is that a training signal’s mechanism can be traced, and tracing it identifies what is editable.

Expert Assessment

Problem choice: Excellent, and the motivation is the right one. Judges are used as training signals, which means their systematic errors are optimised against rather than merely reported — and you cannot reason about the failure modes of a mechanism you have not located.

Method maturity: The instrumentation is the strength: three complementary instruments (causal tracing, logit-lens projection, head knockout) rather than one, and a stimulus design with controlled corruption and token-level maps so that responses are attributable. The base-model control is the strongest design decision, because it converts a descriptive claim into a decomposition — and the resulting “sculpts rather than builds” framing is a genuinely useful way to think about what fine-tuning does.

Experimental integrity: The two judges come from different families, so the pipeline structure is not a single-architecture artefact, and the crystallization depths differ slightly between them while the shape holds. The perturbation taxonomy spans two quality dimensions rather than one. The main limitation is scope: two judges on one task family (summarization), so the stage structure’s generality is suggested rather than established, and the paper is appropriately careful not to claim it holds for all judges.

Writing quality: The abstract is information-dense but follows a clear order — taxonomy, pipeline, control, isolation, conclusion — and the “sculpts an existing substrate” phrase carries the interpretation. Because the practical readership wants to know which layers to intervene on, a short practical section — given this structure, where would you edit a judge to change what it rewards? — would make the finding considerably more usable.

Verdict: strong accept — it replaces an unexamined scoring procedure with a traced two-stage pipeline, and the base-model control isolates precise, actionable components of what judge training installs.

Takeaways

• Trace the mechanism behind any scoring tool you optimise against. A metric that only correlates with quality tells you nothing about when it will fail.
• Use a base-model control at matched scale. It separates what training added from what the architecture already provided, which is the difference between a description and a decomposition.
• Look for a crystallization layer. Knowing where a decision hardens tells you where an intervention would actually take effect.
• Check whether a pipeline stage is computing or routing. The below-layer-15 stage moves a comparison result to a specific position, and routing is an editable operation in a way that a diffuse computation is not.