Decoding the Unfair Judge: How LLM Biases Live in Hidden Geometric Space

Paper: 2607.11871 Authors: Zixiang Xu, Sixian Li, Huaxing Liu, Xiang Wang, Shuai Li, Zirui Song, Xiuying Chen Categories: cs.LG, cs.AI, cs.CL

The Gap

We’ve gotten pretty good at spotting that an LLM judge is biased. The standard playbook is the input-output view: you systematically perturb prompts (e.g., changing a candidate’s name from “John” to “Jamal”), measure the score delta, and then propose prompt-engineering fixes like “Please be impartial.” This works to document the phenomenon but treats the model’s internals as a black box. The fundamental limitation is that we only see the symptom (score differences) and not the mechanism (where and how the bias is encoded in the model’s processing). This paper identifies this boundary: prior work lacks a mechanistic, representation-level account of bias. The logical path is clear: if we can find the internal representation of bias, we might be able to not just describe it, but also control it and predict it before it manifests as a score.

Prior View (Black Box)
[Input Prompt] --(LLM)--> [Biased Score] 
Limitation: No mechanistic understanding
  v
This Paper's Query: Where *inside* the LLM does bias live?
  v
Method: Probe hidden states for geometric structure
  v
Evidence: Found low-dim bias subspaces; can steer scores
  v
Conclusion: Bias is a geometric object in activation space

The Increment

One sentence: Before this paper, we diagnosed LLM judge bias from the outside by observing its scores; after this paper, we can find its geometric address inside the model’s hidden states and use that map to both control and predict bias.

Core Mechanism

The method operates in three phases, all centered on analyzing the hidden-state activations (the internal vectors) the LLM produces for a given input prompt. First, they establish a baseline activation manifold by feeding the judge a large set of “clean,” standard evaluation prompts. The hidden states from these inputs cluster tightly in high-dimensional space. Next, they introduce biased inputs—prompts containing a specific bias type (e.g., position bias, verbosity bias). The hidden states for these biased inputs are not scattered randomly; they are systematically *displaced from the baseline cluster along a consistent direction. This direction, or bias subspace, is identified using techniques like linear probing and PCA. Crucially, this subspace is low-dimensional (a few key directions) and becomes more pronounced (sharper) as you move deeper into the model’s layers.

The second phase is causal intervention. They don’t just observe the displacement; they actively steer the model’s hidden states. Using techniques like activation addition (e.g., adding a scaled version of the bias direction vector to a clean prompt’s activations), they can *reproduce biased scoring behavior on a clean input. Conversely, subtracting that vector from a biased input’s activations can restore fair scoring. This proves the subspace isn’t just correlated with bias—it’s causally involved in producing it. A control experiment using random direction vectors shows no such effect, ruling out noise.

Finally, the operational phase uses this geometric understanding as a predictive tool. They train a simple linear classifier (or even just measure the projection distance) onto the pre-identified bias directions. When this model is tested on completely new benchmarks, it can predict *which prompts are likely to trigger bias and even anticipate the direction of the score shift, outperforming methods that try to predict bias from the text of the prompt alone.

Method Internal Flow:
1. Data Collection
   [Clean Prompts] --> [LLM] --> [Hidden States] --> [Baseline Cluster]
   [Biased Prompts] --> [LLM] --> [Hidden States] --> [Displaced Points]

2. Geometric Analysis (per layer)
   [Baseline + Displaced States] --> [Linear Probe / PCA] --> [Bias Subspace Direction (B)]
   
3. Causal Test (Activation Steering)
   [Clean Hidden State (h)] + alpha*[B] --> [LLM] --> [Biased Score]
   [Biased Hidden State (h)] - beta*[B] --> [LLM] --> [Fair Score]

4. Operational Prediction (on new data)
   [New Prompt] --> [LLM] --> [Hidden State (h_new)] --> [Project h_new onto B] --> [Bias Prediction]

Now, think of the LLM judge as a human judge in a courtroom.

• The input prompt is the case file submitted for review.
• The hidden states are the judge’s internal mental notes and gut reactions as they read the file. We can’t see these directly.
• The baseline manifold is the judge’s state of mind when reading standard, unremarkable case files. Their notes are in a predictable “zone.”
• When a biased file arrives (e.g., one from a favored law firm), the judge’s mental notes shift. They don’t jump to a random thought; they shift along a specific mental axis—like an axis of “prestige” or “familiarity.” This is the bias subspace.
• Steering the activations is like us being able to hypnotically insert or remove that specific “prestige” thought from the judge’s mind. We can make them lean towards a biased verdict on a fair file, or correct their leaning on a biased file.
• The operational prediction is our new ability: just by looking at how a new case file makes the judge’s mental notes move along that known “prestige” axis, we can predict *before the verdict if they’ll be biased, without needing to see the final score.

Key Concepts

• Activation Manifold: Imagine all the hidden-state vectors for every prompt the model has ever processed. They don’t fill the entire high-dimensional space randomly; they live on a lower-dimensional, structured “surface” or “sheet.” This is the activation manifold. The paper shows that fair judging inputs all live on one, tight part of this sheet. Biased inputs get pushed off that tight region in a specific direction.
• Causal Control via Representation Steering: This is the technique that moves from correlation to causation. It’s like finding a specific knob on a machine labeled “bias.” By turning the knob (adding a specific vector to the hidden state), you reliably change the output in a predicted way. The paper’s key evidence is that this “knob” exists and is type-specific, and that randomly fiddling with other knobs (random directions) does nothing. This proves the discovered geometry isn’t a spurious correlation.
• Linear Probing as a Diagnostic: This is a simple but powerful idea. You train a linear model (like basic regression) to predict some property (like “is this input biased?”) using the model’s hidden states as input. If the probe works well, it means the hidden states already encode that property in a way that is *linearly separable—i.e., there’s a simple, flat “plane” that can divide the biased from the unbiased activations. Finding this plane is exactly finding the bias subspace.

Framework Shift

Before (mainstream approach):       After (this paper):
[Input] -> (Black Box LLM) -> [Score]
           |
      Observe score delta
           |
      Hypothesize causes (e.g., name)
           |
      Mitigate via prompt engineering

[Input] -> [LLM] -> [Hidden States] -> [Geometric Analysis]
                    |
              Identify Bias Subspace
                    |
              -> [Causal Steering]
              -> [Operational Prediction]

One sentence: From observing the judge’s *verdicts to mapping the judge’s internal mental geometry, the core shift is treating bias as a discoverable and steerable representation rather than a noisy output phenomenon.

Expert Assessment

Problem choice: Real and important. The “LLM-as-a-judge” paradigm is exploding in use for evaluation and alignment. Understanding its failure modes mechanistically is critical for trust and safety. This paper correctly identifies that the input-output view has hit diminishing returns and that a deeper, representation-level account is a logical and necessary next step. It sits at a natural inflection point in the field’s trajectory.

Method maturity: Clever insight leveraging established mechanistic interpretability tools (probing, activation steering) rather than brute force. The method is principled. A simpler approach being overlooked might be analyzing attention patterns or comparing specific neuron activations, but the manifold/displacement view is arguably more general and geometrically intuitive. The use of three different estimators (PCA, linear probing, etc.) to confirm the subspace adds rigor.

Experimental integrity: Strong. They test across 7 judges, 7 bias types, and 9 benchmarks, which is a broad sweep. The causal steering experiments are the gold standard for proving mechanism, not just correlation. The operational prediction test on three entirely unseen benchmarks is a crucial and convincing external validation. The baselines (text-based prediction, random-direction steering) are appropriate. I don’t see major red flags; the numbers appear to hold up.

Writing quality: The abstract and core narrative are crisp. The authors cut some corners in the Related Work section; it could do a better job explicitly connecting to the broader interpretability literature on “feature” discovery in LLMs beyond bias. The Limitations section is present but could be more expansive on the practical hurdles of scaling this for real-time judge monitoring. A rewrite of the Related Work to position this as a specific instance of “feature vector” discovery would elevate the paper’s framing.

Verdict: strong accept — It provides a clean, causal, and operationally useful mechanistic account of a critical problem, setting a new standard for how to study LLM-as-judge bias.

Takeaways

A practitioner can steal two concrete techniques:

1. Bias Detection via Linear Probing: If you suspect your LLM judge is biased on a certain axis (e.g., verbosity), you don’t need to run thousands of score comparisons. Collect a small set of known biased and clean prompts, run them through your judge, and train a simple linear classifier on the final hidden layer’s activations to predict bias. If it works, you have a lightweight, probe-based bias detector that doesn’t need to invoke the judge’s final scoring head.
2. Bias Mitigation via Representation Steering: The paper’s causal results suggest a potential debiasing technique. For a known bias type, you could compute the “bias direction” from a calibration set. At inference time, for each new prompt, you could subtract a small component of that direction from the hidden state before it flows through the final layers. This is a model-internal fix that could be more robust than prompt engineering, though it requires white-box access.