Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data

Paper: 2606.05122 Authors: XiuYu Zhang, Yi Shan, Junfeng Fang, Zhenkai Liang Categories: cs.CL

The Gap

Current approaches treat LLM self-evaluation as a training problem: teach the model to predict judge scores through extensive reinforcement learning. The RewardBench baseline requires ~5000 examples. This assumes the ability must be acquired from scratch.

But what if the ability is already there, just hidden? The authors find that base models, with only few-shot prompting, already predict external judge scores well above chance. The gap: no one tried to elicit this latent capability efficiently instead of training it from zero.

Problem: LLM needs to predict judge scores
         |
         v
Assumption: Base models lack this ability
         |
         v
Old method: Train with 5000+ RL examples
         |
         v
This paper's insight: Ability already exists latently
         |
         v
New method: Elicit with 160 examples (SEE)
         |
         v
Evidence: Matches RL baseline calibration, 31x fewer examples
         |
         v
Conclusion: Self-evaluation is elicitation, not acquisition

The Increment

One sentence: Before, we trained models to evaluate themselves with thousands of examples; now, we elicit a pre-existing latent ability with 160 examples by coupling answer improvement with judge prediction.

Core Mechanism

SEE (Self-Evaluation Elicitation) has two phases. First, calibration-coupled RL: the model learns to both improve its answer AND predict how a judge will score it. The prediction happens before seeing the actual judge score, forcing the model to internalize quality signals. Second, masked distillation: the model’s answer tokens are frozen, and only the prediction head is updated. This sharpens the self-evaluation without degrading answer quality.

The cycle runs on just 160 unique examples. During calibration-coupled RL, the model generates an answer, predicts the judge’s multi-attribute scores (e.g., helpfulness, harmlessness), then sees the actual scores. It gets reward for both better answers and more accurate predictions. During masked distillation, the model sees the same examples again, but this time only the prediction mechanism is updated using the RL-generated trajectory as teaching signal.

Input prompt
     |
     v
[Generate answer] --> frozen in phase 2
     |
     v
[Predict judge scores] <-- updated in both phases
     |
     v
[See actual judge scores]
     |
     v
RL reward = answer quality + prediction accuracy
     |
     v
Phase 1: update both answer + prediction
Phase 2: update only prediction (answer frozen)

Think of it like learning to cook while learning to taste. Phase 1 is cooking practice: you make a dish, guess how a food critic would rate it, then see their actual rating. You improve both your cooking and your ability to predict the critic. Phase 2 is pure tasting practice: you taste dishes you already made, refine your prediction of the critic’s rating, but don’t change the dishes themselves. The coupling in phase 1 is key—you can’t learn to predict the critic in isolation; you need to feel how changes in your cooking change the scores.

Key Concepts

• Latent judge calibration: The base model already contains implicit knowledge of what makes a good answer. When you ask it to predict how a judge will score its own output, it can do so above chance even without targeted training. This isn’t oracle knowledge—it’s pattern recognition from pretraining. The model saw enough judge-like feedback (ratings, critiques, preferences) during pretraining that it internalized a quality model. SEE surfaces this latent model instead of building a new one from scratch.

• Calibration-coupled RL: Traditional RL optimizes answer quality. Traditional supervised learning optimizes prediction accuracy. Coupling means optimizing both simultaneously with a shared reward signal. The magic: when the model tries to predict the judge while improving the answer, it can’t just memorize judge behavior—it must learn the underlying quality notion that drives the judge’s decisions. It’s like learning physics by predicting experiment outcomes while designing experiments. The coupling forces you to learn principles, not just correlations.

• Masked distillation: After RL, you have a model that’s decent at both answering and predicting. But the answer quality might have drifted, and the prediction could be sharper. Masked distillation freezes the answer generation (masks those tokens) and distills only the prediction capability from the RL trajectory. You’re extracting the “tasting” skill while leaving the “cooking” skill untouched. The RL trajectory becomes the teacher: “here’s how you predicted when you were coupled with answer improvement—now do that consistently.”

Framework Shift

Before (RL baseline):                  After (SEE):

Train from scratch                     Elicit pre-existing ability
    |                                      |
    v                                      v
[5000 examples]                        [160 examples]
    |                                      |
    v                                      v
[RL: optimize answer]                  [Phase 1: couple answer + predict]
    |                                      |
    v                                      v
[Separate: train predictor]            [Phase 2: distill predict, freeze answer]
    |                                      |
    v                                      v
Judge-aligned model                    Judge-aligned model
(acquisition paradigm)                 (elicitation paradigm)

From “train a new skill with lots of data” to “surface a hidden skill with minimal data,” the core shift is recognizing that pretraining already embedded judge calibration.

Expert Assessment

Problem choice: Real gap. The field assumed self-evaluation requires extensive training, but the authors checked the null hypothesis first: does the base model already know something? The few-shot baseline (Table 1) is a simple sanity check no one did. This paper sits at the boundary between scaling (more data solves everything) and efficiency (maybe data isn’t the bottleneck). Important framing shift.

Method maturity: Clever. Coupling answer improvement with prediction is the key insight—it prevents the model from learning superficial judge patterns. The masked distillation phase is a neat trick to preserve answer quality while sharpening prediction. However, the method still uses RL, just less of it. The ablation (Table 2) shows removing coupling or distillation hurts, but doesn’t explore simpler alternatives like direct few-shot calibration with better prompting or retrieval-augmented few-shot. The 31x data reduction is impressive, but we don’t know if 160 is near the minimum or just “good enough.”

Experimental integrity: Solid. Three benchmarks (RewardBench, MT-Bench, AlpacaEval 2), multiple judges (GPT-4, Claude, Llama-3-70B), held-out judge transfer tested. Baselines are fair: they compare to a strong RL method and show few-shot base model performance. The calibration metrics (Table 1, 3) are appropriate. One concern: all experiments use Llama-3.1-8B. No evidence this transfers to other model families or sizes. The “localized in token distribution” claim (Figure 3) needs more investigation—what happens with out-of-distribution judges?

Writing quality: Abstract is dense but precise. Introduction sets up the problem clearly. Methods section is straightforward. The weakness: Section 4 (results) buries the key insight. Table 1’s few-shot baseline is the paper’s most important result—it shows latent ability exists—but it’s presented as a side note. The paper would hit harder if it led with “base models already do this above chance” and then motivated SEE as the efficient path to full performance. The related work section is thin; more engagement with RLHF alternatives (DPO, rejection sampling) would strengthen positioning.

Verdict: weak accept — The reframing from acquisition to elicitation is valuable, the method is efficient, and the experiments are decent, but the scope is narrow (one model family, one setting) and the writing doesn’t foreground the core insight early enough.

Takeaways

Concrete ideas you can steal:

1. Check latent capabilities before training: Before scaling up training data, run a few-shot baseline to see if the model already has a weak version of the target skill. If it does, elicitation methods (prompting, lightweight fine-tuning) might work better than training from scratch.

2. Couple related tasks during training: If you’re training a model to do X and you also want it to predict Y (where Y is related to X’s quality), train both together. The coupling forces the model to learn shared structure rather than task-specific heuristics. Example: training a code generator? Couple it with predicting test pass rate.

3. Masked distillation for selective updates: After training, if one capability drifted but another improved, freeze the drifted part and distill only the improved part from the training trajectory. Useful when you want to sharpen one skill without risking others.

4. Calibration as a diagnostic: Use a model’s ability to predict external judges as a probe for whether it learned generalizable quality notions vs. memorized judge quirks. If predictions transfer across judges, the model learned something real.