Agreement ≠ Validity: Why Your LLM Annotator Might Be Faking It

Paper: 2607.08731 Authors: Manuel Pita Categories: cs.CL, cs.AI, cs.CY

The Gap

Researchers have been benchmarking LLMs as annotators primarily by one metric: agreement with human coders. If an LLM’s codes match human codes at high F1, it gets a gold star. Open models like GPT-4 and Llama-70B score well on moral foundation coding tasks, and smaller national models like Portugal’s AMALIA-9B come within six F1 points—impressive for a model eight to thirteen times smaller.

But here’s the problem nobody was stress-testing: agreement measures reliability, not validity. A model can reach the right code for the wrong reason. For theoretical constructs like moral foundations—where the coding depends on inferring implicit reasoning, not just reading surface words—this distinction matters enormously. Prior work largely accepted high agreement as sufficient evidence of capability.

This paper introduces the recovery gap diagnostic and applies it to AMALIA for the first time. The result is sobering.

Problem: LLM annotator benchmarks test agreement, not reasoning path
  |
  v
Assumption: High agreement = model understands the construct
  |
  v
Method: Decompose holistic prompt into atomic codebook clauses
  |       recombine by construct's explicit rule
  |       measure performance gap (recovery gap)
  |
  v
Evidence: AMALIA-9B loses ~50% F1 on decomposition
  |        multilingual model closes the gap
  |        error analysis shows surface-cue reliance
  |
  v
Conclusion: Agreement alone cannot validate construct measurement
             sovereign LLM benchmarks need evidential route testing

The Increment

One sentence: Before this paper, we could ask “does this LLM agree with humans?”; after, we must also ask “does it reach the right answer through the theory’s reasoning path?”

Core Mechanism

The method has three stages. First, you take a standard annotation task—here, coding whether text expresses the moral foundation of authority—and present it as a holistic prompt: “Does this text express concerns about authority? Code Yes/No with justification.” This is how most benchmarks work, and AMALIA scores well.

Second, you decompose the coding rubric into its atomic clauses. The codebook for authority isn’t one rule; it’s a bundle of sub-criteria: Does it reference respect for tradition? Deference to legitimate hierarchy? Obedience to social order? Each becomes a separate yes/no question posed to the model independently.

Third, you recombine the atomic answers using the codebook’s explicit logical rule (e.g., “code Yes if criterion A or B is met, and C is present”). The gap between the holistic score and the recombined score is the recovery gap. If the model truly understands the construct, the gap should be small—decomposition shouldn’t hurt much because the model was already doing the reasoning internally. If the gap is large, the model was shortcuts-dependent.

Holistic Prompt                    Decomposition
  "Does text express               "Does text reference
   moral authority?"                tradition? (Y/N)"
        |                          "Does text show deference
        |                           to hierarchy? (Y/N)"
        v                          "Does text invoke social
   AMALIA: F1 = 0.78               order? (Y/N)"
                                       |
                                       v
                                   Recombine by codebook rule
                                       |
                                       v
                                   AMALIA: F1 = 0.42
                                       
                                   Recovery gap: 0.36
                                   (nearly half the performance lost)

Think of it like grading a student’s essay. You ask: “Does this essay demonstrate understanding of the French Revolution?” The student gets an A. Impressive. Now you break it down: “Does the essay mention the storming of the Bastille? Does it explain the role of the Third Estate? Does it connect economic crisis to political upheaval?” You grade each piece separately, then apply the rubric’s rule for what counts as “demonstrates understanding.” The student’s grade drops to a C.

What happened? The student had learned surface patterns—the right vocabulary, the right essay structure—that correlated with good grades without actually understanding the causal chain. The decomposition stress-tests whether the reasoning is actually present or merely performed.

That’s exactly what happened with AMALIA. Error analysis showed it was latching onto moral outrage near authority figures as a shortcut—texts that contained emotional language directed at leaders got coded as “authority” even when the construct’s theory wouldn’t support it. The model found a correlated signal that worked in the training distribution but broke under decomposition.

Crucially, an open multilingual LLM (the paper doesn’t specify which, but context suggests something like Aya or BLOOMZ) tested on the same Portuguese corpus with the same instructions did close the recovery gap. This rules out the corpus itself as the problem and points squarely at the model-construct fit.

Key Concepts

• Recovery gap: Imagine you give someone a recipe for a cake and ask “will this taste good?” They say yes, correctly. Now you ask them: “Is there enough sugar? Is the flour-to-butter ratio right? Is the baking time correct?” They can’t answer each sub-question. Their holistic judgment was based on recognizing that this recipe “looks like” good recipes, not on understanding why each ingredient matters. The recovery gap measures the difference between holistic performance and decomposed-then-recombined performance. A small gap means genuine understanding; a large gap means pattern matching.

• Construct validity vs. reliability: Reliability means you get consistent answers. Validity means you get the right answers for the right reasons. A broken clock is reliable (it shows the same time every time you look) but not valid. An LLM that codes all angry political speech as “authority” might be reliable—consistent, reproducible, high-agreement—but invalid, because authority is a specific moral foundation with specific theoretical boundaries that anger alone doesn’t satisfy.

• Sovereign LLM benchmarking: Several countries are building national language models—Portugal’s AMALIA, France’s CamemBERT successors, various Nordic and Arabic initiatives. These models serve linguistic communities that global English-centric models underserve. But how do you know if they’re actually good at understanding your language’s moral, political, and cultural nuances, or just good at surface-level pattern matching? This paper argues that current benchmark batteries—which test agreement—are insufficient. You also need to test the evidential route.

Framework Shift

Before (mainstream approach):        After (this paper):

Prompt --> LLM --> Code              Prompt --> LLM --> Code
                |                                   |
                v                                   v
          Compare to human              Also: Decompose prompt
          labels, report F1             into atomic clauses
                                        |
                                        v
                                  Recombine by theory's rule
                                        |
                                        v
                                  Compare holistic vs
                                  decomposed performance
                                        |
                                        v
                                  Gap = validity diagnostic

From testing whether the model gets the right answer to testing how it gets the right answer, the core shift is from agreement-as-evidence to reasoning-path-as-evidence.

Expert Assessment

Problem choice: This is a genuine and important gap. The field has been lazily conflating agreement with validity for LLM-as-annotator work. The insight that decomposition can expose surface-cue reliance isn’t entirely new in psychometrics, but applying it systematically to LLM annotation validation is fresh and timely given the surge in national model initiatives. Good placement in the field’s trajectory.

Method maturity: The recovery gap concept is elegant—not brute force, but a clever adaptation of construct validation techniques from psychometrics. That said, the decomposition itself requires a well-structured codebook with atomic clauses, which limits applicability to domains where such codebooks exist (a real but manageable constraint). The approach also assumes the codebook’s decomposition rule is the correct one, which is itself a theoretical commitment.

Experimental integrity: The key comparison—AMALIA vs. a multilingual model on the same corpus—is well-designed and controls for corpus effects. The error analysis grounding the “surface correlates” claim is qualitative, which is appropriate for a single-construct case study but would need scaling to be conclusive. One concern: the paper doesn’t name the multilingual model or provide its full benchmark details, which weakens reproducibility. The six-F1-point gap to larger models is reported but the specific open models aren’t enumerated in the abstract.

Writing quality: The abstract is unusually well-written for an academic paper—it tells a complete story with a clear punchline. The methodology description could be more precise about the decomposition protocol: how many atomic clauses? What exactly was the recombination rule? A reader trying to replicate this would need to go deep into the codebook details. Section 3 (if structured conventionally) likely needs the most work.

Verdict: weak accept — The recovery gap is a genuinely useful diagnostic concept that the field needs, and the single-counterexample framing is honest and appropriate. But the empirical contribution is thin (one construct, one corpus, one national model) and the lack of detail on the multilingual comparison model is a reproducibility concern.

Takeaways

Steal the recovery gap framework. If you’re using LLMs to annotate data for any theoretically-grounded construct—not just moral foundations, but framing, sentiment with causal dimensions, legal reasoning, medical diagnosis from narratives—decompose your rubric into atomic sub-criteria and test whether holistic performance survives decomposition. This is a five-minute check that could save you from building on sand.

Stop reporting agreement as sufficiency. If you’re benchmarking an LLM for annotation tasks, agreement with humans is necessary but not valid (ironic, given the paper’s point). Add at least one diagnostic that probes the reasoning path—decomposition, counterfactual perturbation, or explanation consistency checks.

National model builders: expand your benchmark batteries. If you’re evaluating a sovereign LLM, standard NLU benchmarks and human-agreement scores are not enough. You need construct-validity diagnostics specific to the tasks your model will actually perform. This paper shows one model passing surface-level checks while failing deeper ones.