Three Models of RLHF Annotation: Extension, Evidence, and Authority

Paper: 2604.25895 Authors: Steve Coyne Categories: cs.CY, cs.AI, cs.CL

The Gap

RLHF has become the dominant method for aligning language models, but the field treats human annotators as a black box labeled “human feedback.” Existing work focuses on technical pipeline optimization—better reward models, more efficient training, cleaner data collection—while leaving a foundational question unexamined: what normative role do annotators actually play? Are they executing the designers’ vision, reporting objective facts, or exercising independent judgment?

This ambiguity isn’t just philosophical. It creates concrete problems: pipelines designed for one role fail when annotators behave according to another, disagreement is handled inconsistently (sometimes averaged away, sometimes treated as signal), and validation criteria remain ad hoc. The gap is conceptual infrastructure—we’re building increasingly sophisticated systems on an undefined foundation.

Problem: RLHF treats annotators as undifferentiated "feedback"
   |
   v
Assumption: Different annotation tasks require different normative models
   |
   v
Method: Distinguish three models (extension/evidence/authority)
        + Map each to appropriate pipeline design choices
   |
   v
Evidence: Survey of landmark papers shows implicit model mixing
   |
   v
Conclusion: Decompose annotation dimensions, tailor pipelines per model

The Increment

One sentence: Before this paper, RLHF pipelines treated all human judgments uniformly; after, we have a conceptual framework that distinguishes three normative roles and prescribes different handling for each.

Core Mechanism

The paper doesn’t propose a new algorithm—it provides a taxonomy. The core mechanism is conceptual decomposition: take the monolithic notion of “human feedback” and split it into three models based on the annotator’s normative relationship to the output.

Extension model: Annotators are proxies for system designers. They’re trained to internalize and apply the designers’ preferences. Think of them as additional hands executing a centralized vision. Disagreement here is noise—it means training was incomplete or instructions were unclear. The pipeline should maximize inter-annotator agreement through detailed guidelines, calibration sessions, and quality control. Aggregation is straightforward: majority vote or consensus, with outliers flagged for retraining.

Evidence model: Annotators are measurement instruments reporting on external facts. These facts might be linguistic (is this grammatical?), social (would this be considered offensive?), or even moral (does this violate a widely-held norm?). Disagreement here is signal—it reveals uncertainty or genuine variation in the phenomenon being measured. The pipeline should sample diverse annotators to capture this variation, validate against ground truth where available, and aggregate using statistical methods that preserve uncertainty (weighted averaging, confidence intervals).

Authority model: Annotators are representatives of the user population, exercising legitimate judgment about what the system should do. They’re not implementing someone else’s vision or reporting facts—they’re making decisions. Disagreement here is fundamental—different stakeholders have different values. The pipeline should ensure demographic representativeness, make the decision-making process transparent, and aggregate using democratic methods (voting, deliberation) rather than statistical averaging.

Extension Model:          Evidence Model:           Authority Model:
                                                    
Designer                  Phenomenon                Population
   |                         |                         |
   v                         v                         v
Guidelines                Sampling                 Selection
   |                         |                         |
   v                         v                         v
Annotators                Annotators               Annotators
(proxies)                 (instruments)            (representatives)
   |                         |                         |
   v                         v                         v
Consensus                 Distribution             Democratic
(noise reduction)         (capture variance)       (value aggregation)

Think of it like three different ways to use a focus group. Extension: You’re a product designer testing whether your team correctly implemented your vision—you want the focus group trained on your design principles, and you’re looking for unanimous “yes, this matches the brief.” Evidence: You’re a market researcher measuring consumer preferences—you want a representative sample, and you’re mapping the distribution of opinions. Authority: You’re a community deciding what to build—the focus group has decision-making power, and you’re running a vote or deliberation process.

Key Concepts

• Normative role: This isn’t about what annotators do (click buttons, write text) but what justifies using their judgments. In the extension model, justification comes from fidelity to designer intent. In the evidence model, it comes from measurement validity. In the authority model, it comes from representativeness and legitimacy. The same annotation task (e.g., “rate this response”) can have different normative roles depending on why we care about the rating. If we care because it reflects the designer’s quality standards, that’s extension. If we care because it measures user satisfaction, that’s evidence. If we care because users have a right to shape the system, that’s authority.

• Disagreement interpretation: The three models give opposite prescriptions for handling disagreement. Extension treats it as error (fix through training). Evidence treats it as data (preserve through statistical methods). Authority treats it as legitimate pluralism (resolve through democratic process). Most RLHF papers implicitly choose one interpretation without justifying why. This matters because the choice determines whether you average away minority views, weight them by confidence, or give them proportional representation.

• Validation criteria: How do you know if your annotation pipeline is working? Extension model: high inter-annotator agreement, alignment with designer judgments. Evidence model: correlation with external ground truth, test-retest reliability. Authority model: demographic representativeness, procedural fairness. These criteria can conflict—a highly trained, homogeneous annotator pool (good for extension) is terrible for authority. A demographically representative pool (good for authority) may have low agreement (bad for extension).

Framework Shift

Before (mainstream approach):        After (this paper):

Human Feedback                       Extension    Evidence    Authority
      |                                 |            |            |
      v                                 v            v            v
  Annotation                        Consensus   Distribution  Democratic
      |                                 |            |            |
      v                                 v            v            v
  Aggregation                       Designer    Measurement  Stakeholder
  (one-size)                         Intent       Validity   Legitimacy
      |                                 |            |            |
      v                                 +------------+------------+
 Reward Model                                      |
                                                   v
                                            Reward Model
                                          (dimension-specific)

From monolithic “human feedback” to decomposed annotation dimensions, the core shift is recognizing that different aspects of model behavior require different normative justifications and therefore different pipeline designs.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The RLHF literature has been remarkably unreflective about the normative foundations of human feedback. The problem sits at a critical juncture—as these systems move from research to deployment, the “who decides what the model should do?” question becomes unavoidable. The timing is right.

Method maturity: This is conceptual work, not empirical, so “method” means the taxonomy itself. The three-way distinction is clean and well-motivated, drawing on established frameworks from political philosophy and measurement theory. The weakness is that it’s a first cut—there are likely hybrid cases or additional models (e.g., annotators as adversarial red-teamers). The paper acknowledges this but doesn’t fully explore the boundaries.

Experimental integrity: No experiments—this is a conceptual paper. The “evidence” is a literature survey showing how existing papers implicitly mix models. The survey is illustrative rather than systematic, which is fine for a position paper but limits the empirical claims. The author could have strengthened this by analyzing a larger corpus or conducting interviews with RLHF practitioners.

Writing quality: The paper is clear and well-structured, but the middle section (surveying existing work) drags. The failure modes section is the most valuable—concrete examples of what goes wrong when models are conflated—but it’s buried. Rewriting to lead with failure modes, then introduce the taxonomy as a solution, would make the argument more compelling. The normative criteria section (how to choose among models) feels underdeveloped—it’s where the rubber meets the road, but it’s only two pages.

Verdict: weak accept — Addresses an important conceptual gap with a useful framework, but the analysis could be deeper and the prescriptive guidance more concrete.

Takeaways

For practitioners building RLHF pipelines: Don’t use a single annotation protocol for all dimensions. Decompose your task. Safety judgments (is this harmful?) might be evidence-based—you’re measuring against social norms. Style preferences (is this engaging?) might be authority-based—users have legitimate disagreement. Factual accuracy (is this correct?) is evidence-based but with ground truth. Design separate pipelines for each.

For researchers: The extension/evidence/authority distinction is a useful lens for analyzing existing work. When you read an RLHF paper, ask: which model is implicit in their annotation protocol? Does their aggregation method match? Many papers use evidence-model language (measuring preferences) but extension-model methods (training for agreement).

For policy discussions: The authority model provides vocabulary for the “who should control AI?” debate. If you believe users should have a say in model behavior, you need annotation pipelines designed for representativeness and democratic aggregation, not just accuracy or efficiency. This has implications for how companies should structure their feedback processes.

Transferable technique: The “decompose by normative role” approach applies beyond RLHF. Any system that aggregates human judgments—content moderation, recommendation systems, search ranking—faces the same question: are judges implementing a policy, measuring a phenomenon, or exercising authority? The answer determines how you should recruit, train, and aggregate.