ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM Alignment

Paper: 2603.23184 Authors: Hao Wang, Haocheng Yang, Licheng Pan, Lei Shen, Xiaoxi Li, Yinuo Wang, Zhichao Chen, Yuan Lu, Haoxuan Li, Zhouchen Lin Categories: cs.CL

Abstract

Reward modeling is a long-standing challenge in RLHF for aligning language models. Current methods rely heavily on expensive explicit preference data (human annotators choosing between responses). ImplicitRM studies implicit reward modeling — learning from implicit human feedback such as clicks and copies — as a cost-effective alternative. Two fundamental challenges are identified: (1) implicit data lacks definitive negative samples (no-feedback ≠ negative), and (2) user preference bias where different responses have different propensities to elicit feedback. ImplicitRM addresses both by stratifying training samples into four latent groups via likelihood maximization, with provably unbiased learning objectives.

Key Contributions

• Formal problem definition: First principled formulation of implicit reward modeling for RLHF
• Four-group stratification: Decomposes samples into positive-active, negative-active, positive-passive, negative-passive
• Provably unbiased objective: The learning objective is theoretically shown to be an unbiased estimator of the ideal explicit-preference objective
• Model-agnostic: Works across diverse LLMs without requiring explicit preference labels

The Implicit Feedback Problem

Why Explicit Preferences Are Expensive

Standard RLHF requires annotators to compare two responses and choose which is better. This is:

• Labor-intensive and costly
• Hard to scale
• A bottleneck for widespread deployment

Why Implicit Feedback Is Tricky

Implicit signals (clicks, copies, dialogue continuation) are naturally abundant but present two unique challenges:

Challenge 1: No Definitive Negatives

User Action	True Preference	Observed Signal
Copies response	Positive ✓	Feedback (r=1)
Doesn’t copy, dissatisfied	Negative ✗	No feedback (r=0)
Doesn’t copy, but satisfied	Positive ✓	No feedback (r=0)

No-feedback includes both genuinely negative AND satisfied-but-passive users. Standard positive-negative classification methods fail.

Challenge 2: User Preference Bias

Users readily copy good answers in knowledge QA tasks but rarely copy responses in open dialogue regardless of satisfaction. This means the probability of getting feedback varies by response type, not just quality.

The Four-Group Stratification

ImplicitRM models users as falling into four latent groups:

1. Positive-Active: Satisfied user who gives feedback (e.g., copies a good answer)
2. Negative-Active: Dissatisfied user who gives feedback (rare but possible — e.g., copying to show someone what’s wrong)
3. Positive-Passive: Satisfied user who doesn’t give feedback (reads useful response but doesn’t copy)
4. Negative-Passive: Dissatisfied user who doesn’t give feedback (wants to end interaction)

A stratification model estimates each sample’s probability of belonging to each group. The learning objective is then derived via likelihood maximization over these four groups.

Theoretical Guarantee

The proposed objective is proven to be an unbiased estimator of the ideal objective computed on definitive positive and negative samples. This means:

• Even though we never observe true negative labels, the learned reward model converges to the same solution as if we had them
• User preference bias is automatically accounted for in the stratification

Results

Experiments demonstrate that ImplicitRM:

• Learns accurate reward models from implicit preferences alone
• Improves performance across diverse LLMs and benchmark datasets
• Matches or approaches the quality of explicit-preference reward models at a fraction of the data collection cost

Takeaways

• Implicit feedback (clicks, copies, session length) is vastly more abundant than explicit preferences — making it the natural path to scalable RLHF
• The key insight is that “no feedback” is ambiguous and must be decomposed, not naively treated as negative
• Four-group stratification elegantly handles both the missing-negatives problem and user preference bias
• This enables reward modeling at production scale using signals that are already being collected