FIRM: Building Reward Models That Actually Know What They're Scoring

Paper: 2603.12247 Authors: Xiangyu Zhao, Peiyuan Zhang, Junming Lin, Tianhao Liang, Yuchen Duan, Shengyuan Ding, Changyao Tian, Yuhang Zang, Junchi Yan, Xue Yang Categories: cs.CV

The Gap

The RL-for-vision pipeline has a dirty secret: the reward model is usually an afterthought. Prior work like InstructPix2Pix, RLHF-Diffusion, and various GRPO-based image editors all assume that if you have a VLM-based scorer (CLIP, ImageReward, GPT-4V prompting), it’s good enough to guide optimization. It isn’t.

The specific failure mode is hallucination in scoring. A reward model might give a high score to an edited image that looks plausible but ignores the instruction, or penalize a generation that’s actually correct because the background changed. These noisy signals don’t average out — they compound. The generator learns to game a broken critic, and you end up with a model that’s optimized for the wrong thing.

The gap: nobody has systematically addressed reward model robustness before throwing it into the RL loop. The assumption has been “better VLM = better reward model,” but that’s not how it works. A VLM trained for general understanding doesn’t automatically become a reliable judge of instruction following.

Problem: RL-based image editing uses unreliable reward models
         Reward hallucination -> generator learns wrong objective
    |
    v
Assumption: Reward models need explicit robustness training
            before being used for RL optimization
    |
    v
Method: FIRM - train reward model with contrastive pairs
        + adversarial negatives + calibration
    |
    v
Evidence: Improved correlation with human judgments
          Better downstream RL performance
    |
    v
Conclusion: Robust reward modeling is a prerequisite,
            not an afterthought

The Increment

One sentence: Before FIRM, reward models for image editing were trained like classifiers and used like judges — after it, they’re trained explicitly to be robust critics before entering the RL loop.

Core Mechanism

FIRM trains the reward model in three stages. First, contrastive learning on (instruction, original image, edited image) triplets where the model learns to score correct edits higher than incorrect ones. Second, adversarial negative mining where the system generates plausible-looking but instruction-violating edits and trains the reward model to reject them. Third, calibration to ensure the score distribution is well-behaved and doesn’t have pathological peaks that the generator can exploit.

The key insight is that a reward model needs to see failure modes during training, not just positive examples. Standard VLM training shows the model what good looks like; FIRM shows it what almost-good-but-actually-wrong looks like.

Training Pipeline:
  Instruction + Image pairs
        |
        v
  Generate edits (correct + adversarial negatives)
        |
        v
  Contrastive training: score(correct) > score(wrong)
        |
        v
  Calibration: smooth score distribution
        |
        v
  Robust Reward Model
        |
        v
  Use in RL loop for image editing

Think of it like training a wine judge. You don’t just have them taste good wines — you give them wines with subtle flaws (corked, oxidized, over-oaked) and train them to detect the difference. FIRM does the same for image editing: it trains the reward model on near-misses and plausible failures, not just clear successes.

Key Concepts

• Reward Hallucination: When a reward model assigns high scores to outputs that superficially match the instruction but miss the semantic intent. Example: instruction says “make the sky darker” but the model gives high scores to images where the sky is replaced with a different texture entirely. The reward model is fooled by surface similarity.

• Adversarial Negative Mining: Generating hard negative examples that look plausible but violate the instruction in subtle ways. These are used to train the reward model to be more discriminating. It’s the difference between training on random noise (easy negatives) versus training on carefully crafted near-misses (hard negatives).

• Calibration: Ensuring the reward model’s score distribution is well-behaved — no extreme peaks, reasonable dynamic range, scores that correlate with actual quality. Uncalibrated models might give 0.99 to everything decent and 0.01 to everything bad, which doesn’t give the RL algorithm useful gradients.

Framework Shift

Before (mainstream approach):        After (this paper):

  VLM (general purpose)                VLM (general purpose)
        |                                    |
        v                                    v
  Use as reward model                  FIRM training:
  (no adaptation)                      - Contrastive pairs
        |                              - Adversarial negatives
        v                              - Calibration
  RL optimization                            |
        |                                    v
  Generator learns to                  Robust reward model
  game broken critic                         |
                                             v
                                       RL optimization
                                             |
                                             v
                                       Generator learns
                                       correct objective

From using VLMs as-is to explicitly training robust critics, the core shift is: reward model quality is a first-class concern, not an assumed property.

Expert Assessment

Problem choice: Real and well-motivated. Reward model reliability is a known pain point in RL-for-vision, and the failure modes described (hallucination, gaming) are documented in practice. The timing is right — as RL becomes more common for image editing, the reward model bottleneck becomes more visible.

Method maturity: The three-stage training (contrastive + adversarial + calibration) is solid engineering but not deeply novel. Each component exists in prior work; the contribution is the integration and the focus on reward models specifically. The adversarial negative mining is the most interesting piece.

Experimental integrity: Would need to see the full paper for baselines and metrics. The key question is whether the improved reward model actually translates to better downstream RL performance, not just better correlation with human judgments. Correlation is necessary but not sufficient.

Writing quality: The abstract and introduction (based on what’s visible) are clear and well-motivated. The problem framing is strong. Would want to see more detail on the adversarial negative generation process.

Verdict: weak accept — addresses a real problem with solid engineering, but the novelty is in the application rather than the method.

Takeaways

The core lesson: if you’re using a model as a reward signal for RL, train it explicitly for that role. Don’t assume a model trained for task A will be a good judge for task B just because they’re related.

The adversarial negative mining pattern is broadly applicable: whenever you’re training a discriminator or critic, include hard negatives that are plausible but wrong, not just random noise.

The calibration step is often overlooked but critical for RL: the reward model’s score distribution directly affects the RL algorithm’s behavior. A poorly calibrated reward model can make RL training unstable or ineffective.