Bridging RL and SFT: Training Language Models with Verifiable Rewards Plus Human Style

Paper: 2607.01181 Authors: Mehul Damani, Isha Puri, Idan Shenfeld, Jacob Andreas Categories: cs.LG, cs.AI, cs.CL

The Gap

Here’s the state of play: RL with Verifiable Rewards (RLVR) has become the go-to for training language models on tasks where you can objectively check if the answer is right — think code that passes tests, or math proofs that verify. Methods like RLHF, RLEF, and direct RL on pass/fail signals work beautifully for accuracy.

But there’s a dirty secret. These methods optimize only what you can score with a verifier. They completely ignore the stuff that makes outputs feel human: style, structure, tone, conciseness, elegance. The result is well-documented failure modes — diversity collapse (the model finds one trick and repeats it), unnatural outputs (technically correct but reads like a robot), and reward hacking (the model games the verifier). Supervised fine-tuning (SFT) on human demonstrations captures style beautifully but doesn’t push the model beyond the demonstrated frontier for accuracy. You’re stuck choosing: optimize for correctness *or optimize for human-likeness.

This paper asks: what if you don’t have to choose? The path forward is to learn a proxy for the human output distribution — a discriminator — and use it as an additional reward signal alongside verifiable rewards. The discriminator is adversarially trained against the generator, so it keeps providing gradient signal even as the generator improves.

RLVR (prior approach):
  Task with verifiable reward --> RL optimizer --> Model
       |                                             |
       +--> maximize accuracy -----> style degrades  |
                                    diversity drops  |
                                    reward hacking   |
                                                    v
                                               Problem

This paper:
  Task reward + Discriminator reward --> RL optimizer --> Model
       |                                      ^               |
       |                                      |               |
       +--> maximize accuracy                 |    maximize human-likeness
       +--> maximize human-likeness           |               |
                                              +-- Discriminator <--+
                                              |   (trained on      |
                                              |    human demos)    |
                                              +--------------------+

Logic path:
Problem:  RLVR optimizes verifiable rewards, ignores style/structure
          |
Assume:   Human demonstrations encode non-verifiable quality signals
          |
Method:   Adversarial generator-discriminator adds learned human-proxy reward
          |
Evidence: Bug fix (lower edit dist), story gen (higher win rate), less hacking
          |
Conclusion: You can jointly optimize verifiable AND non-verifiable properties

The Increment

One sentence: Before this paper, you had to choose between RL’s accuracy gains and SFT’s human-like style; after this paper, a single training framework gives you both simultaneously.

Core Mechanism

The system has two components trained together: a generator (the language model being optimized) and a discriminator (a learned judge of human-likeness). The generator produces outputs for task prompts. These outputs are scored two ways: first, by a verifiable reward (does the code pass tests? is the math correct?), and second, by the discriminator (how likely is this output to have been written by a human?). The generator is trained via RL to maximize a weighted combination of both signals.

The discriminator is trained simultaneously on a contrastive objective: given a prompt and two completions — one from a human demonstration dataset, one from the current generator — learn to tell them apart. As the generator improves and produces more human-like outputs, the discriminator must sharpen its judgment, creating a co-evolutionary dynamic. The discriminator never needs explicit labels for “good style” — it implicitly captures everything that distinguishes human writing from model writing through the training distribution.

The adversarial setup is crucial for scalability. A fixed reward model would become obsolete as the generator distribution shifts. But because the discriminator trains alongside the generator on fresh samples, it continuously adapts, providing useful gradient signal throughout training. This is the key architectural difference from simply doing RLHF with a frozen reward model.

Generator (LM)                    Discriminator
    |                                  |
    |  prompt                          |
    v                                  |
 Generate output                      |
    |                                  |
    +-----> verifiable reward -------->|
    |         (task-specific)          |
    |                                  |
    +-----> discriminator score ------->|
    |         (human-likeness)         |
    |                                  |
    v                                  |
 RL update                            |
 (maximize both rewards)              |
    |                                  |
    |                                  |
    +--- human demo vs gen output --->|
                                      |
                              Contrastive training
                              (tell human from model)

Data flow:
  prompt --> [Generator] --> output
                |               |
          verifiable_r      disc_score
                |               |
                v               v
         combined_reward --> RL objective --> update Generator
                                      |
         human_demo vs output --> Discriminator objective --> update Discriminator

The Restaurant Critic Metaphor

Think of this like training a young chef through a restaurant feedback system.

The verifiable reward is the health inspector — a binary, objective check. Does the food pass food safety standards? Is the dish technically correct? This is easy to measure and gives the chef a clear floor to hit. But a chef who *only optimizes for passing health inspections will produce bland, safe, uninspired food. You’ve seen this in RLVR models: technically correct but soulless.

The discriminator is a food critic who has eaten at the best restaurants in the city. The critic doesn’t have a checklist of “good dishes” — instead, they’ve internalized thousands of meals and can feel when something tastes *restaurant-quality versus cafeteria-quality. You give the critic one dish from a world-class chef and one from your trainee, and they learn to tell them apart. The trainee chef then tries to fool both the health inspector AND the critic.

As the trainee improves, the critic has to sharpen their palate too — they can’t just rely on obvious tells anymore. The result: the chef learns to pass health inspections (verifiable reward) while also producing food that feels refined and intentional (discriminator reward). The two feedback loops co-evolve, pushing the chef toward food that is both safe AND delicious.

Without the critic, you get technically edible food nobody wants to eat. Without the health inspector, you get creative food that might make people sick. The insight is that you need both, training simultaneously, so neither signal goes stale.

Key Concepts

• RL with Verifiable Rewards (RLVR): Imagine you’re training a student to solve math problems. RLVR is like giving them a calculator to check their own work — right or wrong, instant feedback, no ambiguity. It works great because the feedback is perfectly reliable. But now imagine training the same student to write essays. There’s no calculator for “good writing.” RLVR can only optimize what a verifier can check, so it’s powerful but narrow. This paper’s insight is that RLVR leaves a huge amount of quality on the table — style, structure, naturalness — that verifiers simply can’t see.

• Adversarial co-training: Think of two kids in a game: one draws pictures, the other tries to guess if a picture was drawn by the kid or by a professional artist. At first the guesser is terrible — the kid’s drawings are obviously amateur. But as the kid practices and improves, the guesser has to notice subtler tells (wrong proportions in hands, awkward shading). The guesser getting better forces the kid to improve faster, and vice versa. Neither one stops learning because the other keeps raising the bar. That’s exactly what happens between the generator and discriminator here — they create a self-improving feedback loop that neither a static reward model nor a fixed SFT dataset can provide.

• Non-verifiable properties: These are qualities where reasonable humans might disagree on a score but can still feel the difference. “Is this story well-written?” doesn’t have a clean number. But if you show people two stories, they can reliably say which one reads better. The discriminator learns these implicit preferences from raw examples — no annotation, no rubric, just human-written examples as the gold standard.

Framework Shift

Before (mainstream):                  After (this paper):

  Prompt                              Prompt
    |                                   |
    v                                   v
Generator                         Generator
    |                                   |
    v                                   v
Verifier ----> scalar reward       Verifier ----> scalar reward (task)
    |                                   |
    v                                   |
RL update                             Discriminator --> adversarial reward
(accuracy only)                           ^                |
    |                                     |                v
    v                                     |          combined RL update
Style degrades                    Human demos ---+     (task + human-likeness)
Reward hacking occurs                                 |
    |                                                  v
    v                                          Style preserved
Diversity collapses                       Diversity maintained
                                          Reward hacking suppressed

From optimizing verifiable scores alone to jointly optimizing verifiable scores and human-likeness through adversarial co-training, the core shift is treating “sounds human” as a learnable objective rather than an unsolvable problem.

Expert Assessment

Problem choice: This is a real gap. The tension between RL accuracy gains and SFT style preservation is something every practitioner building with RLVR runs into. The failure modes — diversity collapse, reward hacking, robotic-sounding outputs — are not theoretical; they’re daily headaches. The paper positions itself at a natural inflection point where RLVR is becoming dominant but its limitations are becoming painful. Good instinct.

Method maturity: The approach is conceptually clean — it’s essentially a GAN objective layered onto RLVR, which is not a deep theoretical contribution but a pragmatic architectural one. One could argue simpler alternatives exist: just add a KL penalty to a reference model, or do alternating SFT/RL rounds. The paper addresses KL-penalty baselines and shows they underperform, which is important. However, the adversarial training adds significant complexity (hyperparameters for discriminator capacity, training frequency, reward weighting) and the paper doesn’t do a thorough ablation on these. It’s clever but not the only path — and the adversarial instability issues from GANs are largely hand-waved away.

Experimental integrity: The experiments span three diverse domains (bug fixing, story generation, reward hacking), which is good coverage. Baselines include vanilla RLVR, SFT, and KL-regularized approaches. The bug-fixing results (lower edit distance with matching accuracy) and story generation results (higher win rate) are compelling. The reward hacking benchmark is a smart choice — it’s a clean, interpretable test. However, the reward hacking experiment uses a somewhat toy setup, and the story generation evaluation relies on LLM-as-judge win rates, which is standard but carries known biases. I’d want to see human evaluations for the story generation task before fully trusting those numbers. Also, the paper doesn’t discuss training stability in depth — GAN-style training can be notoriously finicky, and the paper seems to get lucky with convergence rather than providing robustness guarantees.

Writing quality: The paper is well-structured and readable. The framing of “bridging RL and SFT” is clean and memorable. However, the related work section is thin — the connection to prior work on discriminator-based rewards (like earlier GAN-for-text papers) deserves more thorough treatment. The limitations section is also undercooked; they should acknowledge more directly that the discriminator’s quality depends heavily on the human demonstration data, and that distribution shift between the demonstration domain and the target task could be a real problem.

Verdict: weak accept — The idea is intuitive and the results are solid across multiple domains, but the method is more of a clever composition than a deep insight, and the evaluation has some soft spots (LLM-as-judge, toy reward hacking, limited ablations).

Takeaways

Three things you can steal right now:

1. The discriminator-as-style-proxy pattern: If you’re doing RLVR on any task and your outputs feel “off” even when they’re correct, train a small discriminator on human-written examples of good outputs. Use its score as a secondary reward signal. You don’t need this paper’s full framework — even a frozen discriminator fine-tuned on a few hundred human examples can provide meaningful style gradient.

2. The “bridge RL and SFT” framing: Stop thinking of RL and SFT as separate training phases you alternate between. This paper shows you can get both signals in a single training loop. If you’re currently doing SFT-then-RL pipelines, consider whether you can get the SFT signal through a learned reward rather than through direct supervision. This can be cleaner architecturally.

3. Co-training prevents stale rewards: If you’re using any learned reward model, training it alongside your policy (rather than freezing it) prevents the well-known problem of the model exploiting a fixed reward model’s blind spots. This isn’t new (it’s classic GAN thinking), but this paper demonstrates it cleanly in the LM RLVR setting.