Distilling the Change, Not the Champion: A Smarter Way to Transfer RL Improvements

Paper: 2607.05394 Authors: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou Categories: cs.LG, cs.AI, cs.CL

The Gap

Reinforcement Learning from Verifiable Rewards (RLVR) is now the go-to method for boosting language model reasoning. The problem? It’s computationally brutal, especially for the newest, strongest models. You have to make the target model itself generate tons of solution attempts (rollouts) during training. As models scale, this post-training step becomes the bottleneck. Existing “weak-to-strong” ideas tried to cheapen this by running RL on a small (weak) model and then having the strong model mimic the weak model’s final, post-RL version (direct distillation). But that’s flawed: the weak model’s final policy is a messy mix of genuine reasoning gains and the inherent limits of being small. The strong student ends up learning both the good and the bad.

Problem: RL on large models is expensive (rollout bottleneck).
Assumption: Maybe we can transfer knowledge from a smaller, cheaper RL run.
Prior Method (Flawed): Directly distill the weak model's FINAL policy.
         Weak Model (post-RL) -----> [Direct Imitation] -----> Strong Model
                                     (teaches strengths AND weaknesses)
Gap: This "bathwater with the baby" approach limits gains.
Proposed Method: Transfer the RL-Induced POLICY SHIFT, not the final policy.
Conclusion: The change in behavior (implicit reward signal) can be effectively and efficiently transferred.

The Increment

One sentence: Before this paper, reusing a weaker model’s RL training meant copying its flawed final answers; after this paper, we can copy the *improvement trajectory itself, which is a purer, more transferable signal.

Core Mechanism

The core insight of Direct On-Policy Distillation (Direct-OPD) is clever: the most valuable thing from an RL run isn’t the final model, but the direction of change it underwent. To capture this, the method uses a pair of checkpoints from the same weak model: one before RL training (the reference) and one after (the teacher). By comparing what the post-RL teacher does to what the pre-RL reference would have done in the same situation, we can isolate the effect of RL.

Concretely, for a given state (a partial solution step), the method calculates the log probability ratio: log(π_teacher(a|s) / π_reference(a|s)). This ratio is dense—it tells you for every possible action a whether RL made it more or less likely. This log-ratio is then used as an implicit reward signal for the strong student model. The student is trained to maximize this reward on its own generated states (on-policy). The student isn’t copying answers; it’s being guided to change its own behavior in the same direction the weak model’s RL pushed it.

Data Flow:
1. Weak Model Pre-RL (Reference) --> [π_ref(a|s)]
2. Weak Model Post-RL (Teacher) --> [π_teacher(a|s)]
3. For a state 's': Compute Reward = log(π_teacher(a|s) / π_ref(a|s))
4. Strong Student Model generates its own action 'a_student' at state 's'.
5. Student is updated via PPO to maximize this reward signal.

Structural Metaphor: The Piano Instructor Analogy

Imagine teaching someone piano (the strong student) using a video of a beginner (the weak model) who took an intensive course (RL). A naive approach (direct distillation) is to have the student memorize the beginner’s final recital—it’s better than before, but still limited by the beginner’s small hands and basic technique.

Direct-OPD is like a brilliant instructor who doesn’t show the final performance. Instead, they show you side-by-side footage of the beginner before and after the course, and they say: “Look, the course made the beginner much more expressive with the left hand here (the log-ratio is positive for that action), and it made them avoid this clumsy fingering there (the log-ratio is negative).” The strong student, with their own long fingers and base skill, then applies those same kinds of improvements to their own playing. They’re not copying the beginner’s limited expressiveness; they’re learning what kinds of expressiveness are worth developing based on the observable change in a novice.

Key Concepts

• Policy Shift as Implicit Reward: Forget training a separate reward model. The paper’s key concept is that the difference between a model’s behavior before and after RL *is itself a reward signal. Think of it like this: if a diet (RL) makes a food critic (weak model) suddenly love kale and hate doughnuts, the shift in their preferences (“kale is now +5, doughnuts are now -3”) is a useful guide for your own diet, even if you have different taste buds. You don’t copy their exact meals; you learn what changes are valuable.
• On-Policy Distillation: “On-policy” means the student learns from situations it actually encounters itself. The strong model isn’t learning from a static dataset of the weak model’s old solutions. It’s generating its own thought process, and at each step, it gets feedback: “Would the weak model’s RL training make this step more or less likely?” This keeps the learning grounded in the student’s own capabilities and knowledge state, which is crucial for effective transfer.

Framework Shift

Before (mainstream approach):        After (this paper):
Teach the final performance.         Teach the improvement trajectory.
                                     
Weak Model (post-RL)                 Weak Model (post-RL)
    |                                     |
    v                                     v
[Final Policy] ----Imitation----> [Strong Model]   [Policy Shift (Delta)]
    (Static snapshot)                 (Inflexible)        |
                                                          v
                                                   [Strong Model]
                                                    (Adapted growth)

From teaching outcomes to teaching the process of improvement, the core shift is treating the RL run not as a model to clone, but as a reusable supervision signal that describes how to get better.

Expert Assessment

Problem choice: Excellent. This addresses a real, growing pain in the field. As RL post-training becomes standard and models get larger, the computational cost of running RL on frontier models is becoming prohibitive. Finding ways to reuse computation is a high-impact direction. This isn’t manufactured; it’s a pressing practical bottleneck.

Method maturity: Very clever. The elegance is in its simplicity and directness. Instead of adding complex new architectures (like a separate reward model), it derives a reward signal directly from existing model snapshots. It’s a “hack” in the best sense—an insightful observation (the log-ratio of checkpoints is a good signal) that leads to a simple, effective algorithm. It’s not brute force; it’s smart reuse.

Experimental integrity: The results are compelling but narrowly focused. The headline result (boosting Qwen3-1.7B on AIME 2024) is strong and the comparison to step-matched RL is fair. The sequential composition result (stacking shifts from multiple teachers) is particularly interesting and shows the method’s flexibility. However, the experiments are limited to mathematical reasoning benchmarks. It would be reassuring to see this validated on other domains like code generation or complex instruction following. The scale is modest (up to 1.7B/7B models), which is fine for a proof-of-concept but leaves open how well it transfers across much larger gaps (e.g., 7B to 70B).

Writing quality: The paper is generally clear and well-structured. The core idea is communicated effectively early on. However, the related work section feels a bit light; a deeper dive into prior work on knowledge distillation and reward modeling could better position the novelty. The appendices seem essential for understanding the PPO implementation details, which suggests the main text might be slightly over-condensed.

Verdict: weak accept — A clever, practical, and timely contribution that solves a real problem with an elegant insight. Its main limitations are the narrow experimental domain and scale, which prevent a “strong accept” but don’t undermine the core value.

Takeaways

The most stealable idea here is the checkpoint-pair-as-reward-signal technique. This isn’t just for language models. Whenever you have a “before” and “after” snapshot of a system improved via some expensive process (RL, fine-tuning, architecture search), you can treat the *difference in their behaviors as a dense, implicit reward to guide a new, more capable system. This could transfer to robotics policy refinement, recommendation system tuning, or even automated scientific experiment design. The concrete technique: for any task, save your model before and after a costly improvement round. Then, use the log-ratio of their outputs as a reward signal to train a better, larger, or different system on its own generated data.