When Teachers Fail Students: The Hidden Dynamics of On-Policy Distillation

Paper: 2604.13016 Authors: Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu Categories: cs.LG, cs.AI, cs.CL

The Gap

On-policy distillation (OPD) has become the workhorse of LLM post-training—think of it as having a student model learn by generating its own attempts and getting feedback from a teacher. It’s everywhere: RLHF, rejection sampling, iterative refinement. But here’s the puzzle: sometimes a stronger teacher makes the student worse. Practitioners know this empirically, but nobody understood why.

Prior work treated OPD as a black box optimization problem. If the teacher scores higher, distillation should work—that was the assumption. When it failed, people blamed hyperparameters or data quality. This paper asks a more fundamental question: what conditions must hold for OPD to succeed at all?

Empirical Mystery                    Mechanistic Investigation
     |                                        |
     v                                        v
Teacher stronger     ------>     Two necessary conditions:
but student worse                1. Compatible thinking patterns
     |                           2. Genuine new capabilities
     v                                        |
Blame hyperparameters            Token-level dynamics analysis
or data quality                              |
     |                                        v
     v                           Practical recovery strategies
No principled fix    ------>     + Off-policy cold start
                                 + Teacher-aligned prompts

The Increment

One sentence: Before this paper, OPD failures were mysterious hyperparameter issues; after, we know they stem from distributional mismatch and can be diagnosed via token-level alignment patterns.

Core Mechanism

OPD works by having the student generate responses to prompts, then using a teacher to score those responses as training signal. The student updates toward higher-scoring outputs. Sounds simple, but the devil is in the distribution.

The paper identifies the core issue: the student only visits states (partial generations) it would naturally produce. If the teacher’s high-scoring paths require thinking patterns the student never explores, those paths are invisible. It’s like trying to teach someone chess by only commenting on their actual games—if they never consider certain opening moves, you can’t teach those strategies.

The mechanism breaks down into three stages: (1) student generates from its current policy, creating a distribution of visited states; (2) teacher scores these generations; (3) student updates toward higher scores. Success requires the teacher’s preferred paths to overlap with student-visited states AND offer genuinely new signal beyond what the student already knows.

Student Policy (current)
        |
        v
    [Generate]  -----> Visited States (S_student)
        |                      |
        |                      v
        |              Teacher Scores
        |                      |
        v                      v
    Update  <------  High-scoring paths in S_student
        |
        v
Student Policy (next iteration)

Key insight: If teacher's best paths lie outside S_student,
             they're invisible to the update process.

Think of it like a hiking guide teaching you a mountain trail. The guide (teacher) knows an amazing scenic route, but you (student) naturally walk a different path. The guide can only give you feedback on the trail you’re actually walking. If your path never intersects the scenic route, you’ll never discover it—no matter how enthusiastically the guide describes it. OPD only works when your natural exploration overlaps with the teacher’s knowledge, AND the teacher shows you something new along that shared path.

Key Concepts

• Distributional Indistinguishability: Two models are distributionally indistinguishable from a student’s perspective if, when the student samples from its own policy, it can’t tell which model scored the outputs. Formally, if a 1.5B and 7B teacher both give similar scores to student-generated text, they’re indistinguishable—even though the 7B model is objectively stronger. This explains why bigger teachers sometimes don’t help: they’re better at tasks the student never attempts. It’s like having a calculus expert grade your arithmetic homework—their advanced knowledge doesn’t matter if you’re not doing calculus problems.

• High-Probability Token Alignment: Successful OPD shows a specific signature: the student progressively aligns with the teacher on high-probability tokens (those the student already considers likely). The paper finds 97-99% of probability mass concentrates on a small shared token set. This is counterintuitive—you’d think learning happens on low-probability tokens where student and teacher disagree. But OPD works by reinforcing the student’s existing intuitions that happen to align with teacher preferences, not by forcing exploration of unlikely paths.

• Off-Policy Cold Start: When OPD fails due to distribution mismatch, you can bootstrap it by first training on teacher-generated data (off-policy), then switching to student-generated data (on-policy). The off-policy phase shifts the student’s distribution closer to the teacher’s, creating overlap. It’s like learning a language: you start by mimicking native speakers (off-policy) before you can have conversations where they correct your own attempts (on-policy).

Framework Shift

Before (mainstream approach):        After (this paper):

Teacher (stronger model)             Teacher (stronger model)
        |                                    |
        v                                    v
    Score outputs                    Check distribution overlap
        |                                    |
        v                                    |
Student learns from                          +---> Overlap? ---> OPD works
high scores                                  |         |
        |                                    |         v
        v                                    |    No overlap? ---> OPD fails
Better teacher                               |         |
= better student                             |         v
                                             |    Recovery strategies:
(Assumption: teacher                         |    - Off-policy cold start
 quality is sufficient)                      +---> - Aligned prompt selection

                                        (Diagnosis: distribution
                                         mismatch is the root cause)

From “stronger teacher should work” to “teacher and student must share visited states,” the core shift is recognizing OPD as a distribution matching problem, not just a scoring problem.

Expert Assessment

Problem choice: This is a real gap. OPD is production-critical (used in GPT-4, Claude, etc.), and practitioners have war stories about mysterious failures. The paper doesn’t manufacture the problem—it explains existing pain points. It sits at a sweet spot: fundamental enough to matter broadly, specific enough to yield actionable insights.

Method maturity: The investigation is more diagnostic than algorithmic, which is appropriate. The weak-to-strong reverse distillation experiment is clever—showing that 1.5B and 7B teachers are indistinguishable proves the point cleanly. The token-level probing is solid. However, the proposed fixes (off-policy cold start, prompt selection) are somewhat obvious once you understand the problem. The paper’s value is in the diagnosis, not the cure.

Experimental integrity: Baselines are fair. The distributional indistinguishability claim is well-supported by multiple metrics (KL divergence, reward correlation). One concern: the experiments focus on relatively short-form tasks (math, coding). The “long-horizon” caveat in the abstract hints at limitations the paper doesn’t fully explore. The 97-99% probability mass concentration finding is striking but needs more investigation—is this universal or task-dependent?

Writing quality: The phenomenology section (Section 3) is excellent—clear examples of when OPD fails. The mechanism section (Section 4) gets dense with notation that could be simplified. The paper would benefit from reorganizing: put the two conditions (compatible thinking + new capabilities) upfront as the main claim, then use experiments to validate each. The current structure buries the key insight in the middle.

Verdict: weak accept — Solid diagnostic work on a real problem, but the solutions are incremental and the scope is narrower than the abstract suggests.

Takeaways

For practitioners: Before scaling up OPD, run a quick distributional check—sample from your student, score with your teacher, and see if the teacher’s preferences correlate with student probability. If not, you’re wasting compute. Use off-policy warm-up (even just 10-20% of training) to bridge the gap.

For researchers: The “high-probability token alignment” finding is underexplored. If OPD only reinforces existing intuitions, it can’t teach fundamentally new reasoning patterns. This suggests a hard limit on what OPD can achieve—you might need hybrid approaches that force exploration (like curiosity-driven RL) for genuine capability jumps.

Transferable technique: The weak-to-strong reverse distillation setup (using weaker models as teachers to prove a point) is a clever experimental design. It’s a general tool for testing whether your training signal actually contains new information or just confirms what the model already knows.