RP-OPSD: Distilling Only Where the Reasoning Turns

Paper: 2608.06347 Authors: Xinye Wang, Junxiao Liu, Shujian Huang Categories: cs.CL

The Gap

Every frontier reasoning model has the same embarrassing asymmetry: ask it a competition math problem in English and it thinks carefully; ask the same problem in Swahili or Bengali and the chain of thought gets shorter, shakier, and often silently switches languages mid-solution. The capability exists inside the model; it just doesn’t route through non-English inputs reliably.

The prior work splits into three families. Translate-then-SFT takes English chains of thought, machine-translates them, and fine-tunes — cheap, but off-policy: the student learns to imitate sequences it would never have produced itself, so at inference time it walks off the training distribution immediately (classic exposure bias). English-pivot prompting tells the model to translate internally, reason in English, then answer — this works, but it pushes cost to test time and never actually internalizes target-language reasoning. On-policy self-distillation (OPSD) and its relatives (GKD-style objectives, self-distillation with privileged context) fixed the distribution mismatch: let the *student generate the rollout in the target language, then have a teacher — often the same model given privileged information like an English reference solution — score every token of that rollout and minimize a token-level divergence. Dense supervision, on-policy samples, no reward-model needed.

The gap this paper names is narrow but real: the OPSD objective weights every token equally. A multilingual rollout is mostly surface realization — inflections, connectives, particles, target-language phrasing of arithmetic that is already decided. A small minority of tokens are places where the reasoning actually *forks: choosing to set up an equation rather than enumerate cases, deciding to check a boundary condition, updating a running quantity from the problem’s givens. Flat KL spends most of its gradient budget teaching the student to say things in the target language a particular way, when the thing that fails to transfer is the decision-making. The paper’s move: define those forks operationally, and reweight distillation toward them.

[Problem] LLM reasons well in EN, degrades in low-resource langs
     |
     v
[Prior art]
   translate + SFT ..... off-policy, exposure bias
   EN-pivot prompting .. test-time cost, nothing internalized
   OPSD / GKD .......... on-policy, dense token KL  <-- best so far
                         BUT every token weighted equally
     |
     v
[Assumption] target-lang reasoning = surface text + reasoning pivots
             cross-lingual transfer bottlenecks at the pivots only
     |
     v
[Method] RP-OPSD
   two matched teacher views: (q, y_<t) vs (q, EN_ref, y_<t)
   their divergence == pivot score
   pivot score -> weights privileged KL + gates reference anchoring
     |
     v
[Evidence] math reasoning, 17 languages, several difficulty tiers
   > strong multilingual baselines and > OPSD variants
   learned weights land on control tokens + state-update tokens
     |
     v
[Conclusion] *where* you apply privileged supervision matters
             more than how much of it you apply

The Increment

One sentence: Before, on-policy self-distillation for multilingual reasoning spread privileged supervision uniformly across a rollout; after, you get a cheap, label-free way to *locate the tokens where privileged information actually changes the decision, and to spend your distillation budget there.

Core Mechanism

Start with the OPSD skeleton. The student, prompted with a problem in some target language, samples a full solution. That rollout is fixed, and a teacher is asked to evaluate it token by token. The trick that makes this “privileged” is that the teacher is given something the student does not have at inference time — here, an English reference solution to the same problem. The teacher’s next-token distribution is therefore better-informed than anything the student could produce on its own, and minimizing divergence to it is a form of supervision that’s denser than a scalar reward and better-aligned than translated ground truth.

RP-OPSD’s addition is a second, matched teacher pass. Same teacher, same student rollout prefix, same problem — the only difference is whether the English reference solution is in context. Run both. At each position you now have two distributions: p_B (with reference) and p_A (without). Where they agree, the English reference told the teacher nothing it didn’t already know at that position — that position is surface realization, grammatical bookkeeping, or an already-determined continuation. Where they diverge sharply, the reference is actively steering the next decision. That divergence is the paper’s operational proxy for a reasoning pivot. It’s a counterfactual measure of privileged-information influence, computed with two forward passes and no gradients.

The pivot score then does two jobs. First, it weights the privileged distillation term: the token-level divergence between the student and the privileged teacher is scaled up at pivots and down elsewhere, so the gradient concentrates on reasoning-control decisions instead of morphology. Second, it gates reference anchoring: rather than conditioning on the English reference uniformly (which risks the student learning to lean on a crutch that vanishes at test time, or worse, drifting toward English surface form), the reference’s influence is admitted mainly at the positions where it’s diagnostically relevant. The reported analysis is the sanity check that this is doing what it says: high weights fall on reasoning-control tokens (“so”, “instead”, “check whether”) and on problem-conditioned state updates (the numbers and operations that carry the problem’s specifics forward), while tokens that mainly serve surface realization get downweighted.

        problem q (target language)
                 |
                 v
        +--------------------+
        |  STUDENT (policy)  |  sample rollout y = y_1 .. y_T
        +--------------------+  on-policy, in target language
                 |
     +-----------+-----------------------+
     |                                   |
     v                                   v
+------------------+            +----------------------+
| TEACHER view A   |            | TEACHER view B       |
| ctx: q , y_<t    |            | ctx: q , EN_ref, y_<t|
| (no privilege)   |            | (privileged)         |
+------------------+            +----------------------+
     | p_A( . | y_<t )                   | p_B( . | y_<t )
     +-----------------+-----------------+
                       v
            d_t = D( p_B  ||  p_A )        .. pivot score
            "how much did the EN reference
             change the teacher's decision here?"
                       |
                       v
            w_t = f( d_t )   normalize / sharpen
                       |
     +-----------------+-----------------------+
     v                                         v
 sum_t  w_t * KL( p_B || student )      reference anchoring
 privileged distillation,               gated by w_t : pull in
 concentrated at pivots                 EN evidence only where
                                        it is decision-relevant
                       |
                       v
                 update student

The metaphor: a driving lesson in an unfamiliar city, with two navigators.

The student is the one driving, and the route is the chain of thought. Riding along are two navigators who are the same person in two states of knowledge: one has already seen the route map (the English reference solution), one is navigating blind from the destination address alone. You ask both, continuously, “what now?”

On a long straight boulevard, both say “keep going.” The map added nothing — this stretch is surface realization: you still have to steer, stay in lane, signal, but no decision is being made. Then you reach an unmarked five-way intersection. The blind navigator shrugs and guesses straight; the map-holder says “hard left, then immediately right.” That disagreement is the signal. It doesn’t just mean “here is a hard spot” — it means here is a spot where privileged knowledge changes the outcome, which is exactly the definition of a place worth teaching.

RP-OPSD is the driving instructor who keeps hands off the wheel on the boulevard and grabs it firmly at the intersections. And the gating of reference anchoring is the instructor’s rule about the map itself: the learner doesn’t get to drive with the map open on their lap the whole way — that just trains map-reading, not city knowledge. The map comes out at intersections. Everywhere else, drive.

The metaphor is also where the method’s limitation is visible: the divergence detector fires at intersections, but it cannot tell an intersection where the map’s advice is correct from one where the map is being idiosyncratic. Disagreement is not the same as being right.

Key Concepts

• On-policy self-distillation: Two words doing separate work. “On-policy” means the training sequences come from the student’s own sampling, not from a fixed dataset — so the student is corrected on the mistakes it actually makes, in the regions of sequence space it actually visits. Think of grading a student’s own essay drafts rather than making them copy out a model essay: the model essay never tells them what to do after *their bad third paragraph. “Self” means the teacher is the same network (or a lightly modified version of it), so no bigger model is required; the teacher’s advantage comes purely from being given extra context. Together: the model generates, then a better-informed version of itself annotates every token of that generation.

• Privileged information: Anything the teacher sees that the student won’t have at test time. Here it’s an English reference solution. The point isn’t to copy the reference — the student must eventually solve problems with no reference at all — but to use it as a temporary oracle that makes the teacher’s token-level distributions trustworthy. It’s the same logic as a chess coach reviewing your game with an engine open: you can’t bring the engine to the tournament, but the annotations it enables are what you learn from.

• Reasoning pivot (and how to find one without labels): A pivot is a token where the trajectory of the reasoning is chosen rather than merely expressed. “Let” in “Let x be the number of…” is a pivot; the word after it usually isn’t. The genuinely nice idea in this paper is the detection method: don’t try to define pivots linguistically or annotate them by hand. Instead run the same model twice, differing only by the presence of privileged information, and measure where the two next-token distributions pull apart. Positions where the answer key changes your prediction *are the positions where thinking happens. Note the relationship to a nearby idea in RL-for-reasoning — high-entropy “forking tokens” — but with an important difference: entropy measures the model’s own uncertainty, whereas this measures the value of the answer. A token can be uncertain for boring reasons (three synonyms are equally fine), and this proxy correctly ignores those.

Framework Shift

Before (OPSD / GKD, mainstream):        After (RP-OPSD):

 student rollout                         student rollout
  y1  y2  y3  y4  y5  y6                  y1  y2  y3  y4  y5  y6
   |   |   |   |   |   |                   .   .   |   .   |   .
   v   v   v   v   v   v                   .   .   v   .   v   .
  [1] [1] [1] [1] [1] [1]                 .1  .1  .9  .2  .8  .1
   \   |   |   |   |   /                       \      |     /
    \  |   |   |   |  /                         \     |    /
     ONE teacher pass                            +--------+
     ( q + EN_ref )                              | weights from
          |                                      | DIFF of TWO
          v                                      | teacher views |
    flat token-level KL                          +---------------+
    budget spent mostly on                             |
    surface realization                                v
                                              KL concentrated on
                                              decision points only

 supervision = "match the teacher"     supervision = "match the teacher
                                        where the answer key mattered"

From uniform token-level imitation to counterfactually-localized supervision, the core shift is treating the influence of privileged information as the importance signal, so the objective learns reasoning control rather than target-language phrasing.

Expert Assessment

(Caveat: this reading is based on the abstract and stated analysis; I haven’t audited the tables, so the experimental critique below is about what the design must show, not a claim about what the numbers are.)

Problem choice: A real gap, honestly framed, and small. Multilingual reasoning transfer is a genuine deficiency with practical stakes, and “flat KL wastes capacity on surface form” is the kind of criticism that is obviously true once stated. But the field has been converging on token-importance weighting from several directions at once — entropy-based token selection in RLVR, token-level credit assignment in preference optimization, selective/focal knowledge distillation — so this sits comfortably inside a trend rather than opening a direction. The contribution is a well-chosen importance signal for a specific setting, not a new paradigm.

Method maturity: Clever, and cheap in the right way. The two-matched-views diff is the good idea here: it’s essentially context-attribution (the same machinery behind context-aware/contrastive decoding) repurposed as a supervision weight, it requires no annotation, no extra model, no reward function, and the second forward pass is inference-only and cacheable. It also has a clean semantics — “where does the answer key change the prediction” — that entropy-based selection lacks. The worry is the number of unpinned knobs: which divergence, how to sharpen or normalize the weights, what threshold gates reference anchoring. Papers in this genre usually have one temperature-like hyperparameter doing a lot of undisclosed load-bearing, and the abstract gives no reason to think this one is exempt. Simpler things a skeptic will demand: entropy-based weighting, teacher-confidence weighting, and plain top-k selection of the highest-divergence tokens (does soft weighting beat a hard mask?). If a hard mask on the top 20% works as well, the story gets simpler and stronger, not weaker.

Experimental integrity: 17 languages across multiple difficulty tiers is the right breadth, and including difficulty stratification is a good instinct — multilingual math benchmarks like MGSM are close to saturated on strong models, and the interesting transfer failures live on harder problems. Two things I’d check first. (1) Compute parity: RP-OPSD needs two teacher forward passes per rollout. Does the OPSD baseline get an equivalent budget (more rollouts, more steps)? Absent that control, part of the gain is just extra compute. (2) The pivot definition is validated by *inspection — “the weights land on tokens that look like reasoning control” — which is suggestive but circular. The convincing experiment is causal: corrupt or resample the top-scored pivot tokens versus an equal number of low-scored tokens and show accuracy collapses only in the former case. If that ablation is in the paper, it’s the most important table; if it’s absent, the central concept remains an appealing story rather than a demonstrated fact.

Writing quality: The abstract carries a typo in a load-bearing phrase (“problem-condistioned”), which is a small tell about polish. More substantively, the abstract does the classic thing of describing the mechanism at a level (“using the distributional shift … as an operational proxy to guide privileged distillation and reference anchoring”) where you can’t tell whether “reference anchoring” is a second loss term, a context-construction rule, or a decoding constraint. The section that would elevate the paper is a tightened method presentation with a fully worked single-example walkthrough — one target-language rollout, per-token pivot scores printed, the resulting weights, and where the two teacher views disagreed. That would convert the paper from “plausible reweighting scheme” to “here is a phenomenon you can now see.”

Verdict: weak accept — a well-motivated and cheap importance signal with a clean interpretation, but the pivot construct needs causal validation and the compute-matched comparison against vanilla OPSD to be more than a tuning win.

Takeaways

The transferable idea is not multilingual and not distillation-specific: diff two forward passes that differ only in privileged context, and you get a free, per-token map of where that context mattered. That trick generalizes anywhere you have side information the deployed model won’t have.

Concrete places to reuse it:

• RAG training and evaluation: run the model with and without the retrieved passage. The high-divergence tokens are the ones actually grounded in the document; everything else is parametric or filler. Use it to weight a grounding loss, or as a cheap faithfulness/attribution metric with no attention heuristics involved.
• Hint- or solution-conditioned training: for any task where you have gold outputs but want the model to work without them, this localizes which steps the gold answer is really contributing — turning a whole-sequence signal into per-token targets.
• Annotation and data curation triage: the pivot score identifies the small fraction of positions in a long generation where a human reviewer’s attention is worth spending. Same for synthetic data filtering: rollouts with no high-divergence positions are probably trivial and contribute little.
• Tool-use and agent trajectories: run with and without the tool output in context. Where they diverge is where the tool result was decision-relevant, which is exactly what you want to reward or supervise instead of the surrounding boilerplate.

The framing worth stealing: importance should be defined by counterfactual influence of information, not by the model’s own uncertainty. Entropy tells you the model is unsure; this tells you the answer would have changed things. Those are different sets of tokens, and the second one is usually the one you want.

What not to over-read: divergence marks decision points, not correct decisions. If your privileged source is noisy — a mediocre reference solution, a wrong retrieved passage — this method will confidently concentrate your gradient on exactly the places where it’s wrong.