PreRL: Teaching Language Models to Reason by Pruning Wrong Paths Before Fine-Tuning

Paper: 2604.14142 Authors: Yuqiao Tan, Minzheng Wang, Bo Liu, Zichen Liu, Tian Liang, Shizhu He, Jun Zhao, Kang Liu Categories: cs.LG, cs.AI, cs.CL

The Gap

Current reinforcement learning for LLM reasoning (RLVR) optimizes P(y|x) — the probability of output y given input x. This works when the base model already knows how to reason but just needs alignment. The problem: if the base model’s output distribution doesn’t cover correct reasoning paths, no amount of conditional optimization will discover them. You’re rearranging furniture in a house with no foundation.

Pre-training could fix this by expanding P(y), but traditional pre-training uses static datasets and passive learning. It’s like trying to teach someone chess by showing them random games instead of letting them learn from their mistakes.

Problem: RLVR hits ceiling
    |
    v
Observation: P(y|x) bounded by base P(y)
    |
    v
Hypothesis: Optimize P(y) directly with RL
    |
    v
Method: PreRL (pre-train space RL) + NSR (negative sample reinforcement)
    |
    v
Evidence: 14.89x more transitions, 6.54x more reflections
    |
    v
Conclusion: Prune wrong paths first, then fine-tune

The Increment

One sentence: Before this paper, RL for reasoning meant polishing what the model already knew; after, it means expanding the reasoning space first by teaching the model what NOT to do.

Core Mechanism

PreRL applies RL directly to the marginal distribution P(y) during pre-training, not the conditional P(y|x) during fine-tuning. The key insight: gradients of log P(y) and log P(y|x) align strongly, so optimizing P(y) is a valid surrogate for standard RL.

Within PreRL, Negative Sample Reinforcement (NSR) gives negative rewards to incorrect reasoning paths. This isn’t just penalizing wrong answers — it’s actively pruning the reasoning tree. When the model generates a wrong solution, NSR pushes down its probability, forcing the model to explore alternative paths. This triggers endogenous reflection: the model starts generating “wait, that’s wrong” thoughts and backtracking steps.

The full system, Dual Space RL (DSRL), uses a two-phase approach: Phase 1 runs NSR-PreRL to expand the reasoning horizon and prune bad paths. Phase 2 initializes from this pruned space and runs standard RL for fine-grained optimization. Think of it as Policy Reincarnation — the model dies, learns what killed it, and respawns smarter.

Input x --> [Base Model] --> Initial P(y)
                                  |
                                  v
                    [NSR-PreRL: Negative Reinforcement]
                    Prune wrong paths, expand exploration
                                  |
                                  v
                            Expanded P(y)
                                  |
                                  v
                    [Standard RL: Positive Refinement]
                    Polish correct reasoning paths
                                  |
                                  v
                            Final P(y|x)

Structural metaphor: Think of reasoning space as a forest with many trails. Standard RL is like hiring a guide to show you the best trail — but only among trails the guide already knows. PreRL is like burning down the poison ivy and clearing brush before the guide arrives. NSR is the controlled burn: you deliberately set small fires on bad trails (negative reinforcement) so they become impassable. The forest doesn’t shrink — it reorganizes. Dead-end trails disappear, and new paths emerge where the undergrowth cleared. When the guide (standard RL) finally arrives, they’re working with a forest that’s been pre-shaped for navigation.

Key Concepts

• Pre-train Space vs Fine-tune Space: Pre-train space optimizes P(y) — the model’s raw ability to generate any output, independent of input. Fine-tune space optimizes P(y|x) — how well outputs match specific inputs. The difference: P(y) is about what the model CAN say; P(y|x) is about what it SHOULD say given context. If your model can’t generate correct reasoning at all (low P(y) for correct paths), no amount of input conditioning will help. It’s like trying to teach someone to write poetry in a language they don’t speak — you need vocabulary (P(y)) before you can teach style (P(y|x)).

• Negative Sample Reinforcement (NSR): Standard RL gives positive rewards for correct outputs. NSR gives negative rewards for incorrect outputs. The asymmetry matters: positive reinforcement says “do more of this,” but the model might not know what “this” generalizes to. Negative reinforcement says “don’t do THIS specific wrong thing,” which is more precise. Example: if a model solves 2+2=5, NSR doesn’t just say “wrong” — it penalizes the entire reasoning chain that led to 5, making the model less likely to use that chain again. This creates pressure to explore alternatives, like a maze-solver who marks dead ends.

• Gradient Alignment: The paper proves that ∇log P(y) and ∇log P(y|x) point in similar directions. Why does this matter? It means optimizing P(y) (easier, no input conditioning needed) approximately optimizes P(y|x) (what we actually want). It’s like training a runner on a treadmill (P(y)) instead of an outdoor track (P(y|x)) — the mechanics transfer even though the context differs. This justifies using PreRL as a surrogate for standard RL.

Framework Shift

Before (Standard RLVR):              After (DSRL):

Input x                              Input x
  |                                    |
  v                                    v
[Base Model P(y|x)]                [Base Model P(y)]
  |                                    |
  v                                    v
[RL: Reward correct y]               [NSR-PreRL: Punish wrong y]
  |                                  (Prune reasoning space)
  v                                    |
Optimized P(y|x)                       v
(bounded by base)                  Expanded P(y)
                                       |
                                       v
                                   [Standard RL: Reward correct y|x]
                                   (Refine within pruned space)
                                       |
                                       v
                                   Optimized P(y|x)
                                   (unbounded by original base)

One sentence: From polishing a fixed reasoning space to first expanding and pruning that space, then polishing.

Expert Assessment

Problem choice: Real gap. The observation that RLVR is bounded by base model capabilities is empirically obvious but theoretically underexplored. The field has been throwing more compute at fine-tuning without questioning whether the base distribution is adequate. This paper asks the right question.

Method maturity: NSR is elegant — negative reinforcement for reasoning is underused despite being standard in other RL domains. The gradient alignment proof is solid but not groundbreaking (it’s a consequence of the chain rule). The two-phase DSRL approach feels slightly ad-hoc; why not interleave NSR and positive RL? The paper doesn’t explore this design space.

Experimental integrity: Baselines are fair (standard RLVR, vanilla pre-training). The 14.89x increase in transition thoughts is striking, but the paper doesn’t show failure cases or discuss when NSR might over-prune. The experiments focus on math reasoning; generalization to other domains is unclear. One red flag: no ablation on the phase transition timing — when should you stop NSR and start standard RL?

Writing quality: The abstract oversells (“exceptionally effective driver”). Section 3’s theoretical analysis is dense and could use more intuition before formalism. The NSR mechanism is buried in Section 4 when it should be front and center. If the authors rewrote Section 2 to lead with the forest metaphor instead of equations, the paper would be far more accessible.

Verdict: weak accept — solid idea with good empirical results, but the method feels like it’s still finding its final form. The NSR insight is worth publishing, but the paper needs another iteration to clarify design choices and failure modes.

Takeaways

For practitioners: If you’re fine-tuning a model for reasoning and hitting a performance ceiling, try this: before standard RL, run a phase where you explicitly penalize incorrect reasoning chains (not just wrong answers). Track whether the model starts generating more backtracking or self-correction tokens — that’s a signal the reasoning space is expanding.

For researchers: The gradient alignment between P(y) and P(y|x) suggests a broader principle: you can optimize simpler distributions as proxies for complex conditional distributions if their gradients align. This could apply beyond reasoning — anywhere you have expensive conditional optimization, check if marginal optimization is a viable surrogate.

Transferable technique: Negative reinforcement as a pruning mechanism. Most RL focuses on rewarding correct behavior; this paper shows that punishing specific incorrect behaviors can be more effective for exploration. The key is specificity — don’t just say “wrong,” say “this exact path is wrong,” which forces the model to try different paths.