ExpRL: Exploratory RL for LLM Mid-Training

Paper: 2606.17024 Authors: Violet Xiang, Amrith Setlur, Chase Blagden, Nick Haber, Aviral Kumar Categories: cs.LG

The Gap

Current mid-training for LLM reasoning relies on two routes: (1) supervised fine-tuning (SFT) on curated reasoning traces that teach primitive skills like decomposition or self-correction, or (2) sparse-reward reinforcement learning (RL) that only gives feedback when the final answer matches. Both have limitations. SFT requires manually specifying what to learn and cannot easily generalize to harder problems that combine multiple skills. Sparse-reward RL (e.g., GRPO) fails to reinforce partial progress, making exploration inefficient on complex tasks. The gap is that we lack an automated mid-training method that can provide dense, process-level feedback without requiring hand-crafted curricula. ExpRL closes this by using reference solutions as reward scaffolds — not as targets to imitate, but as tools to construct problem-specific grading rubrics for on-policy trajectories.

Problem: Sparse rewards miss partial progress; manual traces don't scale.
    |
    v
Assumption: Dense process rewards from hidden references can guide exploration better.
    |
    v
Method: ExpRL: policy samples trajectory -> LLM Judge compares vs reference -> dense reward.
    |
    v
Evidence: Outperforms SFT, sparse GRPO, and self-distillation on MATH benchmarks.
    |
    v
Conclusion: Reward scaffolds (not imitation) make mid-training both automatic and effective.

The Increment

One sentence: Before ExpRL, mid-training either imitated human traces (SFT) or used final-answer signals (sparse RL); after ExpRL, we can automatically generate dense, process-level rewards from any question-answer corpus, unlocking better exploration and providing a stronger initialization for downstream RL.

Core Mechanism

ExpRL treats a large corpus of human-written question-answer pairs as reward scaffolds. The reference solution (the answer) is kept hidden from the policy during generation. For each problem, the policy samples a full reasoning trajectory. An LLM judge (prompted with a rubric) receives both the trajectory and the hidden reference, then produces a score — either outcome-level (0/1 for correctness) or process-level (step-by-step partial credit). This dense reward signal is used to update the policy via standard RL (e.g., PPO). The key is that the reference is never shown to the policy as a target; it only constrains the judge’s evaluation, allowing the policy to discover novel reasoning paths that still respect the correct steps.

[Data flow]
Problem prompt
    |
    v
Policy (LLM) ---- samples ----> Reasoning Trajectory
    |
    v
LLM Judge <---- Reference (hidden from policy; only used for rubric)
    |
    v
Dense Reward (outcome or process-level)
    -->
    Update Policy via PPO

Structural metaphor: Think of a music student learning to play a new piece. In the old approach (SFT), the teacher hands the student the exact sheet music and says “play this note for note” — the student imitates blindly. In sparse-reward RL, the teacher only says “good” or “bad” after the final note, ignoring all fumbled passages. ExpRL is like the teacher sitting next to the student with the sheet music hidden under a cloth. The student plays freely. The teacher compares what they hear against the hidden sheet music, and says: “The first phrase was correct, but you rushed the second; try again with more patience there.” The student never sees the sheet music, but receives step-by-step feedback that guides practice without suppressing creativity. The LLM judge is that teacher; the hidden reference is the sheet music; the dense reward is the phrase-by-phrase critique.

Key Concepts

• Reward Scaffold: A reference solution used not as an imitation target but as a tool to construct a grading rubric. The policy never sees the reference; only the judge does. This allows the policy to explore diverse reasoning paths while still being guided toward correct intermediate states. Example: For a math word problem “A train leaves at 8am at 60mph…”, the reference might list steps: define variables, set up equation, solve, check. The judge checks each step against the trajectory, awarding partial credit if the trajectory finds a different but valid equation.

• Dense vs. Sparse Reward: Sparse reward gives +1 only if the final answer is correct, 0 otherwise. Dense reward assigns partial credit to each reasoning step or intermediate outcome. ExpRL can give both outcome-level (0/1 based on full trajectory) and process-level (points per step) signals. The process-level variant is more informative but requires the judge to segment steps, which is feasible with modern LLMs. Example: If a student solves 2/3 of a problem correctly but makes a sign error at the end, sparse reward = 0, dense process reward ≈ 0.66.

Framework Shift

Before (mainstream approach):
[Problem] -> [Policy] -> [Final Answer] -> [Sparse 0/1 Reward] -> [Policy Update]
   +                                        |
   +----------------------------------------+
   (no intermediate guidance)

After (this paper):
[Problem] -> [Policy] -> [Full Trajectory] -------+
                                                    |
                                                    v
                                          [LLM Judge] <-- [Hidden Reference]
                                                    |
                                                    v
                                          [Dense Reward (step-level)]
                                                    |
                                                    v
                                          [Policy Update via PPO]
   (policy never sees reference; exploration is guided)

One sentence: From imitate fixed traces or sparse final-reward RL to dense process rewards derived from hidden references, the core shift is using reference solutions as scaffolds for exploration rather than as targets for reproduction.

Expert Assessment

Problem choice: This is a real gap. Mid-training is an increasingly important stage in LLM development, and the field has been stuck between expensive manual curation and reward-sparsity issues. The paper targets a well-motivated, practical bottleneck.

Method maturity: The idea is clever but not revolutionary: using an LLM judge to provide dense rewards has been explored in RLHF and other contexts. The novelty lies in *hiding the reference from the policy and framing it as a scaffold rather than a target. The mechanism is straightforward, but it does add compute cost (one extra LLM call per trajectory for judging). A simpler approach — e.g., using a frozen reward model trained on reference data — might be cheaper, but the authors compare against GRPO and self-distillation, which are standard, so the bar is fair.

Experimental integrity: Baselines include SFT, sparse GRPO, and self-distillation. The results show clear improvement on MATH benchmarks; they also test on mixed-domain tasks, suggesting generalization. No obvious red flags like cherry-picked datasets or missing error bars (though I’d want to see runs seeded multiple times). The paper could be strengthened by including a cost-benefit analysis (compute of extra judge calls vs. gains).

Writing quality: The abstract and method description are clear. The paper cuts corners in the related work section (it feels thin) and lacks a thorough ablation study on the judge’s accuracy or prompt sensitivity. A dedicated section on how the judge rubric is designed and its impact on results would elevate the paper.

Verdict: weak accept — The idea is sound and practically useful, but the contribution margin is modest, and the evaluation could be more rigorous.

Takeaways

1. Use reference solutions as reward scaffolds, not imitation targets. This principle can be applied to any domain where you have a corpus of correct answers but want the model to discover its own solution paths — code generation, theorem proving, or even creative tasks like story writing with a gold-standard plot summary.

2. A hidden reference enables dense process rewards without leaking the answer. Practitioners can take any Q&A dataset (e.g., from textbooks, forums) and use an LLM judge to score on-policy generations. The key engineering detail: the judge must be prompted with a rubric that maps reference steps to partial credit, and the reference must be hidden from the policy’s context.

3. The judge’s prompt is load-bearing. If you adopt this method, invest in crafting a clear, step-by-step grading rubric. The paper’s results depend heavily on the judge’s ability to identify partial progress; a poorly prompted judge will degrade to sparse rewards. A concrete tip: include few-shot examples of what constitutes a half-correct step.