Self-Distillation: Teaching LLMs to Code Better Using Only Their Own Outputs

Paper: 2604.01193 Authors: Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang Categories: cs.CL

The Gap

Code generation models typically improve through three paths: reinforcement learning with verifiers (expensive, needs test cases), distillation from stronger teacher models (needs access to better models), or synthetic data generation (needs external tools). All require additional infrastructure beyond the model itself.

The boundary: Can a model improve at code generation using nothing but its own raw outputs? No verifier to check correctness, no teacher to learn from, no RL reward signal. Prior work assumed you need at least one of these external signals.

Problem: LLMs plateau without external feedback
   |
   v
Assumption: Models already know more than they show
   |                    (decoding constraints hide capability)
   v
Method: Sample diverse outputs → Filter → Fine-tune on filtered set
   |
   v
Evidence: +13% pass@1 on LiveCodeBench (42.4% → 55.3%)
   |
   v
Conclusion: Self-distillation works; gains from reshaping token distributions

The Increment

One sentence: Before this paper, improving code generation required external signals (verifiers/teachers/rewards); after, models can bootstrap from their own diverse outputs through simple supervised fine-tuning.

Core Mechanism

Simple Self-Distillation (SSD) has three steps. First, sample many solutions from the model using higher temperature (0.8) and nucleus sampling (top-p=0.95) to increase diversity. Second, filter these samples by a simple criterion: keep only solutions that pass basic syntax checks and execute without errors. Third, fine-tune the original model on this filtered dataset using standard supervised learning.

The data flow is circular: model → diverse samples → filter → training data → improved model. No external components enter this loop. The filtering step is deliberately minimal—just syntax and runtime checks, not correctness verification. This means the training set contains both correct and incorrect solutions, but all are at least well-formed code.

Original Model (T=0.6, top-p=0.9)
   |
   | sample with T=0.8, top-p=0.95
   v
[Raw Outputs: 100 solutions per problem]
   |
   | filter: syntax + runtime checks only
   v
[Filtered Set: ~30-50 solutions per problem]
   |
   | standard supervised fine-tuning
   v
Improved Model (same inference config)

Think of this like a jazz musician practicing improvisation. During practice (sampling), you play loosely, exploring variations you wouldn’t use in performance. You record these sessions, then listen back and identify passages that “work”—not necessarily perfect, but structurally sound. You drill those passages until they become part of your vocabulary. When you return to performance mode (inference), you haven’t learned new theory, but you’ve internalized patterns that were always in your capability range, just rarely surfaced under performance constraints.

The key insight: the model at temperature 0.8 explores solution spaces it knows but rarely visits at temperature 0.6. Fine-tuning on these explorations doesn’t teach new knowledge—it reshapes the probability landscape so useful patterns become more accessible during normal inference.

Key Concepts

• Precision-Exploration Conflict: In code generation, some tokens demand precision (API names, syntax) while others benefit from exploration (algorithm choices, variable names). Standard decoding uses one temperature for everything, forcing a compromise. Low temperature gives precision but misses creative solutions; high temperature explores but makes syntax errors. SSD resolves this by training the model to internalize when to be precise vs exploratory, rather than forcing a global tradeoff at inference time. Concrete example: when generating requests.get(url), the model must precisely output “requests” and “get”, but can explore different error-handling strategies. Single-temperature decoding can’t optimize both simultaneously.

• Self-Distillation Without a Teacher: Traditional distillation transfers knowledge from a large teacher to a small student. Here, the teacher and student are the same model at different decoding configurations. The “teacher” (high-temperature sampling) doesn’t know more facts—it explores more of the solution space. The “student” (fine-tuned model) learns to access those solutions without needing high temperature. It’s like distilling not knowledge but search strategy.

• Distribution Reshaping: Fine-tuning doesn’t just increase probability of correct tokens—it changes how probability mass distributes across the vocabulary in context-dependent ways. In contexts requiring precision (like after import), SSD suppresses the long tail of low-probability distractors. In contexts allowing exploration (like algorithm implementation), it preserves diversity. The paper shows this through token distribution analysis: entropy decreases in precision-critical positions, stays high in exploration-friendly positions.

Framework Shift

Before (mainstream approach):        After (this paper):

Model → [Verifier/Teacher/RL] → Improved Model
         ^                              
         |                              Model ──┐
    External Signal                            |
                                               | sample (high T)
                                               v
                                          [Raw Outputs]
                                               |
                                               | filter (minimal)
                                               v
                                          [Training Data]
                                               |
                                               | fine-tune
                                               v
                                          Improved Model
                                          (closed loop)

From external-signal-driven improvement to self-bootstrapping through decoding configuration diversity, the core shift is recognizing that capability gaps often reflect access problems, not knowledge gaps.

Expert Assessment

Problem choice: Real gap. The field has been stuck in an expensive equilibrium where improvement requires either massive compute (RL), proprietary models (distillation), or extensive test suites (verification). Showing that models can improve from their own outputs challenges the assumption that external signals are necessary. This sits at an interesting inflection point where models are capable enough that their diverse outputs contain signal, but not so capable that simple sampling already surfaces optimal solutions.

Method maturity: Embarrassingly simple, which is both strength and weakness. The method is literally “sample more, filter, fine-tune”—no novel algorithms. But simplicity here is a feature: it reveals that prior approaches may have been over-engineering. The precision-exploration framing is the real contribution, giving theoretical grounding to why such a naive method works. However, the paper doesn’t explore failure modes deeply—when does this stop working? What happens after multiple rounds?

Experimental integrity: Baselines are fair but limited. The paper compares against the base model and shows consistent gains across model families (Qwen, Llama) and sizes (4B-30B), which is strong. But it doesn’t compare against other self-improvement methods like self-consistency or rejection sampling at inference time. The filtering criterion (syntax + runtime) is suspiciously lenient—how much does training on incorrect-but-syntactic code hurt? The paper claims gains concentrate on harder problems but doesn’t show where the method fails. Ablations are solid (temperature, filtering threshold, data size) but miss some obvious experiments: what if you iterate SSD multiple times? Does it compound or saturate?

Writing quality: The precision-exploration framing in Section 4 is excellent and should have been frontloaded. The introduction buries the key insight under method description. Section 3 (experiments) is thorough but repetitive—the per-model-family results could be condensed into a single table with a summary paragraph. The related work section is perfunctory, missing connections to curriculum learning and self-play literature where similar bootstrapping dynamics appear. If the authors rewrote Section 1 to lead with the precision-exploration conflict and positioned SSD as a solution to that specific problem, the paper would be much stronger.

Verdict: weak accept — The core finding (self-distillation works for code generation) is valuable and the simplicity is refreshing, but the paper feels like it’s reporting a surprising empirical result without fully understanding why it works or when it breaks. The precision-exploration framing is underdeveloped. This is a strong workshop paper or a weak conference paper that needs one more revision to mature the theory.

Takeaways

Practitioners can steal the decoding configuration trick: when you need diverse outputs for any downstream task (not just fine-tuning), don’t just crank up temperature uniformly—use nucleus sampling with p=0.95 and temperature around 0.8, then filter aggressively. This explores more of the model’s capability range than standard sampling.

The precision-exploration lens transfers to any generation task with mixed constraints. In dialogue, some turns need factual precision (answering “what’s the capital of France?”) while others allow creativity (suggesting weekend plans). Current systems use one temperature for everything. You could apply SSD’s insight: sample diverse responses, filter for basic coherence, fine-tune to internalize when to be precise vs exploratory.

For code specifically: if you’re building on top of an LLM and can’t afford RL or don’t have test cases, SSD offers a cheap post-training step. Generate 50-100 solutions per problem in your domain, keep the syntactically valid ones, fine-tune. The paper shows this works even without correctness verification, which is surprising and useful.

The meta-lesson: before adding complex infrastructure (verifiers, teachers, RL), check if your model already has the capability but can’t access it under standard decoding. Sometimes the problem isn’t knowledge, it’s search.