Teaching Compact Models to Think: Knowledge Distillation for Cross-Language Clone Detection

Paper: 2605.02860 Authors: Mohamad Khajezade, Fatemeh H. Fard, Mohamed Sami Shehata Categories: cs.AI, cs.LG, cs.SE

The Gap

Cross-language code clone detection (X-CCD) asks: are these two code snippets in different languages semantically equivalent? Prior approaches fall into two camps. Traditional methods use hand-crafted features or learned embeddings but struggle with semantic equivalence across language boundaries. Recent LLM-based methods (GPT-4, DeepSeek) show promise but are black boxes—expensive, non-reproducible, privacy-concerning, and prone to formatting issues that break downstream parsing.

Compact open-source models (Phi3, Qwen-Coder) could solve the deployment problems, but they fail at reasoning-oriented tasks. When prompted to explain why two code snippets are clones, they either refuse to follow instructions or produce inconsistent output formats that can’t be reliably mapped to binary labels. The gap: we need the reasoning power of large models in the body of small models, with stable, parseable outputs.

Problem: X-CCD needs semantic reasoning across languages
   |
   v
Limitation: LLMs work but are impractical (cost/privacy/format)
             Compact models are practical but can't reason reliably
   |
   v
Assumption: Reasoning capability can be distilled from teacher to student
            Output instability can be fixed with structural constraints
   |
   v
Method: Distill DeepSeek-R1 reasoning into Phi3/Qwen + stabilization heads
   |
   v
Evidence: Improved reliability (response rate) + performance on distribution shift
   |
   v
Conclusion: Reasoning distillation + stabilization makes compact models viable for X-CCD

The Increment

One sentence: Before this paper, you chose between powerful-but-impractical LLMs or practical-but-unreliable compact models for X-CCD; after, you can distill reasoning into compact models and stabilize their outputs with classification heads.

Core Mechanism

The method has three stages. First, synthetic data generation: use DeepSeek-R1 (the teacher) to generate reasoning traces for cross-language code pairs from Project CodeNet. The teacher explains *why two snippets are or aren’t clones, producing chain-of-thought rationales. Second, knowledge distillation: fine-tune compact student models (Phi3, Qwen-Coder) on these reasoning traces using LoRA adapters, teaching them to mimic the teacher’s reasoning process. Third, response stabilization: add structural constraints to ensure consistent outputs—forced conclusion prompting (append “Therefore, the answer is:” to force a decision), a binary classification head (map hidden states directly to clone/not-clone), or a contrastive classification head (learn to separate clone pairs from non-clone pairs in embedding space).

Teacher (DeepSeek-R1)
   |
   | generates reasoning traces
   v
Synthetic Dataset: <code_pair, reasoning, label>
   |
   | fine-tune with LoRA
   v
Student (Phi3/Qwen) + Reasoning Capability
   |
   | add stabilization layer
   v
Output: Reliable binary prediction
   ^
   |
   +-- Forced conclusion prompt
   +-- Binary classification head
   +-- Contrastive classification head

Think of this like training a junior engineer. The senior engineer (teacher) doesn’t just label code pairs as “clone” or “not clone”—they walk through the logic: “These two functions both implement quicksort, but one uses recursion and the other uses a stack. The algorithmic structure is identical, so they’re clones.” The junior (student) learns by studying these explanations. But juniors sometimes freeze up or give vague answers, so you add scaffolding: force them to end every analysis with “Therefore, clone or not clone” (forced conclusion), or give them a checklist that maps directly to a decision (classification head). The scaffolding doesn’t replace understanding—it channels it into a consistent format.

Key Concepts

• Knowledge Distillation for Reasoning: Traditional distillation transfers a model’s output probabilities (soft labels) to a smaller model. Reasoning distillation transfers the *process—the chain of thought. Imagine teaching someone to solve math problems. You could just show them answers (soft labels), but they’d struggle with new problems. Instead, you show them worked examples: “First identify the variables, then set up the equation, then solve step-by-step.” They learn the method, not just the answers. Here, DeepSeek-R1 generates reasoning traces (“These functions differ in syntax but share control flow structure…”), and the student learns to produce similar reasoning before making a prediction.

• Response Stabilization: Compact models often produce outputs that can’t be parsed into binary labels—they might refuse to answer, give ambiguous text, or format inconsistently. Stabilization methods constrain the output space. Forced conclusion prompting is like adding “Circle your final answer” to an exam—it doesn’t help if the student doesn’t know the material, but it prevents them from leaving the answer blank or burying it in prose. Classification heads bypass text generation entirely: the model’s internal representation (hidden states) is mapped directly to a binary decision via a learned linear layer. It’s faster and more reliable because there’s no text parsing step to fail.

• Distribution Shift in X-CCD: The model is trained on certain language pairs (e.g., Python-Java) but tested on others (e.g., Rust-Ruby). This tests whether the model learned general semantic reasoning or just memorized surface patterns. If you train a model to recognize clones in Python and Java, does it understand “semantic equivalence” or just “Python loops that look like Java loops”? Testing on Rust-Ruby reveals whether the reasoning generalizes. The paper shows distilled models handle this shift better, suggesting they learned transferable reasoning rather than language-specific heuristics.

Framework Shift

Before (mainstream approach):        After (this paper):

LLM (black box)                      Teacher LLM
   |                                    |
   | prompt with code pair              | generate reasoning traces
   v                                    v
"Are these clones?"                  Synthetic dataset
   |                                    |
   | parse unreliable text              | distill into student
   v                                    v
Binary label (maybe)                 Student model + reasoning
                                        |
                                        | stabilization head
                                        v
                                     Reliable binary label

OR

Compact model                        (Same as above, but student
   |                                  is compact from the start)
   | prompt with code pair
   v
Inconsistent/no response

From treating models as opaque predictors to explicitly teaching them reasoning processes, then constraining their outputs structurally rather than hoping text generation succeeds.

Expert Assessment

Problem choice: Real gap. X-CCD is a genuine pain point in software engineering (code reuse detection, plagiarism, refactoring analysis), and the LLM deployment issues (cost, privacy, reproducibility) are not manufactured concerns—they’re blockers for production use. The problem sits at the intersection of program analysis and LLM practicality, which is timely.

Method maturity: Solid engineering rather than deep novelty. Knowledge distillation and LoRA fine-tuning are established techniques; the contribution is applying them to reasoning transfer and adding stabilization mechanisms. The classification head idea is straightforward but effective—it’s the kind of “why didn’t we just do this earlier” solution that works. No simpler approach is being overlooked; the paper systematically explores the design space (generation vs classification, different head architectures).

Experimental integrity: Baselines are fair (comparing against the teacher model and vanilla students). The use of Project CodeNet is appropriate—it’s a standard benchmark with verified cross-language equivalences. The distribution shift experiments (training on one language pair, testing on another) are the strongest evidence that reasoning was actually transferred, not just surface patterns. One weakness: no comparison against traditional X-CCD methods (AST-based, embedding-based), so we don’t know if distilled models beat non-LLM approaches. The paper focuses on making LLMs practical, not proving they’re necessary.

Writing quality: Clear structure, but the related work section is thin—doesn’t position the work against recent X-CCD literature beyond LLMs. The ablation studies (Table 3) are well-designed, showing the contribution of each component. The response rate metric is crucial and often ignored in LLM papers—kudos for measuring it. The paper could be elevated by adding a failure analysis: when do distilled models still fail, and why?

Verdict: Weak accept — Solid engineering contribution that makes LLMs practical for X-CCD, with good experimental design, but limited novelty in the core techniques.

Takeaways

Reasoning distillation is a deployment strategy: If you have a task where LLMs work but are impractical, generate synthetic reasoning traces from a strong teacher, then fine-tune a compact student. This transfers capability without ongoing API costs or privacy leaks. The key is generating *reasoning, not just labels—the student learns the process.

Stabilization heads are underused: When LLM text generation is unreliable, bypass it. Add a classification head that maps hidden states directly to structured outputs. This is faster (no text generation) and more reliable (no parsing failures). Applicable beyond X-CCD—any task where you need binary/categorical decisions from LLMs.

Test on distribution shift, not just in-distribution: If your model is supposed to learn a general concept (semantic equivalence, sentiment, causality), test it on data that shares the concept but differs in surface features. If performance collapses, you trained a pattern matcher, not a reasoner. The Rust-Ruby experiments here are a model for this kind of evaluation.