TIDE: Cross-Architecture Distillation for Diffusion Language Models

Paper: 2604.26951 Authors: Gongbo Zhang, Wen Wang, Ye Tian, Li Yuan Categories: cs.CL, cs.AI, cs.LG

The Gap

Diffusion LLMs (dLLMs) can decode in parallel and use bidirectional context, but they need billions of parameters to match autoregressive models. Existing distillation methods for dLLMs only reduce inference steps within the same architecture. No one has tackled cross-architecture distillation, where teacher and student differ in architecture, attention mechanism, and tokenizer. This matters because you can’t shrink a model to mobile-scale without changing its fundamental structure.

Problem: dLLMs need billions of params for good performance
   |
   v
Prior work: Distill within same architecture (reduce steps, not size)
   |
   v
Gap: No cross-architecture distillation (different tokenizers, attention)
   |
   v
TIDE: Three-component framework for heterogeneous distillation
   |
   +---> TIDAL: Noise-aware distillation strength modulation
   +---> CompDemo: Complementary masking for teacher context
   +---> Reverse CALM: Cross-tokenizer likelihood matching
   |
   v
Evidence: 0.6B student matches 8B teacher, +1.53 avg across benchmarks
   |
   v
Conclusion: Cross-architecture distillation viable for dLLMs

The Increment

One sentence: Before TIDE, you couldn’t distill a diffusion LLM into a fundamentally different architecture; after TIDE, you can compress an 8B model into 0.6B across different tokenizers and attention mechanisms.

Core Mechanism

TIDE has three components working in sequence. TIDAL modulates how strongly the student mimics the teacher based on two factors: training progress (early vs late) and diffusion timestep (clean vs noisy). Early in training, the student needs strong guidance; late in training, it should explore on its own. At noisy timesteps, the teacher’s predictions are unreliable, so TIDAL reduces distillation weight. At clean timesteps, the teacher is confident, so TIDAL increases weight.

CompDemo addresses a different problem: when the input is heavily masked, the teacher lacks context to make good predictions. CompDemo splits the mask into complementary subsets, runs the teacher on each subset separately (so each run has more context), then combines the predictions. This gives the student better targets to learn from.

Reverse CALM handles the tokenizer mismatch. The teacher and student use different tokenizers, so their vocabularies don’t align. Instead of matching token-level distributions (which would require alignment), Reverse CALM matches chunk-level likelihoods in reverse: the student generates a chunk, the teacher scores it, and the student learns to maximize that score. This inverts the usual direction (teacher generates, student mimics) and naturally handles tokenizer differences.

Input text with masks
   |
   v
[CompDemo: Split mask into complementary subsets]
   |
   +---> Subset A ---> Teacher prediction A
   +---> Subset B ---> Teacher prediction B
   |                        |
   +------------------------+
   |
   v
Combined teacher prediction (better context)
   |
   v
[TIDAL: Modulate distillation strength]
   |
   +---> Training progress factor (early=high, late=low)
   +---> Timestep factor (noisy=low, clean=high)
   |
   v
Weighted distillation loss
   |
   v
[Reverse CALM: Cross-tokenizer matching]
   |
   +---> Student generates chunk
   +---> Teacher scores chunk likelihood
   +---> Student maximizes teacher's score
   |
   v
0.6B student model (different architecture, tokenizer, attention)

Think of TIDE like training a chef (student) under a master chef (teacher) who speaks a different language and uses different tools. TIDAL is the training schedule: early on, the student watches closely and copies every move; later, the student experiments while the master observes. When the master is tired or distracted (noisy timesteps), the student relies less on their guidance. CompDemo is the master showing the same dish from multiple angles so the student understands it better despite limited view. Reverse CALM is the student cooking a dish, then the master tasting it and giving a score—the student learns what the master likes without needing to understand the master’s exact recipe or ingredients.

Key Concepts

• Diffusion timestep reliability: In diffusion models, generation happens over multiple timesteps, starting from pure noise and gradually denoising. At early timesteps (high noise), the model’s predictions are essentially guesses—there’s too much noise to extract meaningful signal. At late timesteps (low noise), the model can see the structure clearly and make confident predictions. If you distill uniformly across all timesteps, you’re forcing the student to mimic the teacher’s wild guesses at noisy timesteps, which hurts learning. TIDAL weights distillation by timestep: low weight when the teacher is guessing, high weight when the teacher is confident.

• Complementary mask splitting: When you mask 50% of tokens randomly, the teacher sees only half the context. If you mask a different 50%, the teacher sees the other half. CompDemo runs both: mask set A gives prediction A, mask set B gives prediction B. The predictions are complementary—where A is uncertain (because those tokens were masked), B is confident (because those tokens were visible), and vice versa. Combining them gives a prediction that’s confident everywhere, as if the teacher had seen the full context. This is cheaper than running the teacher on the full unmasked input and more informative than running it once on a single mask.

• Reverse likelihood matching: Standard distillation matches distributions: teacher generates a distribution over tokens, student learns to output the same distribution. But if teacher and student use different tokenizers, their token spaces don’t align—you can’t match distributions across different vocabularies. Reverse CALM flips it: student generates a sequence (using its own tokenizer), teacher computes the likelihood of that sequence (using its own tokenizer to parse it), student maximizes that likelihood. The teacher acts as a scorer, not a generator. This works across tokenizers because the teacher only needs to evaluate a concrete sequence, not align vocabulary spaces.

Framework Shift

Before (same-architecture distillation):        After (TIDE cross-architecture):

Teacher (8B, tokenizer A, arch X)               Teacher (8B, tokenizer A, arch X)
   |                                                |
   | generate distribution                         | score student output
   v                                                v
Student (8B, tokenizer A, arch X)               Student (0.6B, tokenizer B, arch Y)
   |                                                ^
   | match distribution                             | maximize score
   v                                                |
Smaller inference steps,                        Different architecture,
same model size                                 30x smaller model

Uniform distillation weight                     Noise-aware modulation (TIDAL)
Single mask per sample                          Complementary masks (CompDemo)
Token-level matching                            Chunk-level reverse matching (CALM)

From homogeneous compression to heterogeneous transfer, the core shift is treating the teacher as an evaluator rather than a template.

Expert Assessment

Problem choice: Real gap. Diffusion LLMs are stuck at billion-parameter scale for competitive performance, which blocks deployment. Prior distillation work (SDLM, DINOISER) only reduced inference steps within the same architecture—useful for speed, not size. Cross-architecture distillation is the natural next step, and the paper correctly identifies the blockers: tokenizer mismatch, noise-dependent teacher reliability, and context degradation under heavy masking.

Method maturity: TIDAL and CompDemo are solid engineering—noise-aware weighting is a known trick in diffusion training, and complementary masking is a clever application of ensemble ideas. Reverse CALM is the genuinely novel piece: inverting the distillation direction to sidestep tokenizer alignment is elegant. The framework is modular, which is good for ablation but raises the question: are all three components necessary, or is one doing most of the work? The ablation table (Table 3 in the paper) shows each component contributes, but the gains are uneven—TIDAL gives the biggest boost, CompDemo is marginal on some benchmarks.

Experimental integrity: Baselines are fair. They compare against SDLM (same-architecture distillation) and standard AR models. The 8B→0.6B distillation is impressive, but the paper also tests 16B MoE→0.6B, which muddies the water—MoE models have different capacity dynamics, and it’s unclear if the gains come from TIDE or from the MoE teacher’s superior knowledge. HumanEval jumping from 32.3 to 48.78 is striking, but code generation is one domain; the average gain across eight benchmarks is 1.53 points, which is meaningful but not transformative. The paper doesn’t report training cost or wall-clock time, which matters for practitioners.

Writing quality: The method section is dense but clear. The related work section is too brief—it mentions SDLM and DINOISER but doesn’t explain why their approaches fail for cross-architecture distillation, forcing the reader to infer. The ablation study is thorough, but the paper buries the key insight: Reverse CALM’s gradient bounding property (mentioned in passing) is what makes cross-tokenizer distillation stable. That deserves a dedicated subsection with math. The conclusion oversells: “first framework for cross-architecture dLLM distillation” is accurate, but “outperforms the baseline by an average of 1.53 points” undersells the real achievement, which is making cross-architecture distillation work at all.

Verdict: weak accept — Solves a real problem with a modular, well-ablated framework, but the gains are incremental and the writing doesn’t highlight the core insight (Reverse CALM’s gradient properties) clearly enough.

Takeaways

Practitioners can steal the noise-aware weighting idea (TIDAL) for any distillation task involving noisy or uncertain teacher predictions—not just diffusion models, but also early-stage RL policies or models trained on noisy data. The complementary masking trick (CompDemo) transfers to any masked language model training: instead of one random mask per sample, use two complementary masks and ensemble the predictions. Reverse CALM’s insight—invert the distillation direction to sidestep alignment problems—applies beyond tokenizers: use it whenever teacher and student operate in different spaces (e.g., distilling a graph neural network into a sequence model, or distilling a multimodal model into a text-only model). The key is making the teacher a scorer, not a generator.