Score Accuracy ≠ Stability: A Hidden Trap in Diffusion Sampling

Paper: 2607.08757 Authors: Yiwei Zhou Categories: stat.ML, cs.LG, math.NA, math.PR

The Gap

Diffusion models learn a score function (the gradient of the log-density) during the forward noising process, then use it to reverse the process and generate samples. The standard theoretical guarantee says: if your score estimate has small error under the forward marginals (the noisy distributions you trained on), you’re in good shape. Papers like Song et al. (2021), Lee et al. (2023), and Chen et al. (2023) built convergence theories on exactly this assumption — small forward-marginal score error implies the reverse process is close to the true one.

But here’s the problem nobody formally pinned down: when you actually run the reverse sampler (say, Euler-Maruyama), the score is evaluated along the sampler’s own trajectory, which drifts away from the forward marginals. Training accuracy doesn’t guarantee accuracy at evaluation time. This paper closes that gap by showing the disconnect is not just a loose-end constant — it can cause complete numerical blowup.

[Score matching on forward marginals]
              |
              v
[Small L2 score error under forward process]
              |
              | But...
              v
[Reverse sampler evaluates score along its OWN trajectory]
              |
              | = distribution shift at evaluation time
              v
[Euler-Maruyama can diverge in every W_p moment]
              |
              v
[Even for smooth, bounded, Lipschitz networks]

The Increment

One sentence: Before this paper, small forward-marginal score error was implicitly treated as sufficient for stable sampling; after this paper, we know it is provably insufficient, and we have both the counterexamples and a clean fix.

Core Mechanism

The paper’s architecture has two parts: a negative result (the counterexample) and a positive result (the fix).

The counterexample construction: Zhou builds a single smooth score field on a simple 1D problem. The learned score has *arbitrarily small L² error under the forward marginals. The resulting continuous reverse-time SDE is perfectly well-behaved — nonexplosive, all moments finite, arbitrarily close to the true reverse in path-space total variation. But when you discretize with Euler-Maruyama, the discrete trajectories can wander into regions where the score error explodes. Every positive moment of the discretized endpoint diverges. Weak convergence holds (the distribution “converges” in a weak sense), but every Wasserstein distance W_p, p ≥ 1, diverges. The paper further shows this can occur within a fixed, bounded, globally Lipschitz neural network — not just an adversarially constructed mathematical function.

The positive result (projection fix): For compactly supported data, Zhou proposes projecting the learned denoiser’s output onto a known bounded closed convex set that contains the data support. This projection preserves pointwise score accuracy, gives grid-uniform moment bounds, and yields Wasserstein convergence under mild local regularity. It’s conceptually simple: just clip the denoiser to stay within the known support.

[Learned score s_theta(x,t)]
         |
         v
[Evaluated on forward marginals] --> small L2 error (guaranteed)
         |
         | but at sampling time...
         v
[Evaluated along reverse trajectory] --> error may explode
         |
         | Fix:
         v
[Project denoiser output onto support]
         |
         v
[Grid-uniform moment bounds + W_p convergence]

Structural metaphor — the flight simulator analogy: Think of score matching as training a pilot in a flight simulator. The simulator reproduces specific weather conditions (forward marginals) very accurately, and your pilot (the neural network) performs flawlessly in those conditions. The small forward-marginal error is like saying “the simulator is 99.9% accurate.” Now you send this pilot into the real sky (the reverse-time sampling trajectory). The real flight may drift into turbulence zones (regions of state space) the simulator never covered. The pilot’s perfectly calibrated responses — tuned for calm air — can cause control oscillations that grow until the plane crashes. The discrete Euler-Maruyama steps are the pilot’s control inputs: each one is tiny, but they compound in the wrong region. The “projection fix” is the equivalent of an autopilot envelope protection system: regardless of what the pilot does, the plane is physically constrained to stay within safe airspeed and altitude bounds. You keep the pilot’s accuracy where it counts, but you prevent catastrophic drift by clipping to a known safe region.

Key Concepts

• Wasserstein vs. Weak Convergence: Imagine you’re predicting where a dart will land. Weak convergence says “the probability of landing in any given region converges” — like your dart board heatmap getting more accurate. Wasserstein distance says “the average *distance between predicted and actual landing spots converges.” This paper shows you can have the first (heatmap looks right) while the second (actual error) blows up. How? The dart usually lands close to where you predicted (weak convergence), but on rare throws it flies wildly off-target, and those rare throws are so bad they make the average distance infinite. It’s the difference between “usually right” and “never catastrophically wrong.”

• Forward Marginals vs. Reverse Trajectory: The forward process starts with real data and adds noise step by step — like slowly dissolving a sculpture in acid. The forward marginals are snapshots of partially-dissolved sculptures at each time. The reverse process starts from pure noise and tries to recover the sculpture. The key insight: the reverse process doesn’t walk through the same states the forward process visited. It carves its own path through state space. Score matching tests your “sculpting instructions” against the forward snapshots, but during generation you’re following those instructions in uncharted territory.

• Projection onto a Convex Set: If you know your data lives inside a box (say, pixel values in [0,255]), and your denoiser predicts something outside that box, just clip it back in. This sounds almost too simple to be a theoretical contribution, but the paper proves it has deep consequences: it uniformly bounds all moments, prevents the numerical blowup, and restores Wasserstein convergence. The non-obvious part is that this trivial-seeming operation provably breaks the failure mode.

Framework Shift

Before (mainstream approach):        After (this paper):

[Score error on]                     [Score error on]
[forward marginals]                  [forward marginals]
       |                                    |
       v                                    v
  "Convergence"                     [May diverge numerically]
  [assumed for                      [if trajectory drifts]
   sampling]                               |
                                           v
                                    [Need trajectory-aware
                                     control: projection]
                                           |
                                           v
                                    [Provable W_p convergence]

From “trust the forward-marginal guarantee” to “verify and constrain along the sampling trajectory,” the core shift is that the training-time metric is fundamentally decoupled from the sampling-time stability guarantee.

Expert Assessment

Problem choice: This is a genuine, important gap. The diffusion theory community has been building convergence rates on top of forward-marginal score error without formally addressing this distribution-shift issue. It’s the kind of hole that everyone half-suspects but nobody pins down. The paper sits at the intersection of stochastic numerics and deep learning theory — exactly where rigorous work is most needed and most rare.

Method maturity: The counterexample construction is clever, not brute force. The 1D reduction is well-chosen — it strips away all architectural complexity and isolates the pure mathematical phenomenon. The Lipschitz-network construction extending the result to realistic architectures is the stronger contribution. The projection fix is elegant in its simplicity, though one might argue it requires knowing the data support, which isn’t always available in high dimensions. The paper doesn’t explore *approximate projections or learned support estimates.

Experimental integrity: The experiments use a small DiT-style network, which is reasonable for a theory paper. The demonstration of large growth along rare trajectories and its suppression by projection is convincing. However, the experiments are limited to low-dimensional settings. A practitioner would want to know: does this failure mode actually bite in latent diffusion models at scale? The paper is honest about this limitation but doesn’t attempt to bridge it.

Writing quality: The mathematical exposition is clean and well-structured. The counterexample-to-fix narrative arc is satisfying. Where the authors cut corners: the connection to practical diffusion model training could be much stronger. Section 5 (experiments) feels like an afterthought — one more figure showing failure rates in a realistic (even if small) image generation task would dramatically increase the paper’s impact. The transition from the 1D theory to the neural network construction (Theorem 4) deserves its own subsection with more intuition.

Verdict: weak accept — A genuinely novel theoretical insight with a clean proof structure, but the practical gap between the 1D counterexample and real-world diffusion models limits its immediate impact.

Takeaways

1. Don’t trust forward-marginal error as your sole convergence guarantee. If you’re building diffusion model theory or doing rigorous evaluation, you need to account for distribution shift between training (forward marginals) and sampling (reverse trajectory).

2. Projection is a cheap, theoretically grounded safety net. If your data has known support (images in [0,1], audio in [-1,1], etc.), projecting your denoiser output onto that set is nearly free and provably prevents the numerical blowup failure mode. This is directly implementable.

3. Weak convergence without Wasserstein convergence is a real trap. In practice, this means your samples *usually look fine, but occasionally you get catastrophically bad outputs. If you’re seeing rare but extreme artifacts in diffusion-generated samples, this paper’s mechanism is a plausible explanation — and projection is the fix.

4. For theorists: The paper opens a concrete research direction — characterizing when and how strongly the forward-to-reverse distribution shift affects specific score network architectures and noise schedules. The 1D construction is a launching pad, not a destination.