One Object, Three Coordinates: A Unifying View of Discrete Diffusion

Paper: 2607.05381 Authors: Rodrigo Casado Noguerales, Bernhard Schölkopf, Thomas Hofmann, Aran Raoufi Categories: cs.LG, cs.AI, cs.CL, cs.IT, stat.ML

The Gap

Here’s the messy state of affairs in discrete diffusion. People have proposed MDM (masked diffusion), UDM (uniform diffusion), SEDD, GIDD — each with its own parameterization choice (denoiser? score? something else?) and its own loss function. But nobody had a clean, unified answer to: what is each method actually optimizing, and does the parameterization choice matter?

Prior work either focused on continuous diffusion (where score vs. velocity is well-understood) or treated discrete variants empirically without rigorous justification. The discrete case is nastier because you’re dealing with continuous-time Markov chains (CTMCs) where jump rates, not Gaussian noises, are the primitive objects. Existing ELBO derivations either ignored boundary terms, hand-waved the relationship between parameterizations, or silently switched coordinate systems mid-derivation.

The gap: a rigorous, coordinate-free framework that tells you exactly what loss each discrete diffusion variant optimizes, how to convert between parameterizations, and which parameterizations are numerically safe.

Problem: Discrete diffusion has many variants (MDM, UDM, SEDD, GIDD)
         but no unified theory connecting their objectives

    What does each loss actually optimize?
                    |
                    v
    Assumption: Denoiser, score, bridge plug-in are different objects?
                |
                v
    Method: Rigorous CTMC ELBO derivation (boundary terms included)
            Oracle Distance theorem
            Coordinate conversion formulas
                |
                v
    Evidence: They are ONE object in 3 coordinates
              Verified on exactly solvable model
                |
                v
    Conclusion: Parameterization choice changes the learned process
                Some parameterizations make ELBO diverge
                All variants share the same best achievable loss: data entropy

The Increment

One sentence: Before this paper, discrete diffusion practitioners chose parameterizations by convention; after this paper, they can compute exact conversions between denoiser/score/bridge-plug-in coordinates and know which ones are numerically stable.

Core Mechanism

The paper starts from scratch with the CTMC evidence lower bound (ELBO). A discrete diffusion model defines a forward Markov chain that corrupts data by jumping between states (think: flipping tokens or masking them). The learned reverse process tries to undo this. The ELBO is the standard training objective.

The authors derive the ELBO rigorously for any noising process, making sure to include boundary terms that most prior work dropped. This derivation leads to the Oracle Distance theorem: the negative ELBO equals the data entropy *plus the KL divergence from the oracle (true) reverse process to the learned one. This is not a bound — it’s an equality. The irreducible cost that even the best model cannot avoid is the rate at which the forward process destroys information about the clean data, measured as (-\frac{d}{dt}I(Z_0; Z_t)).

For sequences with token-factorizing noise (each token jumps independently), the oracle reverse jump rate has a specific algebraic structure. The authors show this structure admits exactly three natural coordinate representations:

1. Denoiser: predicts the clean data distribution given the noisy state
2. Cavity / Bridge plug-in: predicts the reverse jump rate by “filling in” what each token would do if isolated
3. Score: the ratio of the reverse rate to the forward rate

These are not three different models — they are three ways of reading the same neural network output. The paper provides closed-form conversion formulas among all three.

Forward data: X_0 (clean tokens)
        |
        v
    Noising CTMC: jump rates q(z_t | z_0)
        |
        v
    Noisy state: Z_t
        |
        v
    Neural network f(Z_t, t) outputs in ONE coordinate:
        |
        +---> Read as denoiser p(Z_0 | Z_t)        [predicts clean data]
        |
        +---> Read as cavity g(z_i | z_{-i}, t)    [bridge plug-in]
        |
        +----> Read as score S(z_t)                 [rate ratio]
        |
        v
    Conversion formulas (closed-form, exact):
        denoiser <---> cavity <---> score
        (depends on noising process type)
        |
        v
    Reverse CTMC: samples X_0

Structural Metaphor

Think of it like temperature. A thermometer reading of 300K, 27°C, and 80.6°F are three different numbers — but they describe the same physical state. If you plug 300 into a formula expecting Fahrenheit, you’ll get nonsense. The same thing happens here.

The neural network is the thermometer. It measures one thing about the reverse process: how likely each token is to be the “right” one. But there are three scales (coordinates) for reading that measurement:

• Denoiser is like Celsius — intuitive, tells you directly “the clean token is probably X.” For masked diffusion (where the forward process only applies [MASK]), Celsius and Fahrenheit happen to read the same number. That’s why denoiser and cavity coincide for masked diffusion — it’s a special symmetry, like how 0°C = 32°F at the freezing point.

• Cavity (bridge plug-in) is like Fahrenheit — it tells you the reverse jump rate for one token while treating all others as fixed. For uniform diffusion (where tokens can flip to any value), this differs from the denoiser.

• Score is like Kelvin — used in theoretical derivations, related to ratios of probabilities rather than absolute predictions.

The punchline: if you train with one coordinate but evaluate (sample) with another, and you don’t convert, you’re running a different stochastic process. The paper shows this isn’t just a theoretical concern — for uniform diffusion with denoiser parameterization, the ELBO literally diverges at initialization (random network output), while the bridge plug-in coordinate stays finite.

Key Concepts

• Oracle Distance Theorem: The ELBO is not just a lower bound you hope is tight. The authors prove it’s an *equality: negative ELBO = data entropy + KL(oracle reverse ‖ learned reverse). Think of it like this: your model’s loss is exactly “how dumb is the data” (entropy, unavoidable) plus “how much worse is your reverse process compared to God’s reverse process” (KL, reducible with better modeling). The best possible loss is just the data entropy — every noising schedule, every forward process shares this same floor. This means the noising schedule doesn’t affect what’s achievable, only how easy it is to get there.

• Coordinate Invariance: The same neural network can be “read” as producing denoiser predictions, score ratios, or bridge plug-in rates. These readings are related by exact algebraic formulas that depend on the forward noising process. Getting the conversion wrong doesn’t give you a slightly worse model — it gives you a *different model optimizing a different objective. It’s like confusing meters with feet in an engineering calculation: the structure looks right, but everything collapses.

• Token-Factorizing Noise: The forward process corrupts each token independently. This is a strong assumption (no correlated corruption), but it’s what MDM, UDM, SEDD, and GIDD all use. Within this assumption, the three-coordinate framework becomes exact and closed-form. Outside it, you’d need to redo the analysis.

Framework Shift

Before (mainstream approach):        After (this paper):

  MDM        UDM       SEDD      GIDD     Unified framework
  (denoiser) (score)   (mixed)   (mixed)  with coordinate
     |         |         |         |       conversion
     v         v         v         v            |
  "works    "works    "works    "works"         v
   for me"   for me"   for me"       )    [denoiser] <---> [cavity]
                                                ^           ^
                                                |           |
                                                +--> [score] <-+
                                            (exact formulas)

  Each method: own loss, own theory       All methods: one object,
  Confusion about what's optimized        three coordinates, one
  No guidance on parameterization         theorem, explicit tradeoffs

From “each variant is its own thing” to “all variants are coordinate views of one object,” the core shift is recognizing that parameterization is a choice of reading convention, not a modeling decision — but a choice with real numerical consequences.

Expert Assessment

Problem choice: This is a real gap. The discrete diffusion literature has been growing fast with papers proposing different parameterizations without rigorous justification for the choices. The question “what does the network actually learn?” is fundamental, and having a coordinate-free answer prevents a lot of confusion. This sits at a critical juncture: discrete diffusion is moving from “does it work?” to “how should we design it?” — this paper provides the theoretical toolkit for the second question.

Method maturity: The approach is elegant — it’s careful mathematical derivation, not brute force. The Oracle Distance theorem is a clean result that drops out naturally once you do the ELBO derivation correctly with boundary terms. The coordinate conversion formulas are closed-form and practical. That said, the token-factorizing assumption is restrictive; correlated noise processes (common in practice for images, less so for text) are out of scope. The authors are upfront about this.

Experimental integrity: The numerical verification is done on an exactly solvable model — this is unusual and commendable. Rather than showing “our method gets better FID on CIFAR,” they verify every identity analytically. The risk: exactly solvable models are small, and it’s possible that finite-sample or high-dimensional effects could break things. But for a theory paper, this is the right approach. No red flags.

Writing quality: The paper is dense but well-structured. Section 3 (the main derivation) would benefit from a longer worked example woven in *during the derivation rather than saved for the appendix. The ASCII-quality notation (using jump rates rather than matrices) makes some passages harder to parse than necessary. The “coordinate conversion” formulas are the most practically useful contribution but are somewhat buried in the middle — elevating them to a prominent “recipe box” would help practitioners.

Verdict: weak accept — Clean theoretical contribution that unifies a confusing landscape, but the token-factorizing assumption limits immediate practical impact and the writing could be more accessible.

Takeaways

1. Stop treating parameterization as a free choice: If you’re designing a discrete diffusion model, the denoiser/score/bridge-plug-in decision isn’t aesthetic — it affects numerical stability. For uniform-style noise, use the bridge plug-in parameterization to avoid ELBO divergence at initialization.

2. The ELBO floor is data entropy, regardless of noising schedule: Don’t waste time searching over noise schedules hoping for a better ELBO floor. All schedules share the same irreducible cost. Schedules affect *optimization ease, not optimality ceiling.

3. Closed-form coordinate conversions: If you have a trained model in one coordinate, you can convert its outputs to another without retraining. The paper provides the formulas. This is useful if you find a paper using denoiser parameterization but your inference pipeline expects scores.

4. Masked diffusion is special: The fact that denoiser and cavity coincide for masked diffusion is a non-obvious structural property. If you’re working with masked diffusion, you have one fewer coordinate to worry about.