The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning

Paper: 2602.18428 Authors: Mojtaba Sahraee-Ardakan, Mauricio Delbracio, Peyman Milanfar Categories: cs.LG, cs.CV, eess.IV

Abstract

Autonomous generative models like Equilibrium Matching and blind diffusion learn a single time-invariant vector field without explicit noise-level conditioning, challenging conventional diffusion paradigms. This paper resolves a fundamental paradox: how can noise-agnostic networks remain stable near data manifolds where gradients typically diverge? The authors formalize Marginal Energy $E_{\text{marg}}(\mathbf{u}) = -\log p(\mathbf{u})$, where $p(\mathbf{u}) = \int p(\mathbf{u}|t)p(t)dt$ integrates over unknown noise levels. They prove autonomous generation is Riemannian gradient flow on this landscape, where learned fields implicitly incorporate conformal metrics that counteract $1/t^p$ singularities, converting infinite potential wells into stable attractors. The work identifies a “Jensen Gap” causing catastrophic failures in noise-prediction parameterizations while proving velocity-based approaches are inherently stable.

Key Contributions

• Formalization of Marginal Energy as the optimization landscape for autonomous diffusion models, integrating over noise-level uncertainty
• Proof that autonomous generation implements Riemannian gradient flow with implicit conformal metric correction
• Novel relative energy decomposition revealing how time-invariant fields stabilize geometric singularities near data manifolds
• Identification of the “Jensen Gap” phenomenon explaining why noise-prediction parameterizations fail catastrophically in blind settings
• Theoretical guarantee that velocity-based parameterizations satisfy bounded-gain conditions for stable sampling

The Marginal Energy Framework

The core insight lies in recognizing that when noise levels are unknown, the relevant optimization landscape is not the conditional energy $E(\mathbf{u}|t)$ but the Marginal Energy:

$E_{\text{marg}}(\mathbf{u}) = -\log \int p(\mathbf{u}|t)p(t)dt$

This formulation treats the noise level $t$ as a latent random variable with prior $p(t)$. The authors prove that autonomous models performing blind denoising are actually following the gradient flow:

$\frac{d\mathbf{u}}{ds} = -\nabla E_{\text{marg}}(\mathbf{u})$

This perspective transforms the problem from “how can a model denoise without knowing noise levels?” to “what is the geometry of the marginalized energy landscape?” The marginal density naturally weights different noise levels according to their posterior probability given the observation, enabling implicit noise-level estimation through geometric structure alone.

Geometric Stabilization Mechanism

A critical challenge is that near the data manifold, the conditional energy $E(\mathbf{u}|t)$ exhibits a $1/t^p$ singularity in the normal direction, creating infinitely steep gradients. The authors demonstrate through relative energy decomposition that autonomous models learn to incorporate a local conformal metric $g(\mathbf{u})$ such that:

$\nabla_g E_{\text{marg}}(\mathbf{u}) = g(\mathbf{u})^{-1} \nabla E_{\text{marg}}(\mathbf{u})$

This metric precisely counteracts the geometric singularity. While the raw energy landscape has unbounded gradients, the Riemannian gradient flow in the learned metric space remains bounded and stable. The time-invariant network implicitly learns this metric through training, converting what would be an unstable divergent flow into a smooth convergence toward the data manifold.

The stabilization is not accidental but emerges from the structure of the marginal distribution itself. The integration over noise levels in $p(\mathbf{u})$ naturally smooths the singularities present in individual conditional distributions $p(\mathbf{u}|t)$.

Parameterization Stability Analysis

The paper provides rigorous analysis of why different parameterizations exhibit vastly different stability properties in autonomous settings:

Noise-Prediction Parameterization: Models predicting noise $\epsilon$ suffer from the Jensen Gap:

$\mathbb{E}[\epsilon_\theta(\mathbf{u}_t)] \neq \epsilon_{\text{true}}$

This gap acts as a high-gain amplifier. Small estimation errors in the noise prediction get amplified by the $1/t$ scaling factor inherent in converting noise predictions to velocity fields. The authors prove this leads to unbounded error accumulation, explaining catastrophic failures observed empirically in deterministic blind diffusion models.

Velocity Parameterization: Models directly predicting velocity $\mathbf{v}_\theta$ satisfy a bounded-gain condition:

$\|\mathbf{v}_\theta(\mathbf{u}) - \mathbf{v}_{\text{true}}(\mathbf{u})\| \leq C$

The posterior uncertainty over noise levels gets absorbed into smooth geometric drift rather than amplified. This inherent stability makes velocity-based autonomous models practical and robust. The bounded-gain property ensures that even with imperfect noise-level estimation, the sampling trajectory remains well-behaved and converges to the data distribution.

Implications for Generative Modeling

This work fundamentally reframes our understanding of what diffusion models learn. Rather than viewing noise conditioning as essential, the geometric perspective reveals it as one possible implementation choice. Autonomous models learn the intrinsic geometry of the data distribution marginalized over noise uncertainty.

The theoretical framework suggests several practical implications:

• Velocity parameterization should be strongly preferred for autonomous/blind diffusion applications
• The success of models like Equilibrium Matching is not mysterious but follows from principled geometric properties
• Training objectives can be designed to explicitly encourage learning of stabilizing conformal metrics
• The framework extends beyond diffusion to other stochastic generative processes with latent uncertainty

The Marginal Energy perspective also connects diffusion models to broader themes in geometric deep learning and optimal transport, suggesting that successful generative models implicitly learn Riemannian structures adapted to their data distributions.

Takeaways

1. Autonomous diffusion models optimize Marginal Energy, a landscape integrating over noise-level uncertainty rather than conditioning on known noise levels
2. Generation in autonomous models is Riemannian gradient flow where learned networks implicitly incorporate conformal metrics that stabilize geometric singularities
3. The “Jensen Gap” in noise-prediction parameterizations causes high-gain error amplification, explaining catastrophic failures in blind settings
4. Velocity-based parameterizations are provably stable due to bounded-gain conditions that absorb posterior uncertainty into smooth drift
5. The geometric framework reveals that noise conditioning is not fundamental but rather one implementation choice among many possible approaches