Implicit Regularization in Langevin Dynamics with Projected Noise

Paper: 2602.12257 Authors: Govind Menon, Austin J. Stromme, Adrien Vacher Categories: math.PR, cs.AI

Abstract

This work investigates Langevin dynamics where noise is projected orthogonally to an isometric group action, providing theoretical insights into how symmetry affects stochastic gradient descent in over-parametrized neural networks. The central finding establishes that when both initial and target distributions are invariant under the group action, projected-noise Langevin dynamics is equivalent in distribution to standard isotropic Langevin dynamics augmented with an additional drift term. This drift is proportional to the negative logarithmic gradient of the group orbit volume, which the authors identify geometrically as the mean curvature of the orbits. The proof constructs an elegant coupling through an auxiliary process on the group manifold itself.

Key Contributions

• Introduces a mathematical framework for studying Langevin dynamics with noise projected onto directions orthogonal to group symmetries
• Proves distributional equivalence between projected-noise dynamics and isotropic dynamics with geometric drift correction
• Identifies the additional drift term as $-\nabla \log \text{vol}(G \cdot x)$, relating it to the mean curvature of group orbits
• Constructs a novel coupling argument using an auxiliary process on the group to establish the equivalence
• Provides theoretical foundation for understanding implicit regularization induced by symmetry in over-parametrized models

Mathematical Framework and Methodology

The paper considers a smooth manifold $M$ with an isometric action by a compact Lie group $G$. The standard Langevin dynamics with isotropic noise is given by:

$dX_t = -\nabla V(X_t)dt + \sqrt{2\beta^{-1}}dW_t$

where $V$ is a potential function and $\beta$ is inverse temperature. The authors introduce projected Langevin dynamics where the Brownian motion $W_t$ is replaced by its projection onto the orthogonal complement of the group orbit directions.

The key technical innovation is constructing a coupling via a process $(X_t, g_t)$ on $M \times G$ where $g_t$ evolves according to a stochastic differential equation on the group. This allows the authors to show that the marginal distribution of $X_t$ under projected noise matches that of a modified isotropic process with the additional geometric drift term.

Geometric Interpretation and Implicit Regularization

The additional drift term $-\nabla \log \text{vol}(G \cdot x)$ has a profound geometric interpretation. The volume of the group orbit $G \cdot x = \{g \cdot x : g \in G\}$ measures the “size” of the symmetry class at point $x$. The negative gradient of its logarithm pushes the dynamics toward regions where orbits have larger volume.

This geometric drift acts as an implicit regularizer: it biases the stochastic dynamics toward solutions with higher symmetry or larger stabilizer subgroups. In the context of neural network training, this explains why SGD with symmetries tends to find solutions that respect those symmetries even without explicit regularization terms in the loss function.

The identification with mean curvature connects this phenomenon to classical differential geometry. Orbits with high mean curvature (highly curved) correspond to points where the group action is “concentrated,” and the drift naturally moves away from such singular configurations.

Implications for Deep Learning

This theoretical result has significant implications for understanding optimization in over-parametrized neural networks:

Symmetry and generalization: Neural networks often exhibit various symmetries (permutation symmetry of hidden units, scaling symmetries, etc.). This work suggests that SGD naturally exploits these symmetries through implicit regularization, potentially explaining why over-parametrized models generalize well despite having capacity to overfit.

Architecture design: The geometric drift mechanism suggests that architectural choices inducing specific symmetries will automatically bias learning toward solutions respecting those symmetries. This provides theoretical justification for symmetry-aware architectures.

Optimization landscape: The orbit volume term reveals that the effective optimization landscape experienced by projected-noise dynamics differs from the original loss landscape by a geometry-dependent correction. This helps explain convergence properties and the structure of learned solutions.

Takeaways

1. Projecting noise orthogonal to group symmetries in Langevin dynamics induces an implicit regularization effect equivalent to adding a geometric drift term
2. The drift term $-\nabla \log \text{vol}(G \cdot x)$ biases dynamics toward regions with larger orbit volumes, corresponding to higher symmetry
3. This geometric drift is identified as the mean curvature of group orbits, connecting stochastic optimization to differential geometry
4. The result provides theoretical foundation for understanding why SGD respects symmetries in over-parametrized models without explicit regularization
5. The coupling construction through an auxiliary group process offers a powerful technique for analyzing symmetry-constrained stochastic dynamics