Phase Transitions in Fluctuations of Random Neural Networks

Paper: 2604.19738 Authors: Simmaco Di Lillo, Leonardo Maini, Domenico Marinucci (University of Rome Tor Vergata) Categories: cs.LG, stat.ML

Abstract

Establishes central and non-central limit theorems for sequences of functionals of the Gaussian output of an infinitely-wide random neural network on the d-dimensional sphere.

Three Distinct Limiting Regimes

The asymptotic behavior depends crucially on the fixed points of the covariance function:

1. Low-disorder regime (κ’(1) < 1):

   • Convergence to same functional of a limiting Gaussian field
   • Typically non-Gaussian limiting distribution

2. High-disorder regime (κ’(1) > 1):

   • Convergence to Gaussian or non-Gaussian with phase transition
   • Depends on activation function and input space dimension

3. Sparse regime (κ’(1) = 1):

   • Critical boundary behavior
   • Includes ReLU activation with appropriate normalization

Mathematical Tools

Proofs exploit:

• Hermite expansions
• Diagram Formula
• Stein-Malliavin techniques
• Fixed-point analysis of iterative operators

Key Insight

Asymptotic behavior is determined by the fixed-point structure of the iterative operator associated with the covariance, whose nature and stability governs the different limiting regimes.

Takeaways

• Rich phase transition phenomena in deep random networks
• Fixed-point analysis provides unifying framework
• Higher dimensions make Gaussian behavior harder to achieve