Thermodynamic Response Functions in Singular Bayesian Models

Paper: 2603.05480 Authors: Sean Plummer Categories: stat.ML, cs.LG, math.ST

The Gap

Singular learning theory has given us powerful invariants—RLCT (real log canonical threshold) and singular fluctuation—that characterize how neural networks, mixture models, and matrix factorizations behave asymptotically. These models are “singular” because parameters aren’t uniquely identifiable: swap two mixture components, you get the same model. The Fisher information matrix degenerates, regular asymptotics break down.

But here’s the problem: practitioners use WAIC and WBIC for model selection every day, while theorists study RLCT and singular fluctuation. These two worlds don’t talk to each other. WAIC feels like a heuristic, RLCT feels like abstract algebraic geometry. Nobody knows if they’re measuring the same elephant from different angles or completely different animals.

Singular models (mixtures, neural nets)
    |
    v
Regular asymptotics fail
    |
    +---> Singular learning theory    +---> Practical criteria
    |     (RLCT, singular fluctuation) |     (WAIC, WBIC)
    |     [algebraic geometry]         |     [empirical heuristics]
    |                                  |
    +----------------------------------+
              No bridge between them
                      |
                      v
    Assumption: Posterior tempering creates
                a thermodynamic system
                      |
                      v
    Method: Compute response functions
            (derivatives w.r.t. temperature)
                      |
                      v
    Evidence: WAIC = predictive fluctuation
              WBIC ~ free energy at T=0
              Singular fluctuation = curvature
                      |
                      v
    Conclusion: All complexity measures are
                thermodynamic observables

The Increment

One sentence: Before, WAIC and RLCT lived in separate mathematical universes; now they’re different thermodynamic measurements of the same posterior geometry.

Core Mechanism

The method has three layers. First, take your posterior distribution p(θ|D) and add a temperature knob: p_β(θ|D) ∝ p(D|θ)^β p(θ). When β=1, you get the normal posterior. When β→0, you get the prior. When β→∞, you get maximum likelihood. This is posterior tempering.

Second, define observables—functions of parameters you care about, like prediction error or model complexity. For each observable, compute its expectation under the tempered posterior at different temperatures. Plot expectation vs temperature. The derivatives of these curves are response functions: how much does the observable change when you turn the temperature knob?

Third, exploit a universal identity: the derivative of a tempered expectation equals a covariance. This isn’t magic, it’s the same math behind thermodynamic fluctuation-dissipation theorems. When you differentiate ⟨f⟩_β with respect to β, you get Cov_β(f, log likelihood). This links WAIC (which measures predictive variance) to derivatives of free energy, and singular fluctuation (which measures posterior spread) to curvature of the free energy landscape.

Posterior p(theta|D)
    |
    v
Add temperature: p_beta(theta|D) ~ p(D|theta)^beta * p(theta)
    |
    +---> beta = 0: prior
    +---> beta = 1: posterior
    +---> beta -> infinity: max likelihood
    |
    v
Define observable f(theta)
    |
    v
Compute <f>_beta at different temperatures
    |
    v
Take derivative: d<f>/d(beta)
    |
    v
Universal identity: d<f>/d(beta) = Cov_beta(f, log L)
    |
    v
Response functions:
  - First derivative: susceptibility
  - Second derivative: specific heat
  - Peaks indicate phase transitions

Think of it like tuning a radio. Your posterior is a signal at a specific frequency (β=1). By sweeping through frequencies (varying β), you can measure how the signal responds—does it spike at certain frequencies? Does it fade smoothly or jump discontinuously? The response pattern reveals the signal’s internal structure. Here, β is the frequency knob, observables are signal properties (amplitude, phase), and response functions are the spectrum analyzer readout. WAIC measures how noisy the signal is at β=1. RLCT tells you the slope of signal strength as you approach β=0. Singular fluctuation is the curvature—whether the signal bends sharply or gradually. The key insight: all these measurements come from the same spectrum, just read at different points or derivatives.

Key Concepts

• Singular models: Imagine you’re trying to identify a criminal from witness descriptions, but two suspects are identical twins. No matter how many witnesses you interview, you can’t tell them apart—the parameters (which twin) aren’t identifiable. Singular models have this problem baked in: neural networks with permutation symmetry, mixture models where you can swap components, matrix factorizations where you can rotate factors. The Fisher information matrix, which normally tells you how much information each parameter carries, becomes degenerate—some directions carry zero information. This breaks the usual statistical theorems that assume you can uniquely pin down parameters with enough data.

• Posterior tempering: Take your posterior distribution and raise the likelihood to a power β. When β=1, it’s the normal posterior. When β<1, you’re “cooling down”—the likelihood matters less, the posterior spreads out toward the prior. When β>1, you’re “heating up”—the likelihood dominates, the posterior concentrates on maximum likelihood. It’s like adjusting the confidence level: low β means “I’m uncertain, show me all plausible parameters,” high β means “I’m very confident, show me only the best fit.” The magic is that by smoothly varying β, you can watch how the posterior reorganizes—does it split into multiple modes? Do modes merge? This reveals the geometry.

• Response functions: In thermodynamics, if you heat a material and measure how its magnetization changes, that’s magnetic susceptibility—a response function. Here, β is temperature, observables are like magnetization, and response functions measure how observables change with β. The first derivative (susceptibility) tells you sensitivity: does a small temperature change cause a big shift in the observable? The second derivative (specific heat) tells you about phase transitions: does the system reorganize abruptly? In singular models, response functions peak when the posterior undergoes structural changes—like when mixture components start to merge or neural network symmetries break.

Framework Shift

Before (mainstream approach):        After (this paper):

WAIC: empirical variance            Temperature axis (beta)
  |                                      |
  +---> model selection                  v
                                    Free energy F(beta)
WBIC: BIC approximation                  |
  |                                      +---> slope at beta=0: RLCT
  +---> model selection                  +---> value at beta=1: log evidence
                                         +---> curvature: singular fluctuation
RLCT: algebraic invariant                |
  |                                      v
  +---> asymptotic theory           Response functions:
                                      d<f>/d(beta) = Cov(f, log L)
Singular fluctuation:                    |
  posterior spread                       +---> WAIC: predictive response
  |                                      +---> WBIC: zero-temp limit
  +---> asymptotic theory                +---> Peaks: phase transitions

[Disconnected tools]                [Unified thermodynamic system]

From scattered model selection heuristics to a coherent thermodynamic phase diagram, the core shift is recognizing that all complexity measures are observables in a temperature-parameterized system.

Expert Assessment

Problem choice: This is a real gap. The disconnect between singular learning theory (beautiful but opaque) and practical criteria (useful but unprincipled) has frustrated people for years. Watanabe’s work on RLCT is deep but hard to operationalize. WAIC works but feels ad hoc. Bridging them isn’t just mathematical housekeeping—it could change how we think about model complexity in deep learning.

Method maturity: Elegant insight, not brute force. Posterior tempering isn’t new (it’s used in thermodynamic integration), but applying thermodynamic response theory to singular models is fresh. The universal covariance identity is a known result, but deploying it to unify WAIC/WBIC/RLCT is clever. The observable algebra formalism (quotienting out non-identifiable directions) is sophisticated—maybe too sophisticated for the empirical payoff shown here.

Experimental integrity: The experiments are illustrative, not definitive. Gaussian mixtures, reduced-rank regression, and small neural networks are canonical examples, but they’re also the easiest cases. The phase-transition-like behavior is visually compelling (order parameters collapse, susceptibilities peak), but the paper doesn’t quantify how sharp these transitions are or whether they’re artifacts of finite sample size. No comparison to alternative frameworks for understanding singular geometry. The empirical section feels like “here’s what you can see” rather than “here’s rigorous validation.”

Writing quality: The paper front-loads heavy formalism (observable algebras, quotient structures) before showing why you should care. The thermodynamic interpretation is buried in Section 4, but it’s the most intuitive part. Flip the structure: start with the phase transition plots, then explain the thermodynamic framework, then introduce the algebra. The related work section undersells how this connects to existing tempering literature in MCMC and thermodynamic integration. The conclusion overpromises (“natural organizing framework”) when the evidence is suggestive, not conclusive.

Verdict: weak accept — Connects important dots between theory and practice, but empirical validation is thin and the formalism could be more accessible.

Takeaways

If you’re doing model selection with WAIC, you can now interpret it as measuring predictive fluctuation in a thermodynamic sense—not just a variance estimate, but a response function. This reframing might help you understand when WAIC fails (e.g., in highly singular models where the posterior has weird geometry).

If you’re training neural networks, the phase transition perspective suggests you could diagnose training dynamics by tracking response functions during optimization. Are susceptibilities peaking? That might signal the network is reorganizing its internal representations. This is speculative, but it’s a concrete research direction.

The observable algebra trick—quotienting out non-identifiable directions—is technically useful if you’re working with singular models and want to define meaningful order parameters. Instead of tracking raw parameters (which are non-identifiable), track equivalence classes that respect the model’s symmetries.

For theorists, the paper shows that thermodynamic methods aren’t just for physics—they’re a natural language for talking about posterior geometry in singular models. If you’re studying neural network loss landscapes or mixture model identifiability, tempering and response functions might be sharper tools than direct geometric analysis.