Singular Fluctuation as Specific Heat in Bayesian Learning

TL;DR

This paper reveals that singular fluctuation in Bayesian learning acts like specific heat in thermodynamics, providing a new interpretation that explains its role in model complexity and generalization.

Contribution

It establishes a thermodynamic interpretation of singular fluctuation as the curvature of Bayesian free energy, unifying variance identities and WAIC corrections.

Findings

Singular fluctuation equals the curvature of Bayesian free energy with respect to inverse temperature.

It behaves as a thermodynamic response coefficient in Gaussian mixture models and reduced-rank regression.

The work unifies existing variance identities and WAIC corrections under a free-energy curvature framework.

Abstract

Singular learning theory characterizes Bayesian models with non-identifiable parameterizations through two central quantities: the real log canonical threshold (RLCT), which governs marginal likelihood asymptotics, and the singular fluctuation, which determines second-order generalization behavior and the complexity term in WAIC. While the geometric meaning of the RLCT is well understood, the interpretation of singular fluctuation has remained comparatively opaque. We show that singular fluctuation admits a precise thermodynamic interpretation. Under a tempered (Gibbs) posterior, it is exactly the curvature of the Bayesian free energy with respect to inverse temperature; equivalently, the variance of the log-likelihood observable. In this sense, singular fluctuation is the statistical analogue of specific heat. This identity clarifies why singular fluctuation controls the equation of state relating training and generalization error and explains the success of WAIC in singular models: WAIC estimates a fluctuation coefficient rather than a parameter dimension. Across Gaussian mixture models and reduced-rank regression, we demonstrate that singular fluctuation behaves as a thermodynamic response coefficient. As temperature decreases, posterior reorganization suppresses fluctuation directions that affect predictive performance, and model-specific geometric observables track the decay of singular fluctuation. Rather than introducing new asymptotic expansions, this work unifies existing variance identities, equation-of-state results, and WAIC complexity corrections under a single free-energy curvature framework.