The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

TL;DR

This paper applies equilibrium statistical mechanics to generative diffusion models, revealing phase transitions and symmetry breaking phenomena that underpin their generative capabilities, and interprets their dynamics as a free energy minimization process.

Contribution

It introduces a novel equilibrium statistical mechanics framework for understanding diffusion models, highlighting phase transitions and critical phenomena as fundamental to their function.

Findings

Diffusion models undergo second-order phase transitions with symmetry breaking.

These phase transitions belong to the mean-field universality class.

The generative process can be viewed as a stochastic adiabatic free energy minimization.

Abstract

Generative diffusion models have achieved spectacular performance in many areas of machine learning and generative modeling. While the fundamental ideas behind these models come from non-equilibrium physics, variational inference and stochastic calculus, in this paper we show that many aspects of these models can be understood using the tools of equilibrium statistical mechanics. Using this reformulation, we show that generative diffusion models undergo second-order phase transitions corresponding to symmetry breaking phenomena. We show that these phase-transitions are always in a mean-field universality class, as they are the result of a self-consistency condition in the generative dynamics. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities, which are characterized by a set of mean-field critical exponents. Finally, we show that the dynamic equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in thermal equilibrium.