The Information Dynamics of Generative Diffusion

TL;DR

This paper explains how generative diffusion models work by linking information theory and dynamics, showing how noise is transformed into structured data.

Contribution

It introduces a new framework connecting entropy production, score function divergence, and symmetry-breaking in generative diffusion.

Findings

The generative bandwidth is governed by the divergence of the score function’s vector field.

Symmetry-breaking decisions are revealed by peaks in the variance of pathwise conditional entropy.

Generative diffusion is a process of controlled, noise-induced symmetry breaking regulated by the score function.

Abstract

Generative diffusion models have emerged as a powerful class of models in machine learning, yet a unified theoretical understanding of their operation is still developing. This paper provides an integrated perspective on generative diffusion by connecting the information-theoretic, dynamical, and thermodynamic aspects. We demonstrate that the rate of conditional entropy production during generation (i.e., the generative bandwidth) is directly governed by the expected divergence of the score function’s vector field. This divergence, in turn, is linked to the branching of trajectories and generative bifurcations, which we characterize as symmetry-breaking phase transitions in the energy landscape. Beyond ensemble averages, we demonstrate that symmetry-breaking decisions are revealed by peaks in the variance of pathwise conditional entropy, capturing heterogeneity in how individual trajectories resolve uncertainty. Together, these results establish generative diffusion as a process of controlled, noise-induced symmetry breaking, in which the score function acts as a dynamic nonlinear filter that regulates both the rate and variability of information flow from noise to data.

Figures