Information-Theoretic Diffusion

TL;DR

This paper introduces an information-theoretic foundation for diffusion models, connecting data distribution to denoising regression through I-MMSE relations, leading to improved probability estimation and model ensembling.

Contribution

It generalizes I-MMSE relations to relate data distributions with denoising regression, providing a unified framework for continuous and discrete probability learning.

Findings

Theoretical justification for diffusion model ensembling.

Unified approach for continuous and discrete probability learning.

Refined bounds for probability distribution estimation.

Abstract

Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation. We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connect Information with Minimum Mean Square Error regression, the so-called I-MMSE relations. We generalize the I-MMSE relations to exactly relate the data distribution to an optimal denoising regression problem, leading to an elegant refinement of existing diffusion bounds. This new insight leads to several improvements for probability distribution estimation, including theoretical justification for diffusion model ensembling. Remarkably, our framework shows how continuous and discrete probabilities can be learned with the same regression objective, avoiding domain-specific generative models used in variational methods. Code to reproduce experiments is provided at http://github.com/kxh001/ITdiffusion and simplified demonstration code is at http://github.com/gregversteeg/InfoDiffusionSimple.