Lecture Notes in Probabilistic Diffusion Models

TL;DR

Diffusion models are probabilistic generative models inspired by thermodynamics, transforming complex data distributions into simple ones through a diffusion process and outperforming other generative methods like GANs.

Contribution

This paper provides an overview of diffusion models, explaining their basis in thermodynamics, their process of data transformation, and their advantages over existing generative models.

Findings

Diffusion models outperform GANs in image generation tasks.

They use a Markov chain of noise addition and removal to model data distributions.

Diffusion models have latent variables with the same dimensionality as the data.

Abstract

Diffusion models are loosely modelled based on non-equilibrium thermodynamics, where \textit{diffusion} refers to particles flowing from high-concentration regions towards low-concentration regions. In statistics, the meaning is quite similar, namely the process of transforming a complex distribution $p_{\text{complex}}$ on $\mathbb{R}^d$ to a simple distribution $p_{\text{prior}}$ on the same domain. This constitutes a Markov chain of diffusion steps of slowly adding random noise to data, followed by a reverse diffusion process in which the data is reconstructed from the noise. The diffusion model learns the data manifold to which the original and thus the reconstructed data samples belong, by training on a large number of data points. While the diffusion process pushes a data sample off the data manifold, the reverse process finds a trajectory back to the data manifold. Diffusion models have -- unlike variational autoencoder and flow models -- latent variables with the same dimensionality as the original data, and they are currently\footnote{At the time of writing, 2023.} outperforming other approaches -- including Generative Adversarial Networks (GANs) -- to modelling the distribution of, e.g., natural images.