A Tutorial on Diffusion Theory: From Differential Equations to Diffusion Models

TL;DR

This tutorial explains diffusion models using differential equations, deriving forward and reverse processes, training objectives, and sampling methods, connecting various diffusion-based generative models under a unified mathematical framework.

Contribution

It provides a comprehensive differential equations perspective on diffusion models, deriving reverse dynamics and unifying different sampling techniques and training objectives.

Findings

Derives forward and reverse diffusion processes as ODEs and SDEs.

Shows the equivalence of noise prediction and score matching objectives.

Connects DDPM and DDIM within the reverse SDE/ODE framework.

Abstract

This tutorial develops diffusion models from the viewpoint of differential equations. We begin with the conditional Gaussian forward process and show that this path admits both an ordinary differential equation (ODE) representation and a stochastic differential equation (SDE) representation. Averaging the conditional process over the data distribution then yields marginalized forward ODE and SDE formulations that transport the data distribution $p_0=p_{\mathrm{data}}$ to a Gaussian prior $p_1=\mathcal{N}(0,I)$. We next derive the corresponding reverse-time dynamics, namely the reverse SDE and the reverse probability-flow ODE, both of which are governed by the marginal score $\grad\log p_t(x)$. This leads to a training objective for score estimation and shows that the standard noise-prediction objective is equivalent to score matching up to an additive constant independent of the model parameters. We then discuss sampling methods for the learned reverse dynamics, including DPM-Solver, as well as guided sampling through classifier guidance and classifier-free guidance. Finally, we compare DDPM and DDIM with the reverse SDE/ODE framework and show that they share the same training objective, while DDPM sampling corresponds to discrete reverse-SDE sampling and DDIM sampling corresponds to reverse-ODE sampling.