Information-Theoretic Proofs for Diffusion Sampling

TL;DR

This paper introduces an elementary, information-theoretic framework for analyzing discrete-time diffusion sampling methods, providing non-asymptotic convergence guarantees and insights into acceleration techniques.

Contribution

It offers a novel, self-contained analysis directly on discrete-time processes using information theory, avoiding continuous-time approximations, and introduces methods to accelerate convergence.

Findings

Provides non-asymptotic convergence bounds for diffusion sampling

Shows how to accelerate convergence using additional randomness

Establishes a coupling with an idealized Gaussian process

Abstract

This paper provides an elementary, self-contained analysis of diffusion-based sampling methods for generative modeling. In contrast to existing approaches that rely on continuous-time processes and then discretize, our treatment works directly with discrete-time stochastic processes and yields precise non-asymptotic convergence guarantees under broad assumptions. The key insight is to couple the sampling process of interest with an idealized comparison process that has an explicit Gaussian-convolution structure. We then leverage simple identities from information theory, including the I-MMSE relationship, to bound the discrepancy (in terms of the Kullback-Leibler divergence) between these two discrete-time processes. In particular, we show that, if the diffusion step sizes are chosen sufficiently small and one can approximate certain conditional mean estimators well, then the sampling distribution is provably close to the target distribution. Our results also provide a transparent view on how to accelerate convergence by using additional randomness in each step to match higher-order moments in the comparison process.