Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models

TL;DR

This paper develops non-asymptotic convergence theories for diffusion models, showing improved rates for deterministic and stochastic samplers, and introduces accelerated variants with faster convergence.

Contribution

It provides the first non-asymptotic convergence analysis for diffusion models in discrete time with minimal assumptions, and proposes accelerated sampling methods.

Findings

Deterministic sampler convergence rate improved to 1/T

Stochastic sampler achieves 1/√T convergence rate matching state-of-the-art

Accelerated variants reach 1/T^2 and 1/T convergence rates

Abstract

Diffusion models, which convert noise into new data instances by learning to reverse a Markov diffusion process, have become a cornerstone in contemporary generative modeling. While their practical power has now been widely recognized, the theoretical underpinnings remain far from mature. In this work, we develop a suite of non-asymptotic theory towards understanding the data generation process of diffusion models in discrete time, assuming access to $\ell_2$-accurate estimates of the (Stein) score functions. For a popular deterministic sampler (based on the probability flow ODE), we establish a convergence rate proportional to $1/T$ (with $T$ the total number of steps), improving upon past results; for another mainstream stochastic sampler (i.e., a type of the denoising diffusion probabilistic model), we derive a convergence rate proportional to $1/\sqrt{T}$, matching the state-of-the-art theory. Imposing only minimal assumptions on the target data distribution (e.g., no smoothness assumption is imposed), our results characterize how $\ell_2$ score estimation errors affect the quality of the data generation processes. In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach without resorting to toolboxes for SDEs and ODEs. Further, we design two accelerated variants, improving the convergence to $1/T^2$ for the ODE-based sampler and $1/T$ for the DDPM-type sampler, which might be of independent theoretical and empirical interest.