Low-dimensional adaptation of diffusion models: Convergence in total variation

TL;DR

This paper provides theoretical evidence that diffusion models like DDIM and DDPM can efficiently adapt to unknown low-dimensional structures, achieving fast convergence in total variation without requiring smoothness or log-concavity.

Contribution

It proves iteration complexity bounds for DDIM and DDPM under low-dimensional assumptions and establishes the near necessity of certain coefficients for adaptation.

Findings

Iteration complexity is no greater than $k/ ext{epsilon}$ up to log factors.

Results apply broadly without smoothness or log-concavity assumptions.

Provides a lower bound indicating the necessity of specific coefficients for low-dimensional adaptation.

Abstract

This paper investigates how diffusion generative models leverage (unknown) low-dimensional structure to accelerate sampling. Focusing on two mainstream samplers -- the denoising diffusion implicit model (DDIM) and the denoising diffusion probabilistic model (DDPM) -- and assuming accurate score estimates, we prove that their iteration complexities are no greater than the order of $k/\varepsilon$ (up to some log factor), where $\varepsilon$ is the precision in total variation distance and $k$ is some intrinsic dimension of the target distribution. Our results are applicable to a broad family of target distributions without requiring smoothness or log-concavity assumptions. Further, we develop a lower bound that suggests the (near) necessity of the coefficients introduced by Ho et al.(2020) and Song et al.(2020) in facilitating low-dimensional adaptation. Our findings provide the first rigorous evidence for the adaptivity of the DDIM-type samplers to unknown low-dimensional structure, and improve over the state-of-the-art DDPM theory regarding total variation convergence.