On the Generalization Properties of Diffusion Models

TL;DR

This paper provides a theoretical analysis of diffusion models' generalization capabilities, showing they can achieve small errors without the curse of dimensionality and highlighting the impact of mode shifts in data.

Contribution

It offers the first comprehensive theoretical estimates of the generalization gap for diffusion models, including data-dependent scenarios with mode shifts.

Findings

Generalization error scales polynomially with sample size and model capacity.

Early stopping prevents exponential growth of error in high dimensions.

Mode shifts in data negatively impact model generalization.

Abstract

Diffusion models are a class of generative models that serve to establish a stochastic transport map between an empirically observed, yet unknown, target distribution and a known prior. Despite their remarkable success in real-world applications, a theoretical understanding of their generalization capabilities remains underdeveloped. This work embarks on a comprehensive theoretical exploration of the generalization attributes of diffusion models. We establish theoretical estimates of the generalization gap that evolves in tandem with the training dynamics of score-based diffusion models, suggesting a polynomially small generalization error ($O(n^{-2/5}+m^{-4/5})$) on both the sample size $n$ and the model capacity $m$, evading the curse of dimensionality (i.e., not exponentially large in the data dimension) when early-stopped. Furthermore, we extend our quantitative analysis to a data-dependent scenario, wherein target distributions are portrayed as a succession of densities with progressively increasing distances between modes. This precisely elucidates the adverse effect of "modes shift" in ground truths on the model generalization. Moreover, these estimates are not solely theoretical constructs but have also been confirmed through numerical simulations. Our findings contribute to the rigorous understanding of diffusion models' generalization properties and provide insights that may guide practical applications.