Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves

TL;DR

This paper provides a theoretical analysis of diffusion models trained with Denoising Score Matching, revealing how model complexity, data size, and noise samples influence generalization and memorization, supported by precise error expressions.

Contribution

It introduces asymptotically exact formulas for test and train errors in diffusion models with random feature parameterization, elucidating their generalization behaviors.

Findings

Test and train errors depend on data and feature ratios.

Regimes of generalization and memorization are characterized.

Theoretical results align with empirical observations.

Abstract

We theoretically investigate the phenomena of generalization and memorization in diffusion models. Empirical studies suggest that these phenomena are influenced by model complexity and the size of the training dataset. In our experiments, we further observe that the number of noise samples per data sample ($m$) used during Denoising Score Matching (DSM) plays a significant and non-trivial role. We capture these behaviors and shed insights into their mechanisms by deriving asymptotically precise expressions for test and train errors of DSM under a simple theoretical setting. The score function is parameterized by random features neural networks, with the target distribution being $d$-dimensional Gaussian. We operate in a regime where the dimension $d$, number of data samples $n$, and number of features $p$ tend to infinity while keeping the ratios $\psi_n=\frac{n}{d}$ and $\psi_p=\frac{p}{d}$ fixed. By characterizing the test and train errors, we identify regimes of generalization and memorization as a function of $\psi_n,\psi_p$, and $m$. Our theoretical findings are consistent with the empirical observations.