Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data

TL;DR

This paper provides a theoretical analysis of diffusion models with random features on low-dimensional manifolds, revealing how data structure influences learning efficiency and error rates.

Contribution

It derives asymptotic expressions for errors in high dimensions and explores how manifold type affects sample complexity in diffusion models.

Findings

Sample complexity scales linearly with intrinsic dimension for linear manifolds.

Benefits of low-dimensional structure diminish for non-linear manifolds.

Diffusion models can leverage structured data, but the effect depends on manifold type.

Abstract

We study the theoretical behavior of denoising score matching--the learning task associated to diffusion models--when the data distribution is supported on a low-dimensional manifold and the score is parameterized using a random feature neural network. We derive asymptotically exact expressions for the test, train, and score errors in the high-dimensional limit. Our analysis reveals that, for linear manifolds the sample complexity required to learn the score function scales linearly with the intrinsic dimension of the manifold, rather than with the ambient dimension. Perhaps surprisingly, the benefits of low-dimensional structure starts to diminish once we have a non-linear manifold. These results indicate that diffusion models can benefit from structured data; however, the dependence on the specific type of structure is subtle and intricate.