Gaussian Universality for Diffusion Models

TL;DR

This paper demonstrates that for data generated by diffusion models, the test error of linear classifiers depends only on the first two moments of the data, establishing a Gaussian universality principle for diffusion-generated data.

Contribution

It proves a Gaussian universality result for diffusion model data, showing test error depends solely on means and covariances, and highlights challenges in extending existing universality proofs.

Findings

Test error of linear models matches between diffusion data and Gaussian mixtures.

High probability closeness of scalar functions of diffusion samples to their expectations.

Current universality proofs do not extend to diffusion data due to covariance singularities.

Abstract

We investigate Gaussian Universality for data distributions generated via diffusion models. By Gaussian Universality we mean that the test error of a generalized linear model $f(\mathbf{W})$ trained for a classification task on the diffusion data matches the test error of $f(\mathbf{W})$ trained on the Gaussian Mixture with matching means and covariances per class.In other words, the test error depends only on the first and second order statistics of the diffusion-generated data in the linear setting. As a corollary, the analysis of the test error for linear classifiers can be reduced to Gaussian data from diffusion-generated data. Analysing the performance of models trained on synthetic data is a pertinent problem due to the surge of methods such as \cite{sehwag2024stretchingdollardiffusiontraining}. Moreover, we show that, for any $1$- Lipschitz scalar function $\phi$, $\phi(\mathbf{x})$ is close to $\mathbb{E} \phi(\mathbf{x})$ with high probability for $\mathbf{x}$ sampled from the conditional diffusion model corresponding to each class. Finally, we note that current approaches for proving universality do not apply to diffusion-generated data as the covariance matrices of the data tend to have vanishing minimum singular values, contrary to the assumption made in the literature. This leaves extending previous mathematical universality results as an intriguing open question.