Nonparametric estimation of a factorizable density using diffusion models

TL;DR

This paper analyzes diffusion models as a nonparametric density estimation method that leverages low-dimensional structures in high-dimensional data, achieving optimal rates with a specialized neural network architecture.

Contribution

It introduces a statistical framework for diffusion models as implicit density estimators that adapt to factorization structures and attain minimax optimal rates.

Findings

Diffusion models can be viewed as implicit nonparametric density estimators.

The proposed method adapts to low-dimensional structures in high-dimensional data.

Achieves minimax optimal rates in total variation distance.

Abstract

In recent years, diffusion models, and more generally score-based deep generative models, have achieved remarkable success in various applications, including image and audio generation. In this paper, we view diffusion models as an implicit approach to nonparametric density estimation and study them within a statistical framework to analyze their surprising performance. A key challenge in high-dimensional statistical inference is leveraging low-dimensional structures inherent in the data to mitigate the curse of dimensionality. We assume that the underlying density exhibits a low-dimensional structure by factorizing into low-dimensional components, a property common in examples such as Bayesian networks and Markov random fields. Under suitable assumptions, we demonstrate that an implicit density estimator constructed from diffusion models adapts to the factorization structure and achieves the minimax optimal rate with respect to the total variation distance. In constructing the estimator, we design a sparse weight-sharing neural network architecture, where sparsity and weight-sharing are key features of practical architectures such as convolutional neural networks and recurrent neural networks.