Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models

TL;DR

This paper explores how deep neural networks can efficiently approximate score functions in high-dimensional graphical models for diffusion-based generative modeling, overcoming the curse of dimensionality.

Contribution

It establishes that score functions in graphical models can be effectively approximated via variational inference denoising algorithms, enabling efficient neural network representations.

Findings

Demonstrates neural network approximation of score functions in Ising and Boltzmann models.

Provides sample complexity bounds for diffusion-based generative models.

Shows practical applicability to various graphical models.

Abstract

We investigate the approximation efficiency of score functions by deep neural networks in diffusion-based generative modeling. While existing approximation theories utilize the smoothness of score functions, they suffer from the curse of dimensionality for intrinsically high-dimensional data. This limitation is pronounced in graphical models such as Markov random fields, common for image distributions, where the approximation efficiency of score functions remains unestablished. To address this, we observe score functions can often be well-approximated in graphical models through variational inference denoising algorithms. Furthermore, these algorithms are amenable to efficient neural network representation. We demonstrate this in examples of graphical models, including Ising models, conditional Ising models, restricted Boltzmann machines, and sparse encoding models. Combined with off-the-shelf discretization error bounds for diffusion-based sampling, we provide an efficient sample complexity bound for diffusion-based generative modeling when the score function is learned by deep neural networks.