Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions

TL;DR

This paper analyzes the approximation and generalization capabilities of score-based neural network generative models for sub-Gaussian distributions, establishing nearly optimal convergence rates under mild assumptions.

Contribution

It provides a universal framework for convergence analysis of SGMs with neural networks, removing strict assumptions and achieving near minimax rates for broad classes of distributions.

Findings

Neural networks can approximate scores with mean square error $ ilde{O}(n^{-1})$.

SGMs can attain nearly minimax convergence rates under mild conditions.

The framework applies to distributions in Sobolev or Besov classes with early stopping.

Abstract

This paper studies the approximation and generalization abilities of score-based neural network generative models (SGMs) in estimating an unknown distribution $P_0$ from $n$ i.i.d. observations in $d$ dimensions. Assuming merely that $P_0$ is $\alpha$-sub-Gaussian, we prove that for any time step $t \in [t_0, n^{\mathcal{O}(1)}]$, where $t_0 > \mathcal{O}(\alpha^2n^{-2/d}\log n)$, there exists a deep ReLU neural network with width $\leq \mathcal{O}(n^{\frac{3}{d}}\log_2n)$ and depth $\leq \mathcal{O}(\log^2n)$ that can approximate the scores with $\tilde{\mathcal{O}}(n^{-1})$ mean square error and achieve a nearly optimal rate of $\tilde{\mathcal{O}}(n^{-1}t_0^{-d/2})$ for score estimation, as measured by the score matching loss. Our framework is universal and can be used to establish convergence rates for SGMs under milder assumptions than previous work. For example, assuming further that the target density function $p_0$ lies in Sobolev or Besov classes, with an appropriately early stopping strategy, we demonstrate that neural network-based SGMs can attain nearly minimax convergence rates up to logarithmic factors. Our analysis removes several crucial assumptions, such as Lipschitz continuity of the score function or a strictly positive lower bound on the target density.