On the capacity of deep generative networks for approximating distributions

TL;DR

This paper analyzes the capacity of deep generative networks to approximate high-dimensional distributions, providing theoretical bounds on their approximation errors in Wasserstein and f-divergence metrics.

Contribution

It offers new theoretical insights into how neural networks can approximate distributions, highlighting the role of intrinsic dimension and network architecture.

Findings

Neural networks can approximate high-dimensional distributions with errors depending on network size.

Approximation error in Wasserstein distance grows linearly with ambient dimension.

In f-divergences, the source dimension must match the intrinsic dimension of the target.

Abstract

We study the efficacy and efficiency of deep generative networks for approximating probability distributions. We prove that neural networks can transform a low-dimensional source distribution to a distribution that is arbitrarily close to a high-dimensional target distribution, when the closeness are measured by Wasserstein distances and maximum mean discrepancy. Upper bounds of the approximation error are obtained in terms of the width and depth of neural network. Furthermore, it is shown that the approximation error in Wasserstein distance grows at most linearly on the ambient dimension and that the approximation order only depends on the intrinsic dimension of the target distribution. On the contrary, when $f$-divergences are used as metrics of distributions, the approximation property is different. We show that in order to approximate the target distribution in $f$-divergences, the dimension of the source distribution cannot be smaller than the intrinsic dimension of the target distribution.