On the Statistical Capacity of Deep Generative Models

TL;DR

This paper challenges the assumption that deep generative models can perfectly sample from complex distributions, showing they are limited in capturing heavy tails and are not universal generators, especially with Gaussian latent variables.

Contribution

The paper provides a unifying theoretical framework demonstrating the limitations of various deep generative models in universal sampling, especially under common latent distributions.

Findings

Deep generative models are not universal generators.

Models with Gaussian latent variables produce light-tailed samples.

Results extend to models on manifolds with positive Ricci curvature.

Abstract

Deep generative models are routinely used in generating samples from complex, high-dimensional distributions. Despite their apparent successes, their statistical properties are not well understood. A common assumption is that with enough training data and sufficiently large neural networks, deep generative model samples will have arbitrarily small errors in sampling from any continuous target distribution. We set up a unifying framework that debunks this belief. We demonstrate that broad classes of deep generative models, including variational autoencoders and generative adversarial networks, are not universal generators. Under the predominant case of Gaussian latent variables, these models can only generate concentrated samples that exhibit light tails. Using tools from concentration of measure and convex geometry, we give analogous results for more general log-concave and strongly log-concave latent variable distributions. We extend our results to diffusion models via a reduction argument. We use the Gromov--Levy inequality to give similar guarantees when the latent variables lie on manifolds with positive Ricci curvature. These results shed light on the limited capacity of common deep generative models to handle heavy tails. We illustrate the empirical relevance of our work with simulations and financial data.