Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution

TL;DR

This paper provides theoretical bounds showing that Wasserstein GANs with left-invertible maps produce diverse, non-replicating distributions that are statistically close to the true data distribution, balancing novelty and accuracy.

Contribution

It introduces finite-sample bounds on the Wasserstein-1 distance for generative models, demonstrating their ability to avoid replication while maintaining statistical optimality.

Findings

Wasserstein GANs with left-invertible maps avoid replicating observed data.

Explicit finite-sample bounds relate distribution distance to sample size and model parameters.

Theoretical insights balance diversity and fidelity in generative modeling.

Abstract

This paper explores the problem of generative modeling, aiming to simulate diverse examples from an unknown distribution based on observed examples. While recent studies have focused on quantifying the statistical precision of popular algorithms, there is a lack of mathematical evaluation regarding the non-replication of observed examples and the creativity of the generative model. We present theoretical insights into this aspect, demonstrating that the Wasserstein GAN, constrained to left-invertible push-forward maps, generates distributions that avoid replication and significantly deviate from the empirical distribution. Importantly, we show that left-invertibility achieves this without compromising the statistical optimality of the resulting generator. Our most important contribution provides a finite-sample lower bound on the Wasserstein-1 distance between the generative distribution and the empirical one. We also establish a finite-sample upper bound on the distance between the generative distribution and the true data-generating one. Both bounds are explicit and show the impact of key parameters such as sample size, dimensions of the ambient and latent spaces, noise level, and smoothness measured by the Lipschitz constant.