Universality Theorems for Generative Models

TL;DR

This paper proves that neural networks can universally approximate any data manifold, establishing a theoretical foundation for the success of generative models in practice.

Contribution

It provides the first formal universality theorems for generative models, showing neural networks can approximate data manifolds arbitrarily well under mild conditions.

Findings

Neural networks can approximate any data manifold within a specified Hausdorff distance.

Universality theorems are extended to multiclass and cycle generative models.

The results hold under mild assumptions on activation functions.

Abstract

Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice. In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well? We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold. We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.