Wasserstein GAN Can Perform PCA

TL;DR

This paper demonstrates that Wasserstein GANs can approximate PCA solutions in a linear Gaussian setting, highlighting their potential for optimal data representation compared to original GANs.

Contribution

It provides a theoretical analysis showing Wasserstein GANs can recover PCA solutions, unlike the original GAN, in a simplified linear Gaussian data model.

Findings

Wasserstein GAN approaches PCA solution with increasing sample size.

Original GAN fails to recover the PCA solution.

Wasserstein GAN may serve as a basis for optimal generative models.

Abstract

Generative Adversarial Networks (GANs) have become a powerful framework to learn generative models that arise across a wide variety of domains. While there has been a recent surge in the development of numerous GAN architectures with distinct optimization metrics, we are still lacking in our understanding on how far away such GANs are from optimality. In this paper, we make progress on a theoretical understanding of the GANs under a simple linear-generator Gaussian-data setting where the optimal maximum-likelihood generator is known to perform Principal Component Analysis (PCA). We find that the original GAN by Goodfellow et. al. fails to recover the optimal PCA solution. On the other hand, we show that Wasserstein GAN can approach the PCA solution in the limit of sample size, and hence it may serve as a basis for an optimal GAN architecture that yields the optimal generator for a wide range of data settings.