Understanding GANs: the LQG Setting

TL;DR

This paper investigates fundamental issues of GANs using a high-dimensional Gaussian benchmark, proposing a new architecture that achieves maximum-likelihood solutions and better generalization, while analyzing stability of various optimization methods.

Contribution

The paper introduces a GAN architecture that recovers the maximum-likelihood solution and analyzes the stability of different optimization approaches.

Findings

Proposed GAN architecture recovers maximum-likelihood solution.

Demonstrated fast generalization in the Gaussian benchmark.

Analyzed stability and pros/cons of various optimization methods.

Abstract

Generative Adversarial Networks (GANs) have become a popular method to learn a probability model from data. In this paper, we aim to provide an understanding of some of the basic issues surrounding GANs including their formulation, generalization and stability on a simple benchmark where the data has a high-dimensional Gaussian distribution. Even in this simple benchmark, the GAN problem has not been well-understood as we observe that existing state-of-the-art GAN architectures may fail to learn a proper generative distribution owing to (1) stability issues (i.e., convergence to bad local solutions or not converging at all), (2) approximation issues (i.e., having improper global GAN optimizers caused by inappropriate GAN's loss functions), and (3) generalizability issues (i.e., requiring large number of samples for training). In this setup, we propose a GAN architecture which recovers the maximum-likelihood solution and demonstrates fast generalization. Moreover, we analyze global stability of different computational approaches for the proposed GAN optimization and highlight their pros and cons. Finally, we outline an extension of our model-based approach to design GANs in more complex setups than the considered Gaussian benchmark.