Making Method of Moments Great Again? -- How can GANs learn distributions

TL;DR

This paper provides theoretical insights into GAN training, showing that early-stage discriminator efforts focus on matching low-degree moments, enabling the generator to learn complex distributions efficiently.

Contribution

It introduces a theoretical framework explaining how GANs learn distributions by moment matching, highlighting the importance of early training stages.

Findings

Discriminator initially matches low-degree moments of distributions.

Matching moments over polynomially many samples enables learning complex distributions.

Generator can learn distributions generated by two-layer neural networks.

Abstract

Generative Adversarial Networks (GANs) are widely used models to learn complex real-world distributions. In GANs, the training of the generator usually stops when the discriminator can no longer distinguish the generator's output from the set of training examples. A central question of GANs is that when the training stops, whether the generated distribution is actually close to the target distribution, and how the training process reaches to such configurations efficiently? In this paper, we established a theoretical results towards understanding this generator-discriminator training process. We empirically observe that during the earlier stage of the GANs training, the discriminator is trying to force the generator to match the low degree moments between the generator's output and the target distribution. Moreover, only by matching these empirical moments over polynomially many training examples, we prove that the generator can already learn notable class of distributions, including those that can be generated by two-layer neural networks.