Approximability of Discriminators Implies Diversity in GANs

TL;DR

This paper demonstrates that with appropriately powerful discriminators, GANs can learn diverse distributions effectively, addressing mode collapse by focusing on discriminator strength rather than solely on generator complexity.

Contribution

The paper shows that discriminators with strong distinguishing power enable GANs to learn distributions in Wasserstein or KL-divergence, reducing mode collapse and improving diversity.

Findings

Discriminators can be designed for various generator classes to approximate Wasserstein distance.

Successful training implies the learned distribution is close to the true distribution in divergence.

Preliminary experiments suggest optimization issues, not statistical limitations, cause lack of diversity.

Abstract

While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse. The theoretical work of Arora et al. suggests a dilemma about GANs' statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse. By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators). For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of specific architectures) such that the Integral Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence. This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes. Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.