Which Training Methods for GANs do actually Converge?

TL;DR

This paper investigates the convergence properties of GAN training, demonstrating that regularization strategies like instance noise and gradient penalties improve stability, while others like WGAN-GP may not always converge, leading to new insights into GAN stability.

Contribution

The paper provides a theoretical analysis of GAN convergence, introduces conditions under which regularized GANs converge, and empirically validates the effectiveness of certain regularization techniques.

Findings

Instance noise and zero-centered gradient penalties lead to convergence.

WGAN-GP with limited discriminator updates may not always converge.

Regularization techniques improve high-resolution image generation stability.

Abstract

Recent work has shown local convergence of GAN training for absolutely continuous data and generator distributions. In this paper, we show that the requirement of absolute continuity is necessary: we describe a simple yet prototypical counterexample showing that in the more realistic case of distributions that are not absolutely continuous, unregularized GAN training is not always convergent. Furthermore, we discuss regularization strategies that were recently proposed to stabilize GAN training. Our analysis shows that GAN training with instance noise or zero-centered gradient penalties converges. On the other hand, we show that Wasserstein-GANs and WGAN-GP with a finite number of discriminator updates per generator update do not always converge to the equilibrium point. We discuss these results, leading us to a new explanation for the stability problems of GAN training. Based on our analysis, we extend our convergence results to more general GANs and prove local convergence for simplified gradient penalties even if the generator and data distribution lie on lower dimensional manifolds. We find these penalties to work well in practice and use them to learn high-resolution generative image models for a variety of datasets with little hyperparameter tuning.