Gradient descent GAN optimization is locally stable

TL;DR

This paper analyzes the local stability of gradient descent optimization in GANs, showing conditions for stability in traditional GANs, identifying instability in Wasserstein GANs, and proposing regularization to improve convergence.

Contribution

It provides a theoretical stability analysis of GAN optimization, introduces a regularization method to ensure local stability, and demonstrates practical benefits in convergence and mode collapse.

Findings

Traditional GAN equilibrium points are locally asymptotically stable under certain conditions.

Wasserstein GAN can exhibit non-convergent limit cycles near equilibrium.

Regularization improves stability, convergence speed, and mode coverage in GAN training.

Abstract

Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters. We show that even though GAN optimization does not correspond to a convex-concave game (even for simple parameterizations), under proper conditions, equilibrium points of this optimization procedure are still \emph{locally asymptotically stable} for the traditional GAN formulation. On the other hand, we show that the recently proposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which \emph{is} able to guarantee local stability for both the WGAN and the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse.