On Convergence and Stability of GANs

TL;DR

This paper reinterprets GAN training as regret minimization to analyze convergence and mode collapse, proposing a gradient penalty method called DRAGAN that improves stability and reduces mode collapse.

Contribution

It introduces a new perspective on GAN training dynamics and proposes DRAGAN, a gradient penalty scheme that enhances training stability and mitigates mode collapse.

Findings

DRAGAN improves training stability and reduces mode collapse.

Local equilibria with sharp gradients cause mode collapse.

Gradient penalty enables faster and more stable GAN training.

Abstract

We propose studying GAN training dynamics as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions. We analyze the convergence of GAN training from this new point of view to understand why mode collapse happens. We hypothesize the existence of undesirable local equilibria in this non-convex game to be responsible for mode collapse. We observe that these local equilibria often exhibit sharp gradients of the discriminator function around some real data points. We demonstrate that these degenerate local equilibria can be avoided with a gradient penalty scheme called DRAGAN. We show that DRAGAN enables faster training, achieves improved stability with fewer mode collapses, and leads to generator networks with better modeling performance across a variety of architectures and objective functions.