Smoothness and Stability in GANs

TL;DR

This paper develops a theoretical framework to analyze GAN stability, deriving conditions for convergence, and constructs a GAN satisfying these conditions, clarifying the role of stabilization techniques.

Contribution

It introduces a comprehensive theoretical framework for GAN stability, identifying key conditions and designing a GAN that meets them, enhancing understanding of stabilization methods.

Findings

Existing GANs satisfy some but not all stability conditions.

A new GAN design fulfills all derived stability conditions.

Theoretical insights explain the effectiveness of stabilization techniques.

Abstract

Generative adversarial networks, or GANs, commonly display unstable behavior during training. In this work, we develop a principled theoretical framework for understanding the stability of various types of GANs. In particular, we derive conditions that guarantee eventual stationarity of the generator when it is trained with gradient descent, conditions that must be satisfied by the divergence that is minimized by the GAN and the generator's architecture. We find that existing GAN variants satisfy some, but not all, of these conditions. Using tools from convex analysis, optimal transport, and reproducing kernels, we construct a GAN that fulfills these conditions simultaneously. In the process, we explain and clarify the need for various existing GAN stabilization techniques, including Lipschitz constraints, gradient penalties, and smooth activation functions.