The Numerics of GANs

TL;DR

This paper analyzes the numerical challenges in training GANs, identifies key factors affecting convergence, and proposes a new algorithm with improved stability demonstrated through experiments on difficult architectures.

Contribution

The paper introduces a formal analysis of GAN training dynamics, revealing eigenvalue-related convergence issues, and proposes a novel algorithm that improves training stability.

Findings

Eigenvalues with zero real-part hinder convergence.

Large imaginary eigenvalues also impair convergence.

The new algorithm outperforms existing methods on hard-to-train GANs.

Abstract

In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.