Is Generator Conditioning Causally Related to GAN Performance?

TL;DR

This paper investigates the Jacobian singular values of GAN generators, finds their conditioning predicts GAN quality metrics, and introduces Jacobian Clamping to improve training stability and performance.

Contribution

It demonstrates the causal relationship between generator Jacobian conditioning and GAN performance, and proposes a novel regularization method called Jacobian Clamping.

Findings

Jacobian conditioning becomes ill-conditioned early in training.

Jacobian conditioning strongly predicts Inception Score and FID.

Jacobian Clamping improves GAN performance and reduces score variance.

Abstract

Recent work (Pennington et al, 2017) suggests that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning. Motivated by this, we study the distribution of singular values of the Jacobian of the generator in Generative Adversarial Networks (GANs). We find that this Jacobian generally becomes ill-conditioned at the beginning of training. Moreover, we find that the average (with z from p(z)) conditioning of the generator is highly predictive of two other ad-hoc metrics for measuring the 'quality' of trained GANs: the Inception Score and the Frechet Inception Distance (FID). We test the hypothesis that this relationship is causal by proposing a 'regularization' technique (called Jacobian Clamping) that softly penalizes the condition number of the generator Jacobian. Jacobian Clamping improves the mean Inception Score and the mean FID for GANs trained on several datasets. It also greatly reduces inter-run variance of the aforementioned scores, addressing (at least partially) one of the main criticisms of GANs.