Effective Dynamics of Generative Adversarial Networks

TL;DR

This paper introduces an effective model of GAN training dynamics using particles and kernels, revealing conditions for mode collapse and ways to improve convergence.

Contribution

It presents a simplified, interpretable model of GAN training dynamics that captures mode collapse phenomena and suggests regularization strategies for better convergence.

Findings

Mode collapse transition depends on the generator's kernel.

Gradient regularizers can optimize convergence via critical damping.

The model links discriminator type to mode collapse behavior.

Abstract

Generative adversarial networks (GANs) are a class of machine-learning models that use adversarial training to generate new samples with the same (potentially very complex) statistics as the training samples. One major form of training failure, known as mode collapse, involves the generator failing to reproduce the full diversity of modes in the target probability distribution. Here, we present an effective model of GAN training, which captures the learning dynamics by replacing the generator neural network with a collection of particles in the output space; particles are coupled by a universal kernel valid for certain wide neural networks and high-dimensional inputs. The generality of our simplified model allows us to study the conditions under which mode collapse occurs. Indeed, experiments which vary the effective kernel of the generator reveal a mode collapse transition, the shape of which can be related to the type of discriminator through the frequency principle. Further, we find that gradient regularizers of intermediate strengths can optimally yield convergence through critical damping of the generator dynamics. Our effective GAN model thus provides an interpretable physical framework for understanding and improving adversarial training.