GAN: Dynamics

TL;DR

This paper analyzes the dynamics of Wasserstein-GANs in the overparametrization limit, revealing how parameter clipping causes discontinuities leading to non-convergent, periodic behaviors in the mean field limit.

Contribution

It provides a quantitative analysis of Wasserstein-GAN dynamics, linking parameter clipping to blow-up phenomena and long-term periodic solutions.

Findings

Parameter clipping induces discontinuous vector fields.

Mean field dynamics blow up in finite time.

Solutions tend to long-term periodic behavior.

Abstract

We study quantitatively the overparametrization limit of the original Wasserstein-GAN algorithm. Effectively, we show that the algorithm is a stochastic discretization of a system of continuity equations for the parameter distributions of the generator and discriminator. We show that parameter clipping to satisfy the Lipschitz condition in the algorithm induces a discontinuous vector field in the mean field dynamics, which gives rise to blow-up in finite time of the mean field dynamics. We look into a specific toy example that shows that all solutions to the mean field equations converge in the long time limit to time periodic solutions, this helps explain the failure to converge.