GANs as Gradient Flows that Converge

TL;DR

This paper presents a novel perspective on GANs by modeling them as gradient flows in the space of probability distributions, revealing their convergence properties and divergence causes through a connection with distribution-dependent ODEs and Fokker-Planck equations.

Contribution

It establishes a theoretical framework linking GAN training to gradient flows of probability distributions, providing insights into convergence and divergence phenomena.

Findings

GANs can be viewed as gradient flows of probability distributions.

The divergence of GANs is linked to MSE minimization between sample sets.

A unique solution to the distribution-dependent ODE is constructed and analyzed.

Abstract

This paper approaches the unsupervised learning problem by gradient descent in the space of probability density functions. A main result shows that along the gradient flow induced by a distribution-dependent ordinary differential equation (ODE), the unknown data distribution emerges as the long-time limit. That is, one can uncover the data distribution by simulating the distribution-dependent ODE. Intriguingly, the simulation of the ODE is shown equivalent to the training of generative adversarial networks (GANs). This equivalence provides a new "cooperative" view of GANs and, more importantly, sheds new light on the divergence of GANs. In particular, it reveals that the GAN algorithm implicitly minimizes the mean squared error (MSE) between two sets of samples, and this MSE fitting alone can cause GANs to diverge. To construct a solution to the distribution-dependent ODE, we first show that the associated nonlinear Fokker-Planck equation has a unique weak solution, by the Crandall-Liggett theorem for differential equations in Banach spaces. Based on this solution to the Fokker-Planck equation, we construct a unique solution to the ODE, using Trevisan's superposition principle. The convergence of the induced gradient flow to the data distribution is obtained by analyzing the Fokker-Planck equation.