A Differential Equation Approach for Wasserstein GANs and Beyond

TL;DR

This paper introduces a differential equation framework for Wasserstein GANs, leading to a new class of models (W1-FE) that outperform traditional WGANs when trained with carefully integrated persistent training, supported by theoretical and experimental results.

Contribution

The paper develops an ODE-based perspective for WGANs, deriving a new class of models that improve training efficiency and results through a novel integration of persistent training.

Findings

W1-FE reduces to WGAN when persistent training is off.

W1-FE outperforms WGAN in convergence speed and training quality.

Naive persistent training without the ODE framework can worsen results.

Abstract

This paper proposes a new theoretical lens to view Wasserstein generative adversarial networks (WGANs). To minimize the Wasserstein-1 distance between the true data distribution and our estimate of it, we derive a distribution-dependent ordinary differential equation (ODE) which represents the gradient flow of the Wasserstein-1 loss, and show that a forward Euler discretization of the ODE converges. This inspires a new class of generative models that naturally integrates persistent training (which we call W1-FE). When persistent training is turned off, we prove that W1-FE reduces to WGAN. When we intensify persistent training, W1-FE is shown to outperform WGAN in training experiments from low to high dimensions, in terms of both convergence speed and training results. Intriguingly, one can reap the benefits only when persistent training is carefully integrated through our ODE perspective. As demonstrated numerically, a naive inclusion of persistent training in WGAN (without relying on our ODE framework) can significantly worsen training results.