On the regularization of Wasserstein GANs

TL;DR

This paper analyzes regularization techniques for Wasserstein GANs, advocating for weaker regularization methods over weight clipping to improve training stability, supported by theoretical insights and toy data experiments.

Contribution

It provides theoretical justification for using weaker Lipschitz regularization in Wasserstein GANs and compares it to weight clipping, supported by experimental evidence.

Findings

Weaker regularization enforces Lipschitz constraint more effectively.

Theoretical arguments favor weaker regularization over weight clipping.

Experimental results on toy datasets support the theoretical claims.

Abstract

Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. It was proposed that training can be improved by instead augmenting the loss by a regularization term that penalizes the deviation of the gradient of the critic (as a function of the network's input) from one. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on toy data sets.