On Duality Gap as a Measure for Monitoring GAN Training

TL;DR

This paper improves the estimation of the duality gap in GAN training, providing a more reliable measure to monitor progress and tune hyperparameters, supported by theoretical analysis and extensive experiments.

Contribution

It offers a theoretically grounded, more dependable estimation method for the duality gap, enhancing its effectiveness as a training progress indicator in GANs.

Findings

The new estimation method better captures GAN training progress.

Local perturbations help escape non-Nash saddle points.

The approach is computationally efficient and effective across models and datasets.

Abstract

Generative adversarial network (GAN) is among the most popular deep learning models for learning complex data distributions. However, training a GAN is known to be a challenging task. This is often attributed to the lack of correlation between the training progress and the trajectory of the generator and discriminator losses and the need for the GAN's subjective evaluation. A recently proposed measure inspired by game theory - the duality gap, aims to bridge this gap. However, as we demonstrate, the duality gap's capability remains constrained due to limitations posed by its estimation process. This paper presents a theoretical understanding of this limitation and proposes a more dependable estimation process for the duality gap. At the crux of our approach is the idea that local perturbations can help agents in a zero-sum game escape non-Nash saddle points efficiently. Through exhaustive experimentation across GAN models and datasets, we establish the efficacy of our approach in capturing the GAN training progress with minimal increase to the computational complexity. Further, we show that our estimate, with its ability to identify model convergence/divergence, is a potential performance measure that can be used to tune the hyperparameters of a GAN.