A domain agnostic measure for monitoring and evaluating GANs

TL;DR

This paper introduces a domain-agnostic, low-cost evaluation measure for GANs based on the duality gap, effectively ranking models, detecting failures, and monitoring training progress across various data types without labels.

Contribution

It proposes a novel evaluation metric for GANs using duality gap, addressing the need for domain-agnostic, computationally efficient assessment methods.

Findings

Effectively ranks different GAN models

Captures mode collapse and non-convergence

Highly correlates with FID and domain-specific scores

Abstract

Generative Adversarial Networks (GANs) have shown remarkable results in modeling complex distributions, but their evaluation remains an unsettled issue. Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training. The latter cannot be determined by simply inspecting the generator and discriminator loss curves as they behave non-intuitively. We leverage the notion of duality gap from game theory to propose a measure that addresses both (i) and (ii) at a low computational cost. Extensive experiments show the effectiveness of this measure to rank different GAN models and capture the typical GAN failure scenarios, including mode collapse and non-convergent behaviours. This evaluation metric also provides meaningful monitoring on the progression of the loss during training. It highly correlates with FID on natural image datasets, and with domain specific scores for text, sound and cosmology data where FID is not directly suitable. In particular, our proposed metric requires no labels or a pretrained classifier, making it domain agnostic.