On the Evaluation of Generative Adversarial Networks By Discriminative Models

TL;DR

This paper introduces a domain-agnostic, discriminative model-based evaluation metric for GANs that aligns well with human judgment and outperforms traditional metrics like Inception Score.

Contribution

The authors propose a novel evaluation method using Siamese neural networks that is domain-agnostic, robust, and does not rely on pretrained classifiers.

Findings

Outperforms Inception Score in evaluations

Competitive with FID score in accuracy

Robust against mode dropping and invention

Abstract

Generative Adversarial Networks (GANs) can accurately model complex multi-dimensional data and generate realistic samples. However, due to their implicit estimation of data distributions, their evaluation is a challenging task. The majority of research efforts associated with tackling this issue were validated by qualitative visual evaluation. Such approaches do not generalize well beyond the image domain. Since many of those evaluation metrics are proposed and bound to the vision domain, they are difficult to apply to other domains. Quantitative measures are necessary to better guide the training and comparison of different GANs models. In this work, we leverage Siamese neural networks to propose a domain-agnostic evaluation metric: (1) with a qualitative evaluation that is consistent with human evaluation, (2) that is robust relative to common GAN issues such as mode dropping and invention, and (3) does not require any pretrained classifier. The empirical results in this paper demonstrate the superiority of this method compared to the popular Inception Score and are competitive with the FID score.