GM Score: Incorporating inter-class and intra-class generator diversity, discriminability of disentangled representation, and sample fidelity for evaluating GANs

TL;DR

This paper introduces GM Score, a comprehensive metric for evaluating GANs that considers sample quality, diversity, disentangled representations, and discriminability, aligning better with human perception.

Contribution

The paper proposes GM Score, a novel evaluation metric for GANs that integrates multiple aspects like diversity, disentanglement, and sample fidelity, improving upon existing metrics.

Findings

GM Score effectively evaluates different GAN architectures.

Disentangled representations correlate with higher GM Scores.

GM Score aligns well with human perception of sample quality.

Abstract

While generative adversarial networks (GAN) are popular for their higher sample quality as opposed to other generative models like the variational autoencoders (VAE) and Boltzmann machines, they suffer from the same difficulty of the evaluation of generated samples. Various aspects must be kept in mind, such as the quality of generated samples, the diversity of classes (within a class and among classes), the use of disentangled latent spaces, agreement of said evaluation metric with human perception, etc. In this paper, we propose a new score, namely, GM Score, which takes into various factors such as sample quality, disentangled representation, intra-class and inter-class diversity, and other metrics such as precision, recall, and F1 score are employed for discriminability of latent space of deep belief network (DBN) and restricted Boltzmann machine (RBM). The evaluation is done for different GANs (GAN, DCGAN, BiGAN, CGAN, CoupledGAN, LSGAN, SGAN, WGAN, and WGAN Improved) trained on the benchmark MNIST dataset.