On the Robustness of Quality Measures for GANs

TL;DR

This paper demonstrates that popular GAN quality metrics like IS and FID are vulnerable to adversarial manipulations, but replacing the Inception network with a robust version can improve their resilience.

Contribution

The study reveals the susceptibility of GAN evaluation metrics to adversarial attacks and proposes a robust Inception network to enhance metric robustness.

Findings

IS and FID can be manipulated by pixel perturbations

Robust Inception improves metric resilience

GAN models like StyleGANv2 are vulnerable to latent space perturbations

Abstract

This work evaluates the robustness of quality measures of generative models such as Inception Score (IS) and Fr\'echet Inception Distance (FID). Analogous to the vulnerability of deep models against a variety of adversarial attacks, we show that such metrics can also be manipulated by additive pixel perturbations. Our experiments indicate that one can generate a distribution of images with very high scores but low perceptual quality. Conversely, one can optimize for small imperceptible perturbations that, when added to real world images, deteriorate their scores. We further extend our evaluation to generative models themselves, including the state of the art network StyleGANv2. We show the vulnerability of both the generative model and the FID against additive perturbations in the latent space. Finally, we show that the FID can be robustified by simply replacing the standard Inception with a robust Inception. We validate the effectiveness of the robustified metric through extensive experiments, showing it is more robust against manipulation.