The Role of ImageNet Classes in Fr\'echet Inception Distance

TL;DR

This paper investigates the Fréchet Inception Distance (FID) metric, revealing its reliance on ImageNet class features can lead to distortions and discrepancies with human judgment in evaluating generative models.

Contribution

The study uncovers how FID's dependence on ImageNet features can cause it to be manipulated or misrepresent image quality, highlighting limitations of the metric.

Findings

Aligning Top-N classification histograms reduces FID without improving quality.

FID can be distorted by aligning ImageNet class features, not actual image quality.

FastGAN's FID score is misleading compared to human evaluation.

Abstract

Fr\'echet Inception Distance (FID) is the primary metric for ranking models in data-driven generative modeling. While remarkably successful, the metric is known to sometimes disagree with human judgement. We investigate a root cause of these discrepancies, and visualize what FID "looks at" in generated images. We show that the feature space that FID is (typically) computed in is so close to the ImageNet classifications that aligning the histograms of Top-$N$ classifications between sets of generated and real images can reduce FID substantially -- without actually improving the quality of results. Thus, we conclude that FID is prone to intentional or accidental distortions. As a practical example of an accidental distortion, we discuss a case where an ImageNet pre-trained FastGAN achieves a FID comparable to StyleGAN2, while being worse in terms of human evaluation.