Conditional Frechet Inception Distance

TL;DR

This paper introduces the Conditional Frechet Inception Distance (CFID), a new metric for evaluating conditional generative models that improves upon classical FID and MSE by better identifying specific types of failures.

Contribution

The paper develops a closed-form solution for the conditional FID metric and demonstrates its advantages over existing metrics in evaluating conditional generative models.

Findings

CFID outperforms classical FID in detecting unrelated realistic outputs.

CFID better identifies cases with diverse outputs despite realistic individual samples.

Numerical comparisons show CFID's effectiveness in performance evaluation.

Abstract

We consider distance functions between conditional distributions. We focus on the Wasserstein metric and its Gaussian case known as the Frechet Inception Distance (FID). We develop conditional versions of these metrics, analyze their relations and provide a closed form solution to the conditional FID (CFID) metric. We numerically compare the metrics in the context of performance evaluation of modern conditional generative models. Our results show the advantages of CFID compared to the classical FID and mean squared error (MSE) measures. In contrast to FID, CFID is useful in identifying failures where realistic outputs which are not related to their inputs are generated. On the other hand, compared to MSE, CFID is useful in identifying failures where a single realistic output is generated even though there is a diverse set of equally probable outputs.