Investigation into using stochastic embedding representations for evaluating the trustworthiness of the Fr\'{e}chet Inception Distance

TL;DR

This paper explores the use of stochastic embedding representations and predictive variance estimates to evaluate the reliability of the Fréchet Inception Distance (FID) in assessing medical image quality, highlighting its limitations.

Contribution

It introduces a method to quantify the trustworthiness of FID using Monte Carlo dropout-based variance estimates in feature embeddings, especially for medical images.

Findings

Predictive variances correlate with out-of-distribution detection.

Stochastic embedding representations provide insights into FID reliability.

The approach helps identify when FID may be less effective for medical images.

Abstract

Feature embeddings acquired from pretrained models are widely used in medical applications of deep learning to assess the characteristics of datasets; e.g. to determine the quality of synthetic, generated medical images. The Fr\'{e}chet Inception Distance (FID) is one popular synthetic image quality metric that relies on the assumption that the characteristic features of the data can be detected and encoded by an InceptionV3 model pretrained on ImageNet1K (natural images). While it is widely known that this makes it less effective for applications involving medical images, the extent to which the metric fails to capture meaningful differences in image characteristics is not obviously known. Here, we use Monte Carlo dropout to compute the predictive variance in the FID as well as a supplemental estimate of the predictive variance in the feature embedding model's latent representations. We show that the magnitudes of the predictive variances considered exhibit varying degrees of correlation with the extent to which test inputs (ImageNet1K validation set augmented at various strengths, and other external datasets) are out-of-distribution relative to its training data, providing some insight into the effectiveness of their use as indicators of the trustworthiness of the FID.