You Don't Have to Be Perfect to Be Amazing: Unveil the Utility of Synthetic Images

TL;DR

This paper evaluates synthetic images from deep generative models across multiple metrics, revealing trade-offs between fidelity, variety, privacy, and utility, and demonstrating high utility can be achieved even with low fidelity or variety images.

Contribution

It introduces a comprehensive evaluation framework for synthetic images, analyzing over 100k chest X-rays to uncover relationships between quality, utility, and privacy, and provides insights for utility-aware generative models.

Findings

Trade-off exists between fidelity, variety, and privacy of synthetic images.

High utility can be achieved with images of low fidelity or variety.

High utility and privacy can coexist in synthetic images.

Abstract

Synthetic images generated from deep generative models have the potential to address data scarcity and data privacy issues. The selection of synthesis models is mostly based on image quality measurements, and most researchers favor synthetic images that produce realistic images, i.e., images with good fidelity scores, such as low Fr\'echet Inception Distance (FID) and high Peak Signal-To-Noise Ratio (PSNR). However, the quality of synthetic images is not limited to fidelity, and a wide spectrum of metrics should be evaluated to comprehensively measure the quality of synthetic images. In addition, quality metrics are not truthful predictors of the utility of synthetic images, and the relations between these evaluation metrics are not yet clear. In this work, we have established a comprehensive set of evaluators for synthetic images, including fidelity, variety, privacy, and utility. By analyzing more than 100k chest X-ray images and their synthetic copies, we have demonstrated that there is an inevitable trade-off between synthetic image fidelity, variety, and privacy. In addition, we have empirically demonstrated that the utility score does not require images with both high fidelity and high variety. For intra- and cross-task data augmentation, mode-collapsed images and low-fidelity images can still demonstrate high utility. Finally, our experiments have also showed that it is possible to produce images with both high utility and privacy, which can provide a strong rationale for the use of deep generative models in privacy-preserving applications. Our study can shore up comprehensive guidance for the evaluation of synthetic images and elicit further developments for utility-aware deep generative models in medical image synthesis.