Assessing the Distributional Fidelity of Synthetic Chest X-rays using the Embedded Characteristic Score

TL;DR

This paper introduces the embedded characteristic score (ECS), a new evaluation method for synthetic chest X-ray images that detects distributional differences more effectively than existing metrics, ensuring better quality assessment.

Contribution

The paper proposes ECS, a flexible, theoretically grounded evaluation metric that captures higher-order distributional differences in synthetic medical images, improving upon existing methods like FID.

Findings

ECS effectively detects clinically relevant distributional discrepancies.

ECS outperforms FID in sensitivity to higher moments and distribution tails.

ECS provides a reliable evaluation tool for high-stakes medical image synthesis.

Abstract

Chest X-ray (CXR) images are among the most commonly used diagnostic imaging modalities in clinical practice. Stringent privacy constraints often limit the public dissemination of patient CXR images, contributing to the increasing use of synthetic images produced by deep generative models for data sharing and training machine learning models. Given the high-stakes downstream applications of CXR images, it is crucial to evaluate how faithfully synthetic images reflect the underlying target distribution. We propose the embedded characteristic score (ECS), a flexible evaluation procedure that compares synthetic and patient CXR samples through characteristic function transforms of feature embeddings. The choice of embedding can be tailored to the clinical or scientific context of interest. By leveraging the behavior of characteristic functions near the origin, ECS is sensitive to differences in higher moments and distribution tails, aspects that are often overlooked by commonly used evaluation metrics such as the Fr\'echet Inception Distance (FID). We establish theoretical properties of ECS and describe a calibration strategy based on a simple resampling procedure. We compare the empirical performance of ECS against FID via simulations and standard benchmark imaging datasets. Assessing synthetic CXR images with ECS uncovers clinically relevant distributional discrepancies relative to patient CXR images. These results highlight the importance of reliable evaluation of synthetic data that inform high-stakes decisions.