Five Pitfalls When Assessing Synthetic Medical Images with Reference Metrics

TL;DR

This paper discusses five common pitfalls in using reference-based image quality metrics for evaluating synthetic medical images, highlighting issues that can lead to misleading assessments and proposing strategies to avoid them.

Contribution

It identifies and explains five specific pitfalls in applying reference metrics to medical image evaluation, emphasizing the need for careful interpretation.

Findings

Reference metrics can produce misleading scores in medical image evaluation.

Different distortions affect metric correlation with human perception variably.

Strategies are proposed to mitigate pitfalls in using reference metrics.

Abstract

Reference metrics have been developed to objectively and quantitatively compare two images. Especially for evaluating the quality of reconstructed or compressed images, these metrics have shown very useful. Extensive tests of such metrics on benchmarks of artificially distorted natural images have revealed which metric best correlate with human perception of quality. Direct transfer of these metrics to the evaluation of generative models in medical imaging, however, can easily lead to pitfalls, because assumptions about image content, image data format and image interpretation are often very different. Also, the correlation of reference metrics and human perception of quality can vary strongly for different kinds of distortions and commonly used metrics, such as SSIM, PSNR and MAE are not the best choice for all situations. We selected five pitfalls that showcase unexpected and probably undesired reference metric scores and discuss strategies to avoid them.