Similarity and Quality Metrics for MR Image-To-Image Translation

TL;DR

This paper evaluates 23 similarity and quality metrics for assessing synthetic MR images, analyzing their sensitivity to distortions and artifacts to guide effective evaluation of image translation models.

Contribution

It provides a comprehensive quantitative analysis of multiple metrics' sensitivity to distortions in MR images, offering practical recommendations for their use.

Findings

SSIM and PSNR are less sensitive to certain distortions.

Non-reference metrics effectively detect specific artifacts.

Normalization methods significantly influence metric performance.

Abstract

Image-to-image translation can create large impact in medical imaging, as images can be synthetically transformed to other modalities, sequence types, higher resolutions or lower noise levels. To ensure patient safety, these methods should be validated by human readers, which requires a considerable amount of time and costs. Quantitative metrics can effectively complement such studies and provide reproducible and objective assessment of synthetic images. If a reference is available, the similarity of MR images is frequently evaluated by SSIM and PSNR metrics, even though these metrics are not or too sensitive regarding specific distortions. When reference images to compare with are not available, non-reference quality metrics can reliably detect specific distortions, such as blurriness. To provide an overview on distortion sensitivity, we quantitatively analyze 11 similarity (reference) and 12 quality (non-reference) metrics for assessing synthetic images. We additionally include a metric on a downstream segmentation task. We investigate the sensitivity regarding 11 kinds of distortions and typical MR artifacts, and analyze the influence of different normalization methods on each metric and distortion. Finally, we derive recommendations for effective usage of the analyzed similarity and quality metrics for evaluation of image-to-image translation models.