Rethinking Perceptual Metrics for Medical Image Translation

TL;DR

This paper critically evaluates existing perceptual metrics for medical image translation, revealing their limitations and highlighting the need for more suitable evaluation tools tailored to medical imaging's unique constraints.

Contribution

The study systematically assesses common perceptual metrics on medical image translation tasks, identifying their shortcomings and proposing the potential utility of the pixel-level SWD metric.

Findings

Perceptual metrics poorly correlate with segmentation quality in medical images.

FID scores are inconsistent and unreliable for medical image translation evaluation.

Pixel-level SWD may be useful for intra-modality translation assessment.

Abstract

Modern medical image translation methods use generative models for tasks such as the conversion of CT images to MRI. Evaluating these methods typically relies on some chosen downstream task in the target domain, such as segmentation. On the other hand, task-agnostic metrics are attractive, such as the network feature-based perceptual metrics (e.g., FID) that are common to image translation in general computer vision. In this paper, we investigate evaluation metrics for medical image translation on two medical image translation tasks (GE breast MRI to Siemens breast MRI and lumbar spine MRI to CT), tested on various state-of-the-art translation methods. We show that perceptual metrics do not generally correlate with segmentation metrics due to them extending poorly to the anatomical constraints of this sub-field, with FID being especially inconsistent. However, we find that the lesser-used pixel-level SWD metric may be useful for subtle intra-modality translation. Our results demonstrate the need for further research into helpful metrics for medical image translation.