Reliable Evaluation of MRI Motion Correction: Dataset and Insights

TL;DR

This paper introduces a new dataset and metric for evaluating MRI motion correction methods, highlighting the advantages of real-world evaluation combined with a feature-space metric over simulated or reference-free approaches.

Contribution

It provides PMoC3D, a dataset of real-world motion-corrupted MRI scans, and MoMRISim, a new feature-space metric for more reliable evaluation of correction methods.

Findings

Real-world evaluation with MoMRISim is most reliable.

Simulated motion evaluation overestimates performance.

Reference-free evaluation tends to overrate deep learning outputs.

Abstract

Correcting motion artifacts in MRI is important, as they can hinder accurate diagnosis. However, evaluating deep learning-based and classical motion correction methods remains fundamentally difficult due to the lack of accessible ground-truth target data. To address this challenge, we study three evaluation approaches: real-world evaluation based on reference scans, simulated motion, and reference-free evaluation, each with its merits and shortcomings. To enable evaluation with real-world motion artifacts, we release PMoC3D, a dataset consisting of unprocessed Paired Motion-Corrupted 3D brain MRI data. To advance evaluation quality, we introduce MoMRISim, a feature-space metric trained for evaluating motion reconstructions. We assess each evaluation approach and find real-world evaluation together with MoMRISim, while not perfect, to be most reliable. Evaluation based on simulated motion systematically exaggerates algorithm performance, and reference-free evaluation overrates oversmoothed deep learning outputs.