Quality assessment of brain structural MR images: Comparing generalization of deep learning versus hand-crafted feature-based machine learning methods to new sites

TL;DR

This study compares deep learning and hand-crafted feature-based machine learning methods for assessing brain MRI quality, highlighting their strengths and limitations in generalizing across different imaging sites.

Contribution

It provides a comprehensive evaluation of CNN-based versus traditional ML methods for MRI quality assessment across multiple sites, emphasizing generalization challenges.

Findings

Both methods struggle to generalize to new sites.

MRIQC performs better on unseen sites overall.

CNNQC shows higher sensitivity for poor-quality scans.

Abstract

Quality assessment of brain structural MR images is critical for large-scale neuroimaging studies, where motion artifacts can significantly bias clinical estimates. While visual rating remains the gold standard, it is time-consuming and subjective. This study evaluates the relative performance and generalization capabilities of two prominent Automated Quality Assessment (AQA) methods: MRIQC, which uses hand-crafted image-quality metrics with traditional machine learning, and CNNQC, which utilizes a deep learning (DL) architecture. Using a heterogeneous dataset of 1,098 T1-weighted volumes from 17 different sites, we assessed performance on both seen sites and entirely new sites using a leave-one-site-out (LOSO) approach. Our results indicate that both DL and traditional ML methods struggle to generalize to new scanners or sites. While MRIQC generally achieved higher accuracy across most unseen sites, CNNQC demonstrated higher sensitivity for detecting poor-quality scans. Given that DL-based methods like CNNQC offer higher computational efficiency and do not require expensive pre-processing, they may be preferred for widespread deployment, provided that future work focuses on improving cross-site generalizability.