Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects

TL;DR

This empirical study shows that scanner effects significantly impact machine learning results on multi-site neuroimaging data, with current harmonization methods failing to fully remove biases and affecting model generalization.

Contribution

It provides a comprehensive empirical analysis of scanner effects on multi-site MRI data and highlights the limitations of existing harmonization techniques.

Findings

Classifiers can distinguish data origin with high accuracy despite pre-processing.

Current harmonization methods do not fully eliminate scanner-specific biases.

Scanner effects can lead to overly optimistic performance estimates.

Abstract

This is an empirical study to investigate the impact of scanner effects when using machine learning on multi-site neuroimaging data. We utilize structural T1-weighted brain MRI obtained from two different studies, Cam-CAN and UK Biobank. For the purpose of our investigation, we construct a dataset consisting of brain scans from 592 age- and sex-matched individuals, 296 subjects from each original study. Our results demonstrate that even after careful pre-processing with state-of-the-art neuroimaging pipelines a classifier can easily distinguish between the origin of the data with very high accuracy. Our analysis on the example application of sex classification suggests that current approaches to harmonize data are unable to remove scanner-specific bias leading to overly optimistic performance estimates and poor generalization. We conclude that multi-site data harmonization remains an open challenge and particular care needs to be taken when using such data with advanced machine learning methods for predictive modelling.