Identifiability in Functional Connectivity May Unintentionally Inflate Prediction Results

TL;DR

This paper reveals that treating multiple scans of the same subject as independent data can significantly inflate classification accuracy in fMRI studies, highlighting a critical confound in current neuroimaging research.

Contribution

It uncovers how data double-dipping inflates prediction results and demonstrates this effect across multiple large neuroimaging datasets, emphasizing the need for careful data handling.

Findings

Inflation of accuracy from 61% to 86% by treating same-subject scans as independent.

Achieving similar variance explained with fewer subjects using identifiability.

Effect observed across four diverse neuroimaging datasets.

Abstract

Functional magnetic resonance (fMRI) is an invaluable tool in studying cognitive processes in vivo. Many recent studies use functional connectivity (FC), partial correlation connectivity (PC), or fMRI-derived brain networks to predict phenotypes with results that sometimes cannot be replicated. At the same time, FC can be used to identify the same subject from different scans with great accuracy. In this paper, we show a method by which one can unknowingly inflate classification results from 61% accuracy to 86% accuracy by treating longitudinal or contemporaneous scans of the same subject as independent data points. Using the UK Biobank dataset, we find one can achieve the same level of variance explained with 50 training subjects by exploiting identifiability as with 10,000 training subjects without double-dipping. We replicate this effect in four different datasets: the UK Biobank (UKB), the Philadelphia Neurodevelopmental Cohort (PNC), the Bipolar and Schizophrenia Network for Intermediate Phenotypes (BSNIP), and an OpenNeuro Fibromyalgia dataset (Fibro). The unintentional improvement ranges between 7% and 25% in the four datasets. Additionally, we find that by using dynamic functional connectivity (dFC), one can apply this method even when one is limited to a single scan per subject. One major problem is that features such as ROIs or connectivities that are reported alongside inflated results may confuse future work. This article hopes to shed light on how even minor pipeline anomalies may lead to unexpectedly superb results.