Systematic Misestimation of Machine Learning Performance in Neuroimaging Studies of Depression

TL;DR

This study reveals that small sample sizes in neuroimaging machine learning studies of depression often lead to overestimated performance metrics, highlighting the importance of larger test sets for valid results.

Contribution

It systematically demonstrates the risk of performance misestimation in small samples and emphasizes the need for larger test sets to improve reliability in neuroimaging ML studies.

Findings

Small samples can produce inflated accuracy estimates.

Large test sets mitigate performance overestimation.

Current literature may overstate ML performance due to small sample sizes.

Abstract

We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from major depressive disorder (MDD) and healthy control (HC) based on neuroimaging data. Drawing upon structural magnetic resonance imaging (MRI) data from a balanced sample of $N = 1,868$ MDD patients and HC from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of $61\,\%$. Next, we mimicked the process by which researchers would draw samples of various sizes ($N = 4$ to $N = 150$) from the population and showed a strong risk of misestimation. Specifically, for small sample sizes ($N = 20$), we observe accuracies of up to $95\,\%$. For medium sample sizes ($N = 100$) accuracies up to $75\,\%$ were found. Importantly, further investigation showed that sufficiently large test sets effectively protect against performance misestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets, which is readily available in most cases.