Overfitting the literature to one set of stimuli and data

TL;DR

The paper highlights the risk of overfitting in Computational Cognitive Neuroscience due to repeated use of limited stimuli and datasets, emphasizing the need for diverse data collection to ensure generalizable and robust findings.

Contribution

It identifies the problem of overfitting caused by data reuse and advocates for broader data collection to improve the reliability of research in the field.

Findings

Overfitting is a significant risk in current research practices.

Limited stimuli and datasets hinder the generalizability of findings.

Diverse, high-quality open datasets are urgently needed.

Abstract

The fast-growing field of Computational Cognitive Neuroscience is on track to meet its first crisis. A large number of papers in this nascent field are developing and testing novel analysis methods using the same stimuli and neuroimaging datasets. Publication bias and confirmatory exploration will result in overfitting to the limited available data. The field urgently needs to collect more good quality open neuroimaging data using a variety of experimental stimuli, to test the generalisability of current published results, and allow for more robust results in future work.