The Repeated-Stimulus Confound in Electroencephalography

TL;DR

This paper identifies a confound in neural-decoding EEG studies caused by repeated stimuli, which leads to overestimated accuracy and potentially invalidates some scientific claims.

Contribution

It highlights the repeated-stimulus confound in EEG decoding studies, quantifies its impact on reported accuracies, and demonstrates its implications for scientific validity.

Findings

Decoding accuracies were overestimated by 4.46-7.42%.

The overestimation increases by 0.26% per 1% accuracy increase.

The confound can be exploited to support pseudoscientific claims.

Abstract

In neural-decoding studies, recordings of participants' responses to stimuli are used to train models. In recent years, there has been an explosion of publications detailing applications of innovations from deep-learning research to neural-decoding studies. The data-hungry models used in these experiments have resulted in a demand for increasingly large datasets. Consequently, in some studies, the same stimuli are presented multiple times to each participant to increase the number of trials available for use in model training. However, when a decoding model is trained and subsequently evaluated on responses to the same stimuli, stimulus identity becomes a confounder for accuracy. We term this the repeated-stimulus confound. We identify a susceptible dataset, and 16 publications which report model performance based on evaluation procedures affected by the confound. We conducted experiments using models from the affected studies to investigate the likely extent to which results in the literature have been misreported. Our findings suggest that the decoding accuracies of these models were overestimated by between 4.46-7.42%. Our analysis also indicates that per 1% increase in accuracy under the confound, the magnitude of the overestimation increases by 0.26%. The confound not only results in optimistic estimates of decoding performance, but undermines the validity of several claims made within the affected publications. We conducted further experiments to investigate the implications of the confound in alternative contexts. We found that the same methodology used within the affected studies could also be used to justify an array of pseudoscientific claims, such as the existence of extrasensory perception.