The Role of Review Process Failures in Affective State Estimation: An Empirical Investigation of DEAP Dataset

TL;DR

This study critically examines the reliability of affective state estimation using EEG data from the DEAP dataset, revealing widespread methodological flaws that inflate reported accuracy and undermine reproducibility.

Contribution

It provides an empirical analysis of common methodological issues in EEG-based emotion recognition studies and highlights the impact of these flaws on reported performance.

Findings

87% of reviewed studies had methodological errors

Flaws can inflate accuracy by up to 46%

Highlights need for standardized evaluation protocols

Abstract

The reliability of affective state estimation using EEG data is in question, given the variability in reported performance and the lack of standardized evaluation protocols. To investigate this, we reviewed 101 studies, focusing on the widely used DEAP dataset for emotion recognition. Our analysis revealed widespread methodological issues that include data leakage from improper segmentation, biased feature selection, flawed hyperparameter optimization, neglect of class imbalance, and insufficient methodological reporting. Notably, we found that nearly 87% of the reviewed papers contained one or more of these errors. Moreover, through experimental analysis, we observed that such methodological flaws can inflate the classification accuracy by up to 46%. These findings reveal fundamental gaps in standardized evaluation practices and highlight critical deficiencies in the peer review process for machine learning applications in neuroscience, emphasizing the urgent need for stricter methodological standards and evaluation protocols.