Same Brain, Different Prediction: How Preprocessing Choices Undermine EEG Decoding Reliability

TL;DR

This paper reveals that EEG decoding results are highly sensitive to preprocessing choices, which can cause significant prediction variability, and introduces tools to measure and reduce this instability.

Contribution

It formalizes preprocessing as a counterfactual intervention space and provides methods to quantify and mitigate EEG prediction instability due to preprocessing.

Findings

Up to 42% of trial predictions flip with different preprocessing.

Sensitivity to preprocessing is near-additive in the intervention space.

Introduces Preprocessing Uncertainty and NA-PGI as tools to measure and reduce instability.

Abstract

Electroencephalography (EEG) is a cornerstone of brain-computer interfaces and clinical neuroscience, yet deep learning models are typically trained and evaluated under a single, unreported preprocessing pipeline. We formalize preprocessing choices as a counterfactual intervention space and show that EEG predictions are surprisingly unstable under this space: across six datasets spanning four paradigms, up to 42% of trial-level predictions flip when only the preprocessing changes, a variability that standard uncertainty methods do not explicitly quantify because they condition on a fixed preprocessing pipeline. We provide three tools to make this instability measurable, decomposable, and reducible. First, a Walsh-Hadamard decomposition of the 2^7 pipeline space reveals that sensitivity is near-additive in practice under the binary intervention design, enabling efficient step-by-step optimization. Second, we introduce Preprocessing Uncertainty (PU), a per-trial diagnostic that captures a dimension of instability complementary to model-based confidence. Third, we study Normalized Adaptive PGI (NA-PGI), a graph-structured regularizer that exploits the compositional structure of preprocessing interventions as one mitigation strategy with clear scope conditions.