On the challenges of detecting MCI using EEG in the wild

TL;DR

This paper investigates the challenges of detecting Mild Cognitive Impairment using EEG data in real-world settings, highlighting issues with dataset size, distribution shifts, and feature overlap that hinder reliable diagnosis.

Contribution

It identifies key limitations in current EEG-based MCI detection methods, emphasizing the need for high-quality, real-world data collection to improve robustness and practicality.

Findings

Small datasets lead to high variance, overconfident models

Distribution shifts hinder cross-domain generalization

Feature overlap limits detection accuracy

Abstract

Recent studies have shown promising results in the detection of Mild Cognitive Impairment (MCI) using easily accessible Electroencephalogram (EEG) data which would help administer early and effective treatment for dementia patients. However, the reliability and practicality of such systems remains unclear. In this work, we investigate the potential limitations and challenges in developing a robust MCI detection method using two contrasting datasets: 1) CAUEEG, collected and annotated by expert neurologists in controlled settings and 2) GENEEG, a new dataset collected and annotated in general practice clinics, a setting where routine MCI diagnoses are typically made. We find that training on small datasets, as is done by most previous works, tends to produce high variance models that make overconfident predictions, and are unreliable in practice. Additionally, distribution shifts between datasets make cross-domain generalization challenging. Finally, we show that MCI detection using EEG may suffer from fundamental limitations because of the overlapping nature of feature distributions with control groups. We call for more effort in high-quality data collection in actionable settings (like general practice clinics) to make progress towards this salient goal of non-invasive MCI detection.