CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attention

TL;DR

CEReBrO is a compact, efficient EEG foundation model that uses alternating attention to effectively model brain oscillations, achieving high performance on multiple brain activity tasks while being resource-efficient.

Contribution

Introduces CEReBrO, a small EEG foundation model with a novel alternating attention mechanism, addressing limitations of existing models in size, efficiency, and reproducibility.

Findings

Achieves 2x speed and 6x memory reduction compared to standard self-attention.

Sets new benchmarks in emotion and seizure detection tasks.

Demonstrates competitive performance in anomaly classification and gait prediction.

Abstract

Electroencephalograph (EEG) is a crucial tool for studying brain activity. Recently, self-supervised learning methods leveraging large unlabeled datasets have emerged as a potential solution to the scarcity of widely available annotated EEG data. However, current methods suffer from at least one of the following limitations: i) sub-optimal EEG signal modeling, ii) model sizes in the hundreds of millions of trainable parameters, and iii) reliance on private datasets and/or inconsistent public benchmarks, hindering reproducibility. To address these challenges, we introduce a Compact Encoder for Representations of Brain Oscillations using alternating attention (CEReBrO), a new small EEG foundation model. Our tokenization scheme represents EEG signals at a per-channel patch granularity. We propose an alternating attention mechanism that jointly models intra-channel temporal dynamics and inter-channel spatial correlations, achieving 2x speed improvement with 6x less memory required compared to standard self-attention. We present several model sizes ranging from 3.6 million to 85 million parameters. Pre-trained on over 20,000 hours of publicly available scalp EEG recordings with diverse channel configurations, our models set new benchmarks in emotion detection and seizure detection tasks, with competitive performance in anomaly classification and gait prediction. This validates our models' effectiveness and efficiency.