CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding
Jiquan Wang, Sha Zhao, Zhiling Luo, Yangxuan Zhou, Haiteng Jiang, Shijian Li, Tao Li, Gang Pan
TL;DR
CBraMod introduces a novel criss-cross transformer-based EEG foundation model that separately models spatial and temporal dependencies, significantly improving generalizability and performance across diverse BCI tasks.
Contribution
It proposes a new EEG foundation model with a criss-cross transformer and asymmetric positional encoding, addressing heterogeneity and format variability issues.
Findings
Achieves state-of-the-art results on 10 BCI tasks
Demonstrates strong generalizability across diverse datasets
Effectively models spatial and temporal EEG dependencies separately
Abstract
Electroencephalography (EEG) is a non-invasive technique to measure and record brain electrical activity, widely used in various BCI and healthcare applications. Early EEG decoding methods rely on supervised learning, limited by specific tasks and datasets, hindering model performance and generalizability. With the success of large language models, there is a growing body of studies focusing on EEG foundation models. However, these studies still leave challenges: Firstly, most of existing EEG foundation models employ full EEG modeling strategy. It models the spatial and temporal dependencies between all EEG patches together, but ignores that the spatial and temporal dependencies are heterogeneous due to the unique structural characteristics of EEG signals. Secondly, existing EEG foundation models have limited generalizability on a wide range of downstream BCI tasks due to varying…
Peer Reviews
Decision·ICLR 2025 Poster
1. The utilization of a criss-cross transformer to independently model spatial and temporal dependencies is a significant advancement. By partitioning attention mechanisms into Spatial-Attention (S-Attention) and TemporalAttention (T-Attention), CBraMod effectively captures the heterogeneous dependencies in EEG signals, which are often overlooked in traditional full EEG modeling strategies. 2. Asymmetric Conditional Positional Encoding (ACPE): Dynamic Encoding: The ACPE scheme dynamically encod
1. Limited Comparative Analysis with Recent Models. While the paper compares CBraMod with several non-foundation and foundation model baselines, the inclusion of more recent EEG foundation models, e.g. BrainWave (https://arxiv.org/abs/2402.10251) could provide a more comprehensive evaluation of CBraMod’s relative performance. 2. Lack of interpretability. The authors did not provide any kind of interpretation of the obtained models. First of all, it would be very interesting to see what pieces
The paper proposes a novel transformer-based architecture for EEG feature extraction and uses it as a foundation model. Substantial comparative experiments have been conducted to show its good performance on different EEG prediction tasks. The presentation is clear, and superior results have been achieved. It’s a good exploration of using large-scale models to obtain EEG representations.
The current version shows a good architecture for feature extraction instead of a real foundation model. It would be beneficial to give a clearer description of the motivation for proposing such a ‘foundation model’ and how this model facilitates the application of EEG analysis and even brain-computer interfaces. Could we use only a few data for finetuning or tuning specific model faster?
This work addresses the challenges currently faced in EEG models by proposing an effective solution for spatiotemporal modeling through enhancements in model architecture. The model has undergone pretraining on a substantial dataset, and the experimental results are comprehensive, providing support for several contributions.
Due to the pretraining paradigm of CBraMod, it has not effectively addressed the challenges posed by the diversity of EEG data formats. This raises concerns regarding its suitability as a foundation model for EEG. The specific issues are outlined as follows. 1. This work acknowledges that current EEG foundation models still face challenges in handling EEG data of varying formats. However, the solution proposed by CBraMod during the pretraining phase, which involves the selection of 19 common EEG
Code & Models
Videos
Taxonomy
TopicsEEG and Brain-Computer Interfaces
MethodsSoftmax · Attention Is All You Need · Depthwise Convolution · Positional Encoding Generator · Conditional Positional Encoding