BiMoE: Brain-Inspired Experts for EEG-Dominant Affective State Recognition

TL;DR

BiMoE is a brain-inspired mixture of experts framework that improves EEG and physiological signal fusion for affective state recognition, offering better interpretability and accuracy in multimodal sentiment analysis.

Contribution

It introduces a brain-topology-aware partitioning of EEG signals and a dynamic expert fusion mechanism, advancing multimodal sentiment analysis methods.

Findings

BiMoE outperforms state-of-the-art baselines on DEAP and DREAMER datasets.

Achieves accuracy improvements of 0.87% to 5.19% in multimodal sentiment classification.

Demonstrates effective local and global feature extraction from EEG signals.

Abstract

Multimodal Sentiment Analysis (MSA) that integrates Electroencephalogram (EEG) with peripheral physiological signals (PPS) is crucial for the development of brain-computer interface (BCI) systems. However, existing methods encounter three major challenges: (1) overlooking the region-specific characteristics of affective processing by treating EEG signals as homogeneous; (2) treating EEG as a black-box input, which lacks interpretability into neural representations;(3) ineffective fusion of EEG features with complementary PPS features. To overcome these issues, we propose BiMoE, a novel brain-inspired mixture of experts framework. BiMoE partitions EEG signals in a brain-topology-aware manner, with each expert utilizing a dual-stream encoder to extract local and global spatiotemporal features. A dedicated expert handles PPS using multi-scale large-kernel convolutions. All experts are dynamically fused through adaptive routing and a joint loss function. Evaluated under strict subject-independent settings, BiMoE consistently surpasses state-of-the-art baselines across various affective dimensions. On the DEAP and DREAMER datasets, it yields average accuracy improvements of 0.87% to 5.19% in multimodal sentiment classification. The code is available at: https://github.com/HongyuZhu-s/BiMo.