Cross-domain EEG-based Emotion Recognition with Contrastive Learning

TL;DR

This paper presents EmotionCLIP, a novel EEG-text matching framework using contrastive learning and a specialized backbone, achieving state-of-the-art cross-domain emotion recognition accuracy on benchmark datasets.

Contribution

It introduces a new multimodal contrastive learning approach with a tailored backbone for improved cross-domain EEG emotion recognition.

Findings

Achieved cross-subject accuracy of 88.69% on SEED.

Achieved cross-time accuracy of 88.46% on SEED.

Outperformed existing models in experiments.

Abstract

Electroencephalogram (EEG)-based emotion recognition is vital for affective computing but faces challenges in feature utilization and cross-domain generalization. This work introduces EmotionCLIP, which reformulates recognition as an EEG-text matching task within the CLIP framework. A tailored backbone, SST-LegoViT, captures spatial, spectral, and temporal features using multi-scale convolution and Transformer modules. Experiments on SEED and SEED-IV datasets show superior cross-subject accuracies of 88.69\% and 73.50\%, and cross-time accuracies of 88.46\% and 77.54\%, outperforming existing models. Results demonstrate the effectiveness of multimodal contrastive learning for robust EEG emotion recognition. The code is available at https://github.com/Departure2021/EmotionCLIP.