Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding

TL;DR

Neural-MCRL introduces a novel multimodal contrastive learning framework with semantic alignment and a spectral-temporal EEG encoder, significantly improving visual decoding accuracy from EEG signals for brain-machine interfaces.

Contribution

It presents Neural-MCRL, a new framework combining semantic bridging, cross-attention, and a spectral-temporal EEG encoder to enhance neural visual decoding.

Findings

Improved decoding accuracy over state-of-the-art methods

Enhanced model generalization in EEG-based visual decoding

Effective semantic alignment across modalities

Abstract

Decoding neural visual representations from electroencephalogram (EEG)-based brain activity is crucial for advancing brain-machine interfaces (BMI) and has transformative potential for neural sensory rehabilitation. While multimodal contrastive representation learning (MCRL) has shown promise in neural decoding, existing methods often overlook semantic consistency and completeness within modalities and lack effective semantic alignment across modalities. This limits their ability to capture the complex representations of visual neural responses. We propose Neural-MCRL, a novel framework that achieves multimodal alignment through semantic bridging and cross-attention mechanisms, while ensuring completeness within modalities and consistency across modalities. Our framework also features the Neural Encoder with Spectral-Temporal Adaptation (NESTA), a EEG encoder that adaptively captures spectral patterns and learns subject-specific transformations. Experimental results demonstrate significant improvements in visual decoding accuracy and model generalization compared to state-of-the-art methods, advancing the field of EEG-based neural visual representation decoding in BMI. Codes will be available at: https://github.com/NZWANG/Neural-MCRL.