Independent Feature Enhanced Crossmodal Fusion for Match-Mismatch Classification of Speech Stimulus and EEG Response

TL;DR

This paper introduces IFE-CF, a novel crossmodal fusion model that improves match-mismatch classification of speech stimuli and EEG responses by leveraging a crossmodal encoder, fusion modules, and causal masking.

Contribution

The proposed IFE-CF model innovatively combines crossmodal attention, feature fusion, and causal masking to enhance auditory EEG decoding accuracy.

Findings

Achieves higher classification accuracy than baseline methods.

Effectively models the relationship between speech and EEG signals.

Demonstrates robustness in auditory attention decoding.

Abstract

It is crucial for auditory attention decoding to classify matched and mismatched speech stimuli with corresponding EEG responses by exploring their relationship. However, existing methods often adopt two independent networks to encode speech stimulus and EEG response, which neglect the relationship between these signals from the two modalities. In this paper, we propose an independent feature enhanced crossmodal fusion model (IFE-CF) for match-mismatch classification, which leverages the fusion feature of the speech stimulus and the EEG response to achieve auditory EEG decoding. Specifically, our IFE-CF contains a crossmodal encoder to encode the speech stimulus and the EEG response with a two-branch structure connected via crossmodal attention mechanism in the encoding process, a multi-channel fusion module to fuse features of two modalities by aggregating the interaction feature obtained from the crossmodal encoder and the independent feature obtained from the speech stimulus and EEG response, and a predictor to give the matching result. In addition, the causal mask is introduced to consider the time delay of the speech-EEG pair in the crossmodal encoder, which further enhances the feature representation for match-mismatch classification. Experiments demonstrate our method's effectiveness with better classification accuracy, as compared with the baseline of the Auditory EEG Decoding Challenge 2023.