SEE: Semantically Aligned EEG-to-Text Translation

TL;DR

This paper introduces SEE, a novel EEG-to-Text translation method that leverages a pre-trained language model with specialized modules to improve semantic alignment and reduce domain gap, advancing brain-computer interface capabilities.

Contribution

The paper presents a new approach integrating a Cross-Modal Codebook and Semantic Matching Module into a pre-trained BART model for improved EEG-to-Text decoding.

Findings

Enhanced decoding accuracy on ZuCo corpus

Effective domain gap mitigation between EEG and text

Robust semantic alignment despite data noise

Abstract

Decoding neurophysiological signals into language is of great research interest within brain-computer interface (BCI) applications. Electroencephalography (EEG), known for its non-invasiveness, ease of use, and cost-effectiveness, has been a popular method in this field. However, current EEG-to-Text decoding approaches face challenges due to the huge domain gap between EEG recordings and raw texts, inherent data bias, and small closed vocabularies. In this paper, we propose SEE: Semantically Aligned EEG-to-Text Translation, a novel method aimed at improving EEG-to-Text decoding by seamlessly integrating two modules into a pre-trained BART language model. These two modules include (1) a Cross-Modal Codebook that learns cross-modal representations to enhance feature consolidation and mitigate domain gap, and (2) a Semantic Matching Module that fully utilizes pre-trained text representations to align multi-modal features extracted from EEG-Text pairs while considering noise caused by false negatives, i.e., data from different EEG-Text pairs that have similar semantic meanings. Experimental results on the Zurich Cognitive Language Processing Corpus (ZuCo) demonstrate the effectiveness of SEE, which enhances the feasibility of accurate EEG-to-Text decoding.