Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings

TL;DR

This paper introduces a new two-step method for decoding EEG signals into sentences by learning semantic information with a Conformer encoder and retrieving sentences without relying on language models, improving linguistic BCI applications.

Contribution

It proposes a novel pipeline combining a Conformer encoder trained with a contrastive objective and a training-free sentence retrieval method for EEG-based language decoding.

Findings

The Conformer encoder learns semantic categories from EEG data during natural reading.

The retrieval method effectively matches EEG segments to semantically relevant sentences.

Results show promising grouping of EEG segments into meaningful semantic categories.

Abstract

Decoding linguistic information from non-invasive brain signals using EEG has gained increasing research attention due to its vast applicational potential. Recently, a number of works have adopted a generative-based framework to decode electroencephalogram (EEG) signals into sentences by utilizing the power generative capacity of pretrained large language models (LLMs). However, this approach has several drawbacks that hinder the further development of linguistic applications for brain-computer interfaces (BCIs). Specifically, the ability of the EEG encoder to learn semantic information from EEG data remains questionable, and the LLM decoder's tendency to generate sentences based on its training memory can be hard to avoid. These issues necessitate a novel approach for converting EEG signals into sentences. In this paper, we propose a novel two-step pipeline that addresses these limitations and enhances the validity of linguistic EEG decoding research. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading by training a Conformer encoder via a masked contrastive objective for word-level classification. To achieve sentence decoding results, we employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder. Extensive experiments and ablation studies were conducted in this paper for a comprehensive evaluation of the proposed approach. Visualization of the top prediction candidates reveals that our model effectively groups EEG segments into semantic categories with similar meanings, thereby validating its ability to learn patterns from unspoken EEG recordings. Despite the exploratory nature of this work, these results suggest that our method holds promise for providing more reliable solutions for converting EEG signals into text.