Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

TL;DR

This paper introduces a novel EEG-to-text decoding framework using pre-trained language models, enabling open vocabulary and zero-shot sentiment classification, surpassing previous methods limited to small vocabularies and invasive devices.

Contribution

The authors propose a new EEG-to-text decoding approach leveraging pre-trained language models for open vocabulary and zero-shot sentiment tasks, extending beyond prior closed-vocabulary, invasive-device-dependent methods.

Findings

Achieved 40.1% BLEU-1 score on EEG-to-text decoding.

Achieved 55.6% F1 score on zero-shot sentiment classification.

Model generalizes across subjects and data sources.

Abstract

State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. However, current approaches are limited to small closed vocabularies which are far from enough for natural communication. In addition, most of the high-performing approaches require data from invasive devices (e.g., ECoG). In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. We hypothesis that the human brain functions as a special text encoder and propose a novel framework leveraging pre-trained language models (e.g., BART). Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for a high-performance open vocabulary brain-to-text system once sufficient data is available