ETS: Open Vocabulary Electroencephalography-To-Text Decoding and Sentiment Classification

TL;DR

This paper introduces ETS, a novel EEG-based framework that combines eye-tracking data to improve open-vocabulary text decoding and sentiment classification, outperforming previous methods in accuracy and robustness.

Contribution

The study presents a new EEG and eye-tracking integrated model for open-vocabulary decoding and sentiment analysis, addressing noise and variability challenges in EEG data.

Findings

Achieves higher BLEU and Rouge scores for EEG-to-text decoding.

Attains up to 10% F1 score in sentiment classification.

Handles data across multiple subjects and sources effectively.

Abstract

Decoding natural language from brain activity using non-invasive electroencephalography (EEG) remains a significant challenge in neuroscience and machine learning, particularly for open-vocabulary scenarios where traditional methods struggle with noise and variability. Previous studies have achieved high accuracy on small-closed vocabularies, but it still struggles on open vocabularies. In this study, we propose ETS, a framework that integrates EEG with synchronized eye-tracking data to address two critical tasks: (1) open-vocabulary text generation and (2) sentiment classification of perceived language. Our model achieves a superior performance on BLEU and Rouge score for EEG-To-Text decoding and up to 10% F1 score on EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for high performance open vocabulary eeg-to-text system.