RAG-based EEG-to-Text Translation Using Deep Learning and LLMs

TL;DR

This paper introduces a novel RAG-based EEG-to-text decoding method that leverages deep learning and large language models to improve sentence-level decoding from EEG signals, outperforming random baselines.

Contribution

The study presents a new retrieval-augmented generation pipeline combining EEG encoding, semantic retrieval, and LLMs for EEG-to-text translation, demonstrating significant improvements on the ZuCo dataset.

Findings

Outperforms random baseline with 30.45% relative improvement in cosine similarity.

Achieves a mean cosine similarity of 0.181 compared to 0.139 for the baseline.

Statistically significant results across nine subjects.

Abstract

The decoding of linguistic information from electroencephalography (EEG) signals remains an extremely challenging problem in brain-computer interface (BCI) research. In particular, sentence-level decoding from EEG is difficult due to the low signal-to-noise ratio of these recordings. Previous studies tackling this problem have typically failed to surpass random baseline performance unless teacher forcing is used during the inference phase. In this work, we propose a retrieval-augmented generation (RAG)-based sentence-level EEG-to-text decoding pipeline that combines an EEG encoder aligned with semantic sentence embeddings, a vector retrieval stage, and a large language model (LLM) to refine retrieved sentences into coherent output. Experiments are conducted on the Zurich Cognitive Language Processing Corpus (ZuCo) dataset, which contains single-trial EEG recordings collected during silent reading. To evaluate whether the system extracts meaningful information from these EEG signals, the results are compared with a random baseline. In nine subjects, the proposed pipeline outperforms the random baseline, achieving a mean cosine similarity of 0.181 +- 0.022 compared to 0.139 +- 0.029 for the baseline, corresponding to a relative improvement of 30.45%. Statistical analysis further confirms that this improvement is significant, following a strict evaluation workflow where inference is performed without access to ground-truth labels.