BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding

TL;DR

BELT-2 is a novel multi-task model that significantly improves EEG-to-language decoding by integrating byte-pair encoding, multi-task training, and prefix-tuning with large language models, enabling decoding of coherent sentences from brain signals.

Contribution

It introduces the first EEG-language alignment at the byte-pair encoding level and combines multi-task training with prefix-tuning to enhance brain decoding performance.

Findings

Achieved BLEU-1 score of 52.2% on ZuCo dataset.

Significant improvements (31%-162%) on multiple translation benchmarks.

First to decode coherent, readable sentences from non-invasive EEG signals.

Abstract

The remarkable success of large language models (LLMs) across various multi-modality applications is well established. However, integrating large language models with humans, or brain dynamics, remains relatively unexplored. In this paper, we introduce BELT-2, a pioneering multi-task model designed to enhance both encoding and decoding performance from EEG signals. To bolster the quality of the EEG encoder, BELT-2 is the first work to innovatively 1) adopt byte-pair encoding (BPE)-level EEG-language alignment and 2) integrate multi-task training and decoding in the EEG domain. Inspired by the idea of \textbf{\textit{Bridging the Brain with GPT}}, we further connect the multi-task EEG encoder with LLMs by utilizing prefix-tuning on intermediary output from the EEG encoder. These innovative efforts make BELT-2 a pioneering breakthrough, making it the first work in the field capable of decoding coherent and readable sentences from non-invasive brain signals. Our experiments highlight significant advancements over prior techniques in both quantitative and qualitative measures, achieving a decoding performance with a BLEU-1 score of 52.2\% on the ZuCo dataset. Furthermore, BELT-2 shows a remarkable improvement ranging from 31\% to 162\% on other translation benchmarks. Codes can be accessed via the provided anonymous link~\footnote{https://anonymous.4open.science/r/BELT-2-0048}.