NeuSpeech: Decode Neural signal as Speech

TL;DR

This paper introduces NeuSpeech, a novel cross-attention-based model for decoding speech directly from non-invasive MEG neural signals, achieving high accuracy without teacher forcing and advancing brain-computer interface capabilities.

Contribution

First to apply a cross-attention 'whisper' model for MEG-based speech decoding without teacher forcing, outperforming previous methods on major datasets.

Findings

Achieved BLEU-1 scores of 60.30 and 52.89 on two datasets.

Demonstrated effectiveness of the model without pretraining or teacher forcing.

Provided comprehensive analysis of neural speech decoding techniques.

Abstract

Decoding language from brain dynamics is an important open direction in the realm of brain-computer interface (BCI), especially considering the rapid growth of large language models. Compared to invasive-based signals which require electrode implantation surgery, non-invasive neural signals (e.g. EEG, MEG) have attracted increasing attention considering their safety and generality. However, the exploration is not adequate in three aspects: 1) previous methods mainly focus on EEG but none of the previous works address this problem on MEG with better signal quality; 2) prior works have predominantly used $``teacher-forcing"$ during generative decoding, which is impractical; 3) prior works are mostly $``BART-based"$ not fully auto-regressive, which performs better in other sequence tasks. In this paper, we explore the brain-to-text translation of MEG signals in a speech-decoding formation. Here we are the first to investigate a cross-attention-based ``whisper" model for generating text directly from MEG signals without teacher forcing. Our model achieves impressive BLEU-1 scores of 60.30 and 52.89 without pretraining $\&$ teacher-forcing on two major datasets ($\textit{GWilliams}$ and $\textit{Schoffelen}$). This paper conducts a comprehensive review to understand how speech decoding formation performs on the neural decoding tasks, including pretraining initialization, training $\&$ evaluation set splitting, augmentation, and scaling law. Code is available at https://github.com/NeuSpeech/NeuSpeech1$.