SSM2Mel: State Space Model to Reconstruct Mel Spectrogram from the EEG

TL;DR

This paper introduces SSM2Mel, a novel state space model that reconstructs continuous speech mel spectrograms from EEG signals, significantly improving decoding accuracy for imagined speech.

Contribution

The paper presents a new SSM-based framework with modules like Mamba, S4-UNet, and ESM to enhance continuous speech reconstruction from EEG, addressing previous limitations.

Findings

Achieved a Pearson correlation of 0.069, a 38% improvement over baseline.

Effectively models long EEG sequences with the Mamba module.

Enhances local feature extraction using S4-UNet.

Abstract

Decoding speech from brain signals is a challenging research problem that holds significant importance for studying speech processing in the brain. Although breakthroughs have been made in reconstructing the mel spectrograms of audio stimuli perceived by subjects at the word or letter level using noninvasive electroencephalography (EEG), there is still a critical gap in precisely reconstructing continuous speech features, especially at the minute level. To address this issue, this paper proposes a State Space Model (SSM) to reconstruct the mel spectrogram of continuous speech from EEG, named SSM2Mel. This model introduces a novel Mamba module to effectively model the long sequence of EEG signals for imagined speech. In the SSM2Mel model, the S4-UNet structure is used to enhance the extraction of local features of EEG signals, and the Embedding Strength Modulator (ESM) module is used to incorporate subject-specific information. Experimental results show that our model achieves a Pearson correlation of 0.069 on the SparrKULee dataset, which is a 38% improvement over the previous baseline.