Extracting Different Levels of Speech Information from EEG Using an LSTM-Based Model

TL;DR

This paper investigates how an LSTM-based neural network decodes various levels of speech information from EEG signals, revealing that it primarily utilizes features like silences, intensity, and phonetic classes, with mel spectrograms providing the best performance.

Contribution

The study analyzes the specific speech features that an LSTM model uses to decode EEG signals, highlighting the importance of certain features like mel spectrograms for accurate speech information extraction.

Findings

The model exploits silences, intensity, and broad phonetic classes from EEG.

Mel spectrogram features yield the highest decoding accuracy at 84%.

Different speech features contribute variably to EEG-based speech decoding.

Abstract

Decoding the speech signal that a person is listening to from the human brain via electroencephalography (EEG) can help us understand how our auditory system works. Linear models have been used to reconstruct the EEG from speech or vice versa. Recently, Artificial Neural Networks (ANNs) such as Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) based architectures have outperformed linear models in modeling the relation between EEG and speech. Before attempting to use these models in real-world applications such as hearing tests or (second) language comprehension assessment we need to know what level of speech information is being utilized by these models. In this study, we aim to analyze the performance of an LSTM-based model using different levels of speech features. The task of the model is to determine which of two given speech segments is matched with the recorded EEG. We used low- and high-level speech features including: envelope, mel spectrogram, voice activity, phoneme identity, and word embedding. Our results suggest that the model exploits information about silences, intensity, and broad phonetic classes from the EEG. Furthermore, the mel spectrogram, which contains all this information, yields the highest accuracy (84%) among all the features.