Neural networks based EEG-Speech Models

TL;DR

This paper introduces an end-to-end neural network framework for translating imagined EEG signals into speech, incorporating language and acoustic features, and demonstrates superior classification performance over traditional methods.

Contribution

The paper presents three novel neural network models for EEG-to-speech translation, integrating language models, acoustic mapping, and RBM-based feature learning, with two augmented models leveraging spoken EEG signals.

Findings

All three models outperform SVM baseline in EEG-speech classification.

Models achieve comparable results to deep belief networks in binary classification.

Proposed models effectively represent multichannel EEG and speech signals.

Abstract

In this paper, we propose an end-to-end neural network (NN) based EEG-speech (NES) modeling framework, in which three network structures are developed to map imagined EEG signals to phonemes. The proposed NES models incorporate a language model based EEG feature extraction layer, an acoustic feature mapping layer, and a restricted Boltzmann machine (RBM) based the feature learning layer. The NES models can jointly realize the representation of multichannel EEG signals and the projection of acoustic speech signals. Among three proposed NES models, two augmented networks utilize spoken EEG signals as either bias or gate information to strengthen the feature learning and translation of imagined EEG signals. Experimental results show that all three proposed NES models outperform the baseline support vector machine (SVM) method on EEG-speech classification. With respect to binary classification, our approach achieves comparable results relative to deep believe network approach.