Predicting Different Acoustic Features from EEG and towards direct synthesis of Audio Waveform from EEG

TL;DR

This paper introduces a deep learning model that directly converts raw EEG signals into audio waveforms and predicts multiple acoustic features, advancing speech synthesis from neural signals.

Contribution

It presents a novel deep learning approach for direct EEG-to-audio synthesis and predicts multiple acoustic features from EEG, improving understanding of neural speech representations.

Findings

Deep learning model directly produces audio from EEG signals

Predicts 16 acoustic features from EEG data

Shows relation between EEG signals and acoustic features during speech

Abstract

In [1,2] authors provided preliminary results for synthesizing speech from electroencephalography (EEG) features where they first predict acoustic features from EEG features and then the speech is reconstructed from the predicted acoustic features using griffin lim reconstruction algorithm. In this paper we first introduce a deep learning model that takes raw EEG waveform signals as input and directly produces audio waveform as output. We then demonstrate predicting 16 different acoustic features from EEG features. We demonstrate our results for both spoken and listen condition in this paper. The results presented in this paper shows how different acoustic features are related to non-invasive neural EEG signals recorded during speech perception and production.