Neural2Speech: A Transfer Learning Framework for Neural-Driven Speech Reconstruction

TL;DR

Neural2Speech introduces a transfer learning framework that enables high-quality speech reconstruction from limited neural data by pre-training on speech corpora and adapting to neural recordings, advancing brain-computer interface communication.

Contribution

The paper presents a novel two-phase transfer learning approach that significantly improves speech reconstruction from small neural datasets compared to prior methods.

Findings

Effective reconstruction with only 20 minutes of neural data

Outperforms existing baseline methods in speech fidelity

Demonstrates feasibility of neural-driven speech synthesis

Abstract

Reconstructing natural speech from neural activity is vital for enabling direct communication via brain-computer interfaces. Previous efforts have explored the conversion of neural recordings into speech using complex deep neural network (DNN) models trained on extensive neural recording data, which is resource-intensive under regular clinical constraints. However, achieving satisfactory performance in reconstructing speech from limited-scale neural recordings has been challenging, mainly due to the complexity of speech representations and the neural data constraints. To overcome these challenges, we propose a novel transfer learning framework for neural-driven speech reconstruction, called Neural2Speech, which consists of two distinct training phases. First, a speech autoencoder is pre-trained on readily available speech corpora to decode speech waveforms from the encoded speech representations. Second, a lightweight adaptor is trained on the small-scale neural recordings to align the neural activity and the speech representation for decoding. Remarkably, our proposed Neural2Speech demonstrates the feasibility of neural-driven speech reconstruction even with only 20 minutes of intracranial data, which significantly outperforms existing baseline methods in terms of speech fidelity and intelligibility.