Neural Decoding of Overt Speech from ECoG Using Vision Transformers and Contrastive Representation Learning

TL;DR

This paper introduces a novel neural decoding pipeline using Vision Transformers and contrastive learning to directly reconstruct speech from surface ECoG signals, demonstrating potential for long-term, wireless BCI applications.

Contribution

It presents the first attempt to decode speech from a fully implantable wireless epidural ECoG system using advanced deep learning techniques.

Findings

Successful speech reconstruction from ECoG signals.

Effective use of Vision Transformers and contrastive learning.

Potential for long-term wireless BCI speech decoding.

Abstract

Speech Brain Computer Interfaces (BCIs) offer promising solutions to people with severe paralysis unable to communicate. A number of recent studies have demonstrated convincing reconstruction of intelligible speech from surface electrocorticographic (ECoG) or intracortical recordings by predicting a series of phonemes or words and using downstream language models to obtain meaningful sentences. A current challenge is to reconstruct speech in a streaming mode by directly regressing cortical signals into acoustic speech. While this has been achieved recently using intracortical data, further work is needed to obtain comparable results with surface ECoG recordings. In particular, optimizing neural decoders becomes critical in this case. Here we present an offline speech decoding pipeline based on an encoder-decoder deep neural architecture, integrating Vision Transformers and contrastive learning to enhance the direct regression of speech from ECoG signals. The approach is evaluated on two datasets, one obtained with clinical subdural electrodes in an epileptic patient, and another obtained with the fully implantable WIMAGINE epidural system in a participant of a motor BCI trial. To our knowledge this presents a first attempt to decode speech from a fully implantable and wireless epidural recording system offering perspectives for long-term use.