Brain2Char: A Deep Architecture for Decoding Text from Brain Recordings

TL;DR

This paper introduces Brain2Char, a deep learning architecture that decodes text directly from brain recordings, achieving state-of-the-art accuracy and enabling advanced brain-computer communication.

Contribution

The study presents a novel deep network combining 3D Inception, bidirectional RNNs, and language models with regularizations based on cortical speech encoding, advancing brain-to-text decoding.

Findings

Achieved 7-10.6% Word Error Rate on 1200-1900 word vocabulary

Performed well in silent miming tasks

Set new state-of-the-art in brain-to-text decoding

Abstract

Decoding language representations directly from the brain can enable new Brain-Computer Interfaces (BCI) for high bandwidth human-human and human-machine communication. Clinically, such technologies can restore communication in people with neurological conditions affecting their ability to speak. In this study, we propose a novel deep network architecture Brain2Char, for directly decoding text (specifically character sequences) from direct brain recordings (called Electrocorticography, ECoG). Brain2Char framework combines state-of-the-art deep learning modules --- 3D Inception layers for multiband spatiotemporal feature extraction from neural data and bidirectional recurrent layers, dilated convolution layers followed by language model weighted beam search to decode character sequences, optimizing a connectionist temporal classification (CTC) loss. Additionally, given the highly non-linear transformations that underlie the conversion of cortical function to character sequences, we perform regularizations on the network's latent representations motivated by insights into cortical encoding of speech production and artifactual aspects specific to ECoG data acquisition. To do this, we impose auxiliary losses on latent representations for articulatory movements, speech acoustics and session specific non-linearities. In 3 participants tested here, Brain2Char achieves 10.6\%, 8.5\% and 7.0\% Word Error Rates (WER) respectively on vocabulary sizes ranging from 1200 to 1900 words. Brain2Char also performs well when 2 participants silently mimed sentences. These results set a new state-of-the-art on decoding text from brain and demonstrate the potential of Brain2Char as a high-performance communication BCI.