A Penny for Your Thoughts: Decoding Speech from Inexpensive Brain Signals

TL;DR

This paper demonstrates that neural networks can decode speech from inexpensive EEG brain signals by aligning EEG embeddings with speech representations, with personalized model modifications improving accuracy for brain-computer interface applications.

Contribution

The study introduces three architectural enhancements to EEG-to-speech decoding models, improving performance and advancing brain-computer interface technology.

Findings

Subject-specific attention layers improved WER by 0.15%.

Personalized spatial attention improved WER by 0.45%.

Dual-path RNN with attention improved WER by 1.87%.

Abstract

We explore whether neural networks can decode brain activity into speech by mapping EEG recordings to audio representations. Using EEG data recorded as subjects listened to natural speech, we train a model with a contrastive CLIP loss to align EEG-derived embeddings with embeddings from a pre-trained transformer-based speech model. Building on the state-of-the-art EEG decoder from Meta, we introduce three architectural modifications: (i) subject-specific attention layers (+0.15% WER improvement), (ii) personalized spatial attention (+0.45%), and (iii) a dual-path RNN with attention (-1.87%). Two of the three modifications improved performance, highlighting the promise of personalized architectures for brain-to-speech decoding and applications in brain-computer interfaces.