Relating the Neural Representations of Vocalized, Mimed, and Imagined Speech

TL;DR

This study explores the neural similarities among vocalized, mimed, and imagined speech using EEG data, showing shared representations and effective cross-condition decoding with linear models.

Contribution

It demonstrates that neural representations of different speech conditions are shared, enabling cross-condition decoding and revealing the potential for unified speech neural models.

Findings

Linear decoders transfer across speech conditions.

Shared neural representations exist among vocalized, mimed, and imagined speech.

Linear models outperform nonlinear ones in stimulus discriminability.

Abstract

We investigated the relationship among neural representations of vocalized, mimed, and imagined speech recorded using publicly available stereotactic EEG recordings. Most prior studies have focused on decoding speech responses within each condition separately. Here, instead, we explore how responses across conditions relate by training linear spectrogram reconstruction models for each condition and evaluate their generalization across conditions. We demonstrate that linear decoders trained on one condition generally transfer successfully to others, implying shared speech representations. This commonality was assessed with stimulus-level discriminability by performing a rank-based analysis demonstrating preservation of stimulus-specific structure in both within- and across-conditions. Finally, we compared linear reconstructions to those from a nonlinear neural network. While both exhibited cross-condition transfer, linear models achieve superior stimulus-level discriminability.