Decoding individual words from non-invasive brain recordings across 723 participants

TL;DR

This study introduces a deep learning approach to decode individual words from non-invasive EEG and MEG brain signals across 723 participants, demonstrating improved accuracy and revealing factors influencing decoding success.

Contribution

The paper presents a novel deep learning pipeline capable of decoding words from non-invasive brain recordings in a large-scale, multilingual, multi-task setting, outperforming existing methods.

Findings

Decoding performance is higher with MEG and reading tasks.

More data per participant improves decoding accuracy.

The model captures semantic, syntactic, and surface word properties.

Abstract

Deep learning has recently enabled the decoding of language from the neural activity of a few participants with electrodes implanted inside their brain. However, reliably decoding words from non-invasive recordings remains an open challenge. To tackle this issue, we introduce a novel deep learning pipeline to decode individual words from non-invasive electro- (EEG) and magneto-encephalography (MEG) signals. We train and evaluate our approach on an unprecedentedly large number of participants (723) exposed to five million words either written or spoken in English, French or Dutch. Our model outperforms existing methods consistently across participants, devices, languages, and tasks, and can decode words absent from the training set. Our analyses highlight the importance of the recording device and experimental protocol: MEG and reading are easier to decode than EEG and listening, respectively, and it is preferable to collect a large amount of data per participant than to repeat stimuli across a large number of participants. Furthermore, decoding performance consistently increases with the amount of (i) data used for training and (ii) data used for averaging during testing. Finally, single-word predictions show that our model effectively relies on word semantics but also captures syntactic and surface properties such as part-of-speech, word length and even individual letters, especially in the reading condition. Overall, our findings delineate the path and remaining challenges towards building non-invasive brain decoders for natural language.