Optimizing fMRI Data Acquisition for Decoding Natural Speech with Limited Participants

TL;DR

This study explores strategies for decoding natural speech from fMRI data with limited participants, highlighting the importance of extensive individual data and the challenges of multi-subject approaches.

Contribution

It demonstrates that deep neural networks can decode speech features from fMRI with limited data and shows that multi-subject training offers limited benefits in this context.

Findings

Multi-subject training does not improve decoding accuracy over single-subject training.

Decoders better model syntactic features than semantic features.

Complex syntax and semantic content make decoding more difficult.

Abstract

We investigate optimal strategies for decoding perceived natural speech from fMRI data acquired from a limited number of participants. Leveraging Lebel et al. (2023)'s dataset of 8 participants, we first demonstrate the effectiveness of training deep neural networks to predict LLM-derived text representations from fMRI activity. Then, in this data regime, we observe that multi-subject training does not improve decoding accuracy compared to single-subject approach. Furthermore, training on similar or different stimuli across subjects has a negligible effect on decoding accuracy. Finally, we find that our decoders better model syntactic than semantic features, and that stories containing sentences with complex syntax or rich semantic content are more challenging to decode. While our results demonstrate the benefits of having extensive data per participant (deep phenotyping), they suggest that leveraging multi-subject for natural speech decoding likely requires deeper phenotyping or a substantially larger cohort.