From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretraining

TL;DR

This paper presents a scalable intracranial speech decoding framework using extensive week-long recordings and supervised pretraining, significantly improving performance over traditional methods and addressing cross-day variability.

Contribution

Introduces a pretraining framework leveraging large-scale intracranial recordings, enhancing speech decoding accuracy and addressing day-to-day neural variability.

Findings

Pretraining with large datasets outperforms classic models.

Model performance scales log-linearly with dataset size.

Brain activity representations drift across days, requiring models to handle variability.

Abstract

Decoding speech from brain activity has typically relied on limited neural recordings collected during short and highly controlled experiments. Here, we introduce a framework to leverage week-long intracranial and audio recordings from patients undergoing clinical monitoring, effectively increasing the training dataset size by over two orders of magnitude. With this pretraining, our contrastive learning model substantially outperforms models trained solely on classic experimental data, with gains that scale log-linearly with dataset size. Analysis of the learned representations reveals that, while brain activity represents speech features, its global structure largely drifts across days, highlighting the need for models that explicitly account for cross-day variability. Overall, our approach opens a scalable path toward decoding and modeling brain representations in both real-life and controlled task settings.