Unlocking Non-Invasive Brain-to-Text

TL;DR

This paper presents a breakthrough in non-invasive brain-to-text transcription, significantly surpassing previous methods by leveraging large language models, predictive in-filling, and scalable deep learning, thus advancing non-invasive brain-computer interfaces.

Contribution

It introduces a novel non-invasive B2T system with LLM-based rescoring, OOV word handling, and scalable training, achieving substantial accuracy improvements over prior work.

Findings

BLEU score increased by 1.4-2.6x over previous methods

Vocabulary expansion via predictive in-filling improves accuracy

Scaling models across datasets enhances performance by 2.1-2.3x

Abstract

Despite major advances in surgical brain-to-text (B2T), i.e. transcribing speech from invasive brain recordings, non-invasive alternatives have yet to surpass even chance on standard metrics. This remains a barrier to building a non-invasive brain-computer interface (BCI) capable of restoring communication in paralysed individuals without surgery. Here, we present the first non-invasive B2T result that significantly exceeds these critical baselines, raising BLEU by $1.4\mathrm{-}2.6\times$ over prior work. This result is driven by three contributions: (1) we extend recent word-classification models with LLM-based rescoring, transforming single-word predictors into closed-vocabulary B2T systems; (2) we introduce a predictive in-filling approach to handle out-of-vocabulary (OOV) words, substantially expanding the effective vocabulary; and (3) we demonstrate, for the first time, how to scale non-invasive B2T models across datasets, unlocking deep learning at scale and improving accuracy by $2.1\mathrm{-}2.3\times$. Through these contributions, we offer new insights into the roles of data quality and vocabulary size. Together, our results remove a major obstacle to realising practical non-invasive B2T systems.