Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Dataset

TL;DR

This paper introduces a non-invasive brain-computer interface benchmark for keyword spotting using the LibriBrain dataset, providing standardized evaluation protocols, a baseline model, and insights into factors affecting detection performance.

Contribution

It proposes a new benchmark task for brain-to-text communication, with standardized data splits, evaluation metrics, and a baseline model, advancing research in practical neural keyword detection.

Findings

Baseline model achieves 13x higher AUPRC than permutation baseline.

Performance improves log-linearly with more training hours.

Word frequency and duration systematically affect detectability.

Abstract

Non-invasive brain-computer interfaces (BCIs) are beginning to benefit from large, public benchmarks. However, current benchmarks target relatively simple, foundational tasks like Speech Detection and Phoneme Classification, while application-ready results on tasks like Brain-to-Text remain elusive. We propose Keyword Spotting (KWS) as a practically applicable, privacy-aware intermediate task. Using the deep 52-hour, within-subject LibriBrain corpus, we provide standardized train/validation/test splits for reproducible benchmarking, and adopt an evaluation protocol tailored to extreme class imbalance. Concretely, we use area under the precision-recall curve (AUPRC) as a robust evaluation metric, complemented by false alarms per hour (FA/h) at fixed recall to capture user-facing trade-offs. To simplify deployment and further experimentation within the research community, we are releasing an updated version of the pnpl library with word-level dataloaders and Colab-ready tutorials. As an initial reference model, we present a compact 1-D Conv/ResNet baseline with focal loss and top-k pooling that is trainable on a single consumer-class GPU. The reference model achieves approximately 13x the permutation baseline AUPRC on held-out sessions, demonstrating the viability of the task. Exploratory analyses reveal: (i) predictable within-subject scaling - performance improves log-linearly with more training hours - and (ii) the existence of word-level factors (frequency and duration) that systematically modulate detectability.