NeuralBench: A Unifying Framework to Benchmark NeuroAI Models

TL;DR

NeuralBench is an open-source, unified benchmarking framework for AI models of brain activity, currently focusing on EEG with 36 tasks and 14 architectures, revealing current models' limitations and challenges.

Contribution

This work introduces NeuralBench, a comprehensive, extensible platform for standardized evaluation of neuroimaging AI models across multiple modalities and datasets.

Findings

Foundation models only marginally outperform task-specific models.

Many neuroimaging tasks remain highly challenging for current models.

Abstract

Deep learning and large public datasets have recently catalyzed the proliferation of AI models for processing brain recordings. However, systematically evaluating these models remains a challenge: not only do the preprocessing pipelines, training and finetuning approaches largely vary across studies, but their downstream evaluation is often limited to small sets of tasks and/or datasets. Here, we present NeuralBench: a unified framework for benchmarking AI models of brain activity. We accompany this framework with NeuralBench-EEG v1.0 -- a large EEG benchmark that includes 36 electroencephalography (EEG) tasks and 14 deep learning architectures, and is evaluated on 94 datasets accessed through a standardized interface. This first EEG-focused release already highlights two main findings. First, current foundation models only marginally outperform task-specific models. Second, a large set of tasks (e.g. cognitive decoding, clinical predictions) remain highly challenging, even for the best models. Critically, NeuralBench is designed for the integration of new tasks, datasets, models, and neuroimaging modalities, as illustrated by preliminary extensions to MEG and fMRI datasets and models. Through this white paper, we invite the community to expand this open-source framework and work together toward a unified benchmarking standard for neuroimaging models.