Scaling laws for decoding images from brain activity

TL;DR

This study systematically compares how different non-invasive neuroimaging devices and data quantities affect the accuracy of decoding images from brain activity, revealing key scaling laws and the importance of data per subject.

Contribution

It provides the first large-scale benchmark comparing multiple neuroimaging modalities for image decoding and uncovers how decoding performance scales with data amount and device type.

Findings

Higher precision devices yield better decoding performance.

Deep learning models outperform linear models on noisier data.

Decoding performance scales log-linearly with data amount per subject.

Abstract

Generative AI has recently propelled the decoding of images from brain activity. How do these approaches scale with the amount and type of neural recordings? Here, we systematically compare image decoding from four types of non-invasive devices: electroencephalography (EEG), magnetoencephalography (MEG), high-field functional Magnetic Resonance Imaging (3T fMRI) and ultra-high field (7T) fMRI. For this, we evaluate decoding models on the largest benchmark to date, encompassing 8 public datasets, 84 volunteers, 498 hours of brain recording and 2.3 million brain responses to natural images. Unlike previous work, we focus on single-trial decoding performance to simulate real-time settings. This systematic comparison reveals three main findings. First, the most precise neuroimaging devices tend to yield the best decoding performances, when the size of the training sets are similar. However, the gain enabled by deep learning - in comparison to linear models - is obtained with the noisiest devices. Second, we do not observe any plateau of decoding performance as the amount of training data increases. Rather, decoding performance scales log-linearly with the amount of brain recording. Third, this scaling law primarily depends on the amount of data per subject. However, little decoding gain is observed by increasing the number of subjects. Overall, these findings delineate the path most suitable to scale the decoding of images from non-invasive brain recordings.