Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG

TL;DR

This study demonstrates that fine-tuning language encoding models on slow fMRI data enhances the prediction of fast ECoG signals, bridging the gap between different neural recording modalities.

Contribution

The paper introduces a method to improve ECoG prediction by leveraging non-invasive fMRI data through fine-tuning language models, despite the difference in temporal resolution.

Findings

fMRI fine-tuning improves ECoG prediction performance

Models trained on downsampled fMRI data still predict ECoG effectively

ECoG prediction scales with the amount of fMRI data used for tuning

Abstract

Neuroscientists have recently turned to intracranial brain recording methods, like electrocorticography (ECoG), for human experiments because of the fine spatial and temporal resolution that they afford. Models trained on this data, however, are fundamentally restricted by the patient populations that can receive the implants necessary for recording. We propose using non-invasive fMRI to bridge the gap in training data. Using spoken language representations fine-tuned on fMRI, we build encoding models of ECoG. These representations showed improved prediction performance in ECoG, even though the temporal resolution of fMRI is two orders of magnitude worse. Prediction improved in frequency bands well beyond what is directly measured in fMRI. Next, to test the procedure's generalization ability, we fine-tuned models on fMRI responses that were temporally downsampled by a factor of 2. Despite the loss in resolution, these models were able to predict fMRI and ECoG responses at levels comparable to the original fMRI-tuned models. Finally, we showed that ECoG performance steadily scales with the amount of fMRI-tuning data. Our results show that "slow" data like fMRI can be a valuable resource for building better models of "fast" brain data like ECoG. In the future, integrating across multiple recording methods may further improve performance in other applications, like decoding.