NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery

TL;DR

NSD-Imagery is a new benchmark dataset of fMRI activity paired with mental images, enabling evaluation of vision decoding models on mental imagery and highlighting the importance of model architecture for generalization.

Contribution

This paper introduces NSD-Imagery, a benchmark dataset for mental imagery, and evaluates existing models, revealing that simple architectures generalize better to mental images than complex ones.

Findings

Decoding performance on mental images is largely independent of vision reconstruction performance.

Simple linear and multimodal models generalize better to mental imagery.

Complex architectures tend to overfit visual training data.

Abstract

We release NSD-Imagery, a benchmark dataset of human fMRI activity paired with mental images, to complement the existing Natural Scenes Dataset (NSD), a large-scale dataset of fMRI activity paired with seen images that enabled unprecedented improvements in fMRI-to-image reconstruction efforts. Recent models trained on NSD have been evaluated only on seen image reconstruction. Using NSD-Imagery, it is possible to assess how well these models perform on mental image reconstruction. This is a challenging generalization requirement because mental images are encoded in human brain activity with relatively lower signal-to-noise and spatial resolution; however, generalization from seen to mental imagery is critical for real-world applications in medical domains and brain-computer interfaces, where the desired information is always internally generated. We provide benchmarks for a suite of recent NSD-trained open-source visual decoding models (MindEye1, MindEye2, Brain Diffuser, iCNN, Takagi et al.) on NSD-Imagery, and show that the performance of decoding methods on mental images is largely decoupled from performance on vision reconstruction. We further demonstrate that architectural choices significantly impact cross-decoding performance: models employing simple linear decoding architectures and multimodal feature decoding generalize better to mental imagery, while complex architectures tend to overfit visual training data. Our findings indicate that mental imagery datasets are critical for the development of practical applications, and establish NSD-Imagery as a useful resource for better aligning visual decoding methods with this goal.