BOLDreams: Dreaming with pruned in-silico fMRI Encoding Models of the Visual Cortex

TL;DR

This paper develops and evaluates neural encoding models of the visual cortex using the NSD dataset, employing explainable AI techniques to interpret feature contributions and identify optimal model configurations.

Contribution

It introduces a comprehensive assessment of feature-weighted receptive field models with various modalities and hyperparameters, utilizing XAI methods for interpretability.

Findings

Models with different modalities show varied performance.

Explainable AI reveals biologically plausible features.

A minimal model balances simplicity and performance.

Abstract

In this article we use the Natural Scenes Dataset (NSD) to train a family of feature-weighted receptive field neural encoding models. These models use a pre-trained vision or text backbone and map extracted features to the voxel space via receptive field readouts. We comprehensively assess such models, quantifying performance changes based on using different modalities like text or images, toggling finetuning, using different pre-trained backbones, and changing the width of the readout. We also dissect each model using explainable AI (XAI) techniques, such as feature visualization via input optimization, also referred to as ``dreaming'' in the AI literature, and the integrated gradients approach to calculate implicit attention maps to illustrate which features drive the predicted signal in different brain areas. These XAI tools illustrate biologically plausible features that drive the predicted signal. Traversing the model hyperparameter space reveals the existence of a maximally minimal model, balancing simplicity while maintaining performance.