Contrastive Learning to Fine-Tune Feature Extraction Models for the Visual Cortex

TL;DR

This paper introduces a contrastive learning approach to fine-tune CNNs for better neural response prediction in the visual cortex, leveraging large-scale fMRI data to improve encoding accuracy and explore neural processing lateralization.

Contribution

It adapts contrastive learning to enhance feature extraction models for neural response prediction, outperforming baseline methods and enabling cross-subject transfer and analysis of neural lateralization.

Findings

Contrastive learning improves encoding accuracy in early visual ROIs.

Fine-tuned models transfer effectively across subjects and datasets.

Analysis reveals lateralization in early visual processing.

Abstract

Predicting the neural response to natural images in the visual cortex requires extracting relevant features from the images and relating those feature to the observed responses. In this work, we optimize the feature extraction in order to maximize the information shared between the image features and the neural response across voxels in a given region of interest (ROI) extracted from the BOLD signal measured by fMRI. We adapt contrastive learning (CL) to fine-tune a convolutional neural network, which was pretrained for image classification, such that a mapping of a given image's features are more similar to the corresponding fMRI response than to the responses to other images. We exploit the recently released Natural Scenes Dataset (Allen et al., 2022) as organized for the Algonauts Project (Gifford et al., 2023), which contains the high-resolution fMRI responses of eight subjects to tens of thousands of naturalistic images. We show that CL fine-tuning creates feature extraction models that enable higher encoding accuracy in early visual ROIs as compared to both the pretrained network and a baseline approach that uses a regression loss at the output of the network to tune it for fMRI response encoding. We investigate inter-subject transfer of the CL fine-tuned models, including subjects from another, lower-resolution dataset (Gong et al., 2023). We also pool subjects for fine-tuning to further improve the encoding performance. Finally, we examine the performance of the fine-tuned models on common image classification tasks, explore the landscape of ROI-specific models by applying dimensionality reduction on the Bhattacharya dissimilarity matrix created using the predictions on those tasks (Mao et al., 2024), and investigate lateralization of the processing for early visual ROIs using salience maps of the classifiers built on the CL-tuned models.