A visual encoding model based on deep neural networks and transfer learning

TL;DR

This paper introduces a novel visual encoding model that leverages transfer learning with deep neural networks and nonlinear mappings to significantly improve the prediction of brain responses in fMRI data, surpassing traditional linear models.

Contribution

The study presents a new framework combining pre-trained DNN features and nonlinear mappings, enhancing prediction accuracy of neural responses over existing linear models.

Findings

Predicts responses of over 20% of voxels in early visual areas.

Achieves higher prediction accuracy than conventional models across all early visual areas.

Significantly improves prediction in anterior visual regions like V4 and LO.

Abstract

Background: Building visual encoding models to accurately predict visual responses is a central challenge for current vision-based brain-machine interface techniques. To achieve high prediction accuracy on neural signals, visual encoding models should include precise visual features and appropriate prediction algorithms. Most existing visual encoding models employ hand-craft visual features (e.g., Gabor wavelets or semantic labels) or data-driven features (e.g., features extracted from deep neural networks (DNN)). They also assume a linear mapping between feature representation to brain activity. However, it remains unknown whether such linear mapping is sufficient for maximizing prediction accuracy. New Method: We construct a new visual encoding framework to predict cortical responses in a benchmark functional magnetic resonance imaging (fMRI) dataset. In this framework, we employ the transfer learning technique to incorporate a pre-trained DNN (i.e., AlexNet) and train a nonlinear mapping from visual features to brain activity. This nonlinear mapping replaces the conventional linear mapping and is supposed to improve prediction accuracy on brain activity. Results: The proposed framework can significantly predict responses of over 20% voxels in early visual areas (i.e., V1-lateral occipital region, LO) and achieve unprecedented prediction accuracy. Comparison with Existing Methods: Comparing to two conventional visual encoding models, we find that the proposed encoding model shows consistent higher prediction accuracy in all early visual areas, especially in relatively anterior visual areas (i.e., V4 and LO). Conclusions: Our work proposes a new framework to utilize pre-trained visual features and train non-linear mappings from visual features to brain activity.