Deep Neural Encoder-Decoder Model to Relate fMRI Brain Activity with Naturalistic Stimuli

TL;DR

This paper introduces a deep neural encoder-decoder model that links fMRI brain activity with naturalistic visual stimuli, enabling prediction and reconstruction of visual inputs from neural data, and analyzing brain regions involved.

Contribution

The paper presents a novel end-to-end deep neural model that encodes and decodes fMRI responses to naturalistic stimuli using temporal convolutional layers.

Findings

Identifies key brain regions involved in visual processing during film viewing.

Successfully reconstructs visual features such as edges, faces, and contrasts from neural activity.

Demonstrates the model's ability to relate brain activity to complex naturalistic stimuli.

Abstract

We propose an end-to-end deep neural encoder-decoder model to encode and decode brain activity in response to naturalistic stimuli using functional magnetic resonance imaging (fMRI) data. Leveraging temporally correlated input from consecutive film frames, we employ temporal convolutional layers in our architecture, which effectively allows to bridge the temporal resolution gap between natural movie stimuli and fMRI acquisitions. Our model predicts activity of voxels in and around the visual cortex and performs reconstruction of corresponding visual inputs from neural activity. Finally, we investigate brain regions contributing to visual decoding through saliency maps. We find that the most contributing regions are the middle occipital area, the fusiform area, and the calcarine, respectively employed in shape perception, complex recognition (in particular face perception), and basic visual features such as edges and contrasts. These functions being strongly solicited are in line with the decoder's capability to reconstruct edges, faces, and contrasts. All in all, this suggests the possibility to probe our understanding of visual processing in films using as a proxy the behaviour of deep learning models such as the one proposed in this paper.