May I see what you see? Predicting visual features from neuronal activity

TL;DR

This paper shows how brain activity from fMRI scans can be used to reconstruct and predict visual features and objects from video stimuli.

Contribution

A novel autoencoder-based method is introduced to map fMRI responses to visual features and reconstruct video frames.

Findings

The model successfully reconstructs video frames from fMRI data.

fMRI responses can predict objects like faces in the original visual stimuli.

Latent representations from fMRI data are highly clustered with actual video frame representations.

Abstract

Understanding brain response to audiovisual stimuli is a key challenge in understanding neuronal processes. In this paper, we describe our effort aimed at reconstructing video frames from observed functional MRI images. We also demonstrate that our model can predict visual objects. Our method constructs an autoencoder model for a set of training video segments to code video streams into their corresponding latent representations. Next, we learn a mapping from the observed fMRI response to the corresponding latent video frame representation. Finally, we pass the latent vectors computed using the fMRI response through the decoder to reconstruct the predicted image. We show that the representations of video frames and those constructed from corresponding fMRI images are highly clustered, the latent representations can be used to predict objects in video frames using just the fMRI frames, and fMRI responses can be used to reconstruct the inputs to predict the presence of faces.

•NN architecture showing similarity in representations of visual stimuli and fMRI•Predict objects in video frames using just the fMRI frames using the representations•fMRI responses can be used to reconstruct faces that were present in visual stimulus

NN architecture showing similarity in representations of visual stimuli and fMRI

Predict objects in video frames using just the fMRI frames using the representations

fMRI responses can be used to reconstruct faces that were present in visual stimulus

Medical imaging; Systems neuroscience; Signal processing; Signal reconstruction; Machine learning

Figures