From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

TL;DR

This paper introduces a self-supervised method for reconstructing natural images from fMRI data, leveraging both labeled and unlabeled data to improve adaptability and performance in brain-image decoding.

Contribution

It proposes a novel Encoder-Decoder architecture that utilizes unlabeled data for training, enabling better adaptation to new test fMRI data.

Findings

Effective reconstruction of images from fMRI data.

Improved adaptability to new test data using self-supervision.

Utilization of unlabeled data enhances model performance.

Abstract

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.