Multiscale Voxel Based Decoding For Enhanced Natural Image Reconstruction From Brain Activity

TL;DR

This paper introduces a novel method combining object decoding, class-conditional GANs, and neural style transfer to improve the quality and semantic accuracy of natural image reconstructions from fMRI brain activity.

Contribution

It presents a new framework that merges existing decoding and reconstruction techniques with generative models to enhance image fidelity from brain signals.

Findings

Improved semantic similarity in reconstructed images.

Framework effectively combines decoding and generative models.

Enhances natural image reconstruction from fMRI data.

Abstract

Reconstructing perceived images from human brain activity monitored by functional magnetic resonance imaging (fMRI) is hard, especially for natural images. Existing methods often result in blurry and unintelligible reconstructions with low fidelity. In this study, we present a novel approach for enhanced image reconstruction, in which existing methods for object decoding and image reconstruction are merged together. This is achieved by conditioning the reconstructed image to its decoded image category using a class-conditional generative adversarial network and neural style transfer. The results indicate that our approach improves the semantic similarity of the reconstructed images and can be used as a general framework for enhanced image reconstruction.