Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks

TL;DR

This paper demonstrates a method to reconstruct face images from fMRI brain activity patterns using a deep generative neural network, achieving high accuracy and enabling decoding of face perception and imagination.

Contribution

It introduces a novel approach combining a VAE and GAN trained on celebrity faces to decode and reconstruct faces from fMRI data, advancing brain decoding techniques.

Findings

Achieved over 95% pairwise decoding accuracy.

Significantly outperformed PCA baseline in reconstruction quality.

Enabled decoding of face gender and imagined faces.

Abstract

While objects from different categories can be reliably decoded from fMRI brain response patterns, it has proved more difficult to distinguish visually similar inputs, such as different instances of the same category. Here, we apply a recently developed deep learning system to the reconstruction of face images from human fMRI patterns. We trained a variational auto-encoder (VAE) neural network using a GAN (Generative Adversarial Network) unsupervised training procedure over a large dataset of celebrity faces. The auto-encoder latent space provides a meaningful, topologically organized 1024-dimensional description of each image. We then presented several thousand face images to human subjects, and learned a simple linear mapping between the multi-voxel fMRI activation patterns and the 1024 latent dimensions. Finally, we applied this mapping to novel test images, turning the obtained fMRI patterns into VAE latent codes, and ultimately the codes into face reconstructions. Qualitative and quantitative evaluation of the reconstructions revealed robust pairwise decoding (>95% correct), and a strong improvement relative to a baseline model (PCA decomposition). Furthermore, this brain decoding model can readily be recycled to probe human face perception along many dimensions of interest; for example, the technique allowed for accurate gender classification, and even to decode which face was imagined, rather than seen by the subject. We hypothesize that the latent space of modern deep learning generative models could serve as a valid approximation for human brain representations.