Double-Flow GAN model for the reconstruction of perceived faces from brain activities

TL;DR

This paper introduces a Double-Flow GAN framework for reconstructing perceived faces from brain activity, effectively capturing high-level features and multiple face attributes, and outperforming previous models in accuracy and state-of-the-art performance.

Contribution

The study presents a novel Double-Flow GAN model with a pretraining process and cross-subject alignment for improved face reconstruction from fMRI data.

Findings

Outperforms traditional models in face attribute reconstruction

Achieves state-of-the-art reconstruction accuracy

Effectively handles cross-subject brain variability

Abstract

Face plays an important role in humans visual perception, and reconstructing perceived faces from brain activities is challenging because of its difficulty in extracting high-level features and maintaining consistency of multiple face attributes, such as expression, identity, gender, etc. In this study, we proposed a novel reconstruction framework, which we called Double-Flow GAN, that can enhance the capability of discriminator and handle imbalances in images from certain domains that are too easy for generators. We also designed a pretraining process that uses features extracted from images as conditions for making it possible to pretrain the conditional reconstruction model from fMRI in a larger pure image dataset. Moreover, we developed a simple pretrained model for fMRI alignment to alleviate the problem of cross-subject reconstruction due to the variations of brain structure among different subjects. We conducted experiments by using our proposed method and traditional reconstruction models. Results showed that the proposed method is significant at accurately reconstructing multiple face attributes, outperforms the previous reconstruction models, and exhibited state-of-the-art reconstruction abilities.