MinD-3D: Reconstruct High-quality 3D objects in Human Brain

TL;DR

This paper introduces MinD-3D, a three-stage framework that reconstructs high-quality 3D objects from fMRI signals, advancing cognitive neuroscience and computer vision by decoding the brain's visual information.

Contribution

We propose MinD-3D, a novel framework for reconstructing 3D objects from fMRI data, supported by the new fMRI-Shape dataset and a comprehensive three-stage decoding process.

Findings

MinD-3D achieves high semantic relevance in reconstructed 3D objects.

The framework demonstrates strong spatial similarity with original objects.

Our analysis reveals significant correlation between model features and brain ROIs.

Abstract

In this paper, we introduce Recon3DMind, an innovative task aimed at reconstructing 3D visuals from Functional Magnetic Resonance Imaging (fMRI) signals, marking a significant advancement in the fields of cognitive neuroscience and computer vision. To support this pioneering task, we present the fMRI-Shape dataset, which includes data from 14 participants and features 360-degree videos of 3D objects to enable comprehensive fMRI signal capture across various settings, thereby laying a foundation for future research. Furthermore, we propose MinD-3D, a novel and effective three-stage framework specifically designed to decode the brain's 3D visual information from fMRI signals, demonstrating the feasibility of this challenging task. The framework begins by extracting and aggregating features from fMRI frames through a neuro-fusion encoder, subsequently employs a feature bridge diffusion model to generate visual features, and ultimately recovers the 3D object via a generative transformer decoder. We assess the performance of MinD-3D using a suite of semantic and structural metrics and analyze the correlation between the features extracted by our model and the visual regions of interest (ROIs) in fMRI signals. Our findings indicate that MinD-3D not only reconstructs 3D objects with high semantic relevance and spatial similarity but also significantly enhances our understanding of the human brain's capabilities in processing 3D visual information. Project page at: https://jianxgao.github.io/MinD-3D.