Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction

TL;DR

This paper introduces CineNeuron, a hierarchical framework that enhances fMRI-to-video reconstruction by semantically enriching signals and integrating memory, leading to improved results over existing methods.

Contribution

CineNeuron is a novel dual-stage framework that captures comprehensive semantics and dynamically fuses memories to improve fMRI-to-video reconstruction.

Findings

CineNeuron outperforms state-of-the-art methods on two benchmarks.

The semantic enrichment stage captures rich textual and visual semantics.

Memory integration refines video reconstruction by leveraging prior data.

Abstract

Reconstructing dynamic visual experiences as videos from functional magnetic resonance imaging (fMRI) is pivotal for advancing the understanding of neural processes. However, current fMRI-to-video reconstruction methods are hindered by a semantic gap between noisy fMRI signals and the rich content of videos, stemming from a reliance on incomplete semantic embeddings that neither capture video-specific cues (e.g., actions) nor integrate prior knowledge. To this end, we draw inspiration from the dual-pathway processing mechanism in human brain and introduce CineNeuron, a novel hierarchical framework for semantically enhanced video reconstruction from fMRI signals with two synergistic stages. First, a bottom-up semantic enrichment stage maps fMRI signals to a rich embedding space that comprehensively captures textual semantics, image contents, action concepts, and object categories. Second, a top-down memory integration stage utilizes the proposed Mixture-of-Memories method to dynamically select relevant "memories" from previously seen data and fuse them with the fMRI embedding to refine the video reconstruction. Extensive experimental results on two fMRI-to-video benchmarks demonstrate that CineNeuron surpasses state-of-the-art methods across various metrics.