SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance

TL;DR

SemVideo introduces a hierarchical semantic guidance framework for reconstructing videos from brain activity, significantly improving visual accuracy and temporal coherence over previous methods.

Contribution

The paper presents SemVideo, a novel fMRI-to-video reconstruction method utilizing hierarchical semantic cues and a tripartite attention architecture, advancing the state-of-the-art in neural decoding.

Findings

Achieves superior semantic alignment with brain signals.

Enhances temporal coherence in reconstructed videos.

Sets new benchmarks on CC2017 and HCP datasets.

Abstract

Reconstructing dynamic visual experiences from brain activity provides a compelling avenue for exploring the neural mechanisms of human visual perception. While recent progress in fMRI-based image reconstruction has been notable, extending this success to video reconstruction remains a significant challenge. Current fMRI-to-video reconstruction approaches consistently encounter two major shortcomings: (i) inconsistent visual representations of salient objects across frames, leading to appearance mismatches; (ii) poor temporal coherence, resulting in motion misalignment or abrupt frame transitions. To address these limitations, we introduce SemVideo, a novel fMRI-to-video reconstruction framework guided by hierarchical semantic information. At the core of SemVideo is SemMiner, a hierarchical guidance module that constructs three levels of semantic cues from the original video stimulus: static anchor descriptions, motion-oriented narratives, and holistic summaries. Leveraging this semantic guidance, SemVideo comprises three key components: a Semantic Alignment Decoder that aligns fMRI signals with CLIP-style embeddings derived from SemMiner, a Motion Adaptation Decoder that reconstructs dynamic motion patterns using a novel tripartite attention fusion architecture, and a Conditional Video Render that leverages hierarchical semantic guidance for video reconstruction. Experiments conducted on the CC2017 and HCP datasets demonstrate that SemVideo achieves superior performance in both semantic alignment and temporal consistency, setting a new state-of-the-art in fMRI-to-video reconstruction.