MindShot: Multi-Shot Video Reconstruction from fMRI with LLM Decoding

TL;DR

This paper introduces a novel framework for reconstructing multi-shot videos from fMRI data by explicitly decomposing signals, using large-scale data synthesis, and leveraging large language models for semantic captioning, significantly improving reconstruction fidelity.

Contribution

The paper presents a divide-and-decode framework with shot boundary prediction and LLM-based captioning, addressing multi-shot challenges in fMRI video reconstruction for the first time.

Findings

Outperforms state-of-the-art in multi-shot reconstruction fidelity.

fMRI decomposition improves caption CLIP similarity by 71.8%.

Semantic captioning enables accurate visual narrative recovery.

Abstract

Reconstructing dynamic videos from fMRI is important for understanding visual cognition and enabling vivid brain-computer interfaces. However, current methods are critically limited to single-shot clips, failing to address the multi-shot nature of real-world experiences. Multi-shot reconstruction faces fundamental challenges: fMRI signal mixing across shots, the temporal resolution mismatch between fMRI and video obscuring rapid scene changes, and the lack of dedicated multi-shot fMRI-video datasets. To overcome these limitations, we propose a novel divide-and-decode framework for multi-shot fMRI video reconstruction. Our core innovations are: (1) A shot boundary predictor module explicitly decomposing mixed fMRI signals into shot-specific segments. (2) Generative keyframe captioning using LLMs, which decodes robust textual descriptions from each segment, overcoming temporal blur by leveraging high-level semantics. (3) Novel large-scale data synthesis (20k samples) from existing datasets. Experimental results demonstrate our framework outperforms state-of-the-art methods in multi-shot reconstruction fidelity. Ablation studies confirm the critical role of fMRI decomposition and semantic captioning, with decomposition significantly improving decoded caption CLIP similarity by 71.8%. This work establishes a new paradigm for multi-shot fMRI reconstruction, enabling accurate recovery of complex visual narratives through explicit decomposition and semantic prompting.