SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstruction

TL;DR

SynMind introduces a semantic-aware framework for fMRI-based image reconstruction, significantly reducing hallucinations by integrating explicit semantic descriptions with visual priors, leading to more accurate and human-aligned images.

Contribution

The paper proposes SynMind, a novel approach that incorporates sentence-level semantic parsing and grounded VLMs to improve the semantic fidelity of fMRI-based image reconstructions.

Findings

Outperforms state-of-the-art methods on multiple metrics

Produces reconstructions more aligned with human perception

Engages broader, more relevant brain regions in neurovisualization

Abstract

Recent advances in fMRI-based image reconstruction have achieved remarkable photo-realistic fidelity. Yet, a persistent limitation remains: while reconstructed images often appear naturalistic and holistically similar to the target stimuli, they frequently suffer from severe semantic misalignment -- salient objects are often replaced or hallucinated despite high visual quality. In this work, we address this limitation by rethinking the role of explicit semantic interpretation in fMRI decoding. We argue that existing methods rely too heavily on entangled visual embeddings which prioritize low-level appearance cues -- such as texture and global gist -- over explicit semantic identity. To overcome this, we parse fMRI signals into rich, sentence-level semantic descriptions that mirror the hierarchical and compositional nature of human visual understanding. We achieve this by leveraging grounded VLMs to generate synthetic, human-like, multi-granularity textual representations that capture object identities and spatial organization. Built upon this foundation, we propose SynMind, a framework that integrates these explicit semantic encodings with visual priors to condition a pretrained diffusion model. Extensive experiments demonstrate that SynMind outperforms state-of-the-art methods across most quantitative metrics. Notably, by offloading semantic reasoning to our text-alignment module, SynMind surpasses competing methods based on SDXL while using the much smaller Stable Diffusion 1.4 and a single consumer GPU. Large-scale human evaluations further confirm that SynMind produces reconstructions more consistent with human visual perception. Neurovisualization analyses reveal that SynMind engages broader and more semantically relevant brain regions, mitigating the over-reliance on high-level visual areas.