Brain-Grasp: Graph-based Saliency Priors for Improved fMRI-based Visual Brain Decoding

TL;DR

This paper introduces a graph-informed saliency prior framework that enhances fMRI-based visual brain decoding by preserving object structure and semantics using a diffusion model conditioned on structural cues.

Contribution

It presents a novel, lightweight decoding approach that integrates saliency priors and semantic embeddings to improve object and scene reconstruction from brain signals.

Findings

Improved structural similarity in reconstructed images.

Enhanced semantic fidelity compared to previous methods.

Single-model approach reduces complexity and computational cost.

Abstract

Recent progress in brain-guided image generation has improved the quality of fMRI-based reconstructions; however, fundamental challenges remain in preserving object-level structure and semantic fidelity. Many existing approaches overlook the spatial arrangement of salient objects, leading to conceptually inconsistent outputs. We propose a saliency-driven decoding framework that employs graph-informed saliency priors to translate structural cues from brain signals into spatial masks. These masks, together with semantic information extracted from embeddings, condition a diffusion model to guide image regeneration, helping preserve object conformity while maintaining natural scene composition. In contrast to pipelines that invoke multiple diffusion stages, our approach relies on a single frozen model, offering a more lightweight yet effective design. Experiments show that this strategy improves both conceptual alignment and structural similarity to the original stimuli, while also introducing a new direction for efficient, interpretable, and structurally grounded brain decoding.