Perception Activator: An intuitive and portable framework for brain cognitive exploration

TL;DR

This paper introduces Perception Activator, a portable framework that enhances brain-vision decoding by integrating fMRI signals into multi-scale image features, improving semantic object reconstruction and understanding of visual perception.

Contribution

The paper presents a novel framework that uses fMRI data to improve semantic alignment in brain-vision decoding, addressing limitations of pixel-level approaches.

Findings

Incorporating fMRI signals improves object detection accuracy.

fMRI contains rich multi-object semantic cues.

The framework enhances semantic object reconstruction fidelity.

Abstract

Recent advances in brain-vision decoding have driven significant progress, reconstructing with high fidelity perceived visual stimuli from neural activity, e.g., functional magnetic resonance imaging (fMRI), in the human visual cortex. Most existing methods decode the brain signal using a two-level strategy, i.e., pixel-level and semantic-level. However, these methods rely heavily on low-level pixel alignment yet lack sufficient and fine-grained semantic alignment, resulting in obvious reconstruction distortions of multiple semantic objects. To better understand the brain's visual perception patterns and how current decoding models process semantic objects, we have developed an experimental framework that uses fMRI representations as intervention conditions. By injecting these representations into multi-scale image features via cross-attention, we compare both downstream performance and intermediate feature changes on object detection and instance segmentation tasks with and without fMRI information. Our results demonstrate that incorporating fMRI signals enhances the accuracy of downstream detection and segmentation, confirming that fMRI contains rich multi-object semantic cues and coarse spatial localization information-elements that current models have yet to fully exploit or integrate.