Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging

TL;DR

This paper introduces FgB2I, a novel method that uses fine-grained text descriptions as a bridge to enhance the accuracy and detail of brain-to-image reconstruction from fMRI data.

Contribution

It proposes a three-stage approach incorporating fine-grained text generation and semantic guidance to improve visual stimulus reconstruction from brain activity.

Findings

Enhanced image reconstruction with more details and semantic consistency.

Validated the importance of fine-grained captions generated from vision-language models.

Guided decoding using reward metrics improves the semantic alignment of reconstructed images.

Abstract

Brain-to-Image reconstruction aims to recover visual stimuli perceived by humans from brain activity. However, the reconstructed visual stimuli often missing details and semantic inconsistencies, which may be attributed to insufficient semantic information. To address this issue, we propose an approach named Fine-grained Brain-to-Image reconstruction (FgB2I), which employs fine-grained text as bridge to improve image reconstruction. FgB2I comprises three key stages: detail enhancement, decoding fine-grained text descriptions, and text-bridged brain-to-image reconstruction. In the detail-enhancement stage, we leverage large vision-language models to generate fine-grained captions for visual stimuli and experimentally validate its importance. We propose three reward metrics (object accuracy, text-image semantic similarity, and image-image semantic similarity) to guide the language model in decoding fine-grained text descriptions from fMRI signals. The fine-grained text descriptions can be integrated into existing reconstruction methods to achieve fine-grained Brain-to-Image reconstruction.