BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Models

TL;DR

BrainChat is a novel framework that decodes semantic information from fMRI data using vision-language pretrained models, enabling tasks like captioning and question answering with high accuracy and flexibility.

Contribution

This paper introduces BrainChat, a new generative approach leveraging vision-language models and contrastive learning to decode semantic brain activity, including first-time fMRI question answering.

Findings

Outperforms existing methods in fMRI captioning

Achieves first implementation of fMRI question answering

Effective even without image data

Abstract

Semantic information is vital for human interaction, and decoding it from brain activity enables non-invasive clinical augmentative and alternative communication. While there has been significant progress in reconstructing visual images, few studies have focused on the language aspect. To address this gap, leveraging the powerful capabilities of the decoder-based vision-language pretrained model CoCa, this paper proposes BrainChat, a simple yet effective generative framework aimed at rapidly accomplishing semantic information decoding tasks from brain activity, including fMRI question answering and fMRI captioning. BrainChat employs the self-supervised approach of Masked Brain Modeling to encode sparse fMRI data, obtaining a more compact embedding representation in the latent space. Subsequently, BrainChat bridges the gap between modalities by applying contrastive loss, resulting in aligned representations of fMRI, image, and text embeddings. Furthermore, the fMRI embeddings are mapped to the generative Brain Decoder via cross-attention layers, where they guide the generation of textual content about fMRI in a regressive manner by minimizing caption loss. Empirically, BrainChat exceeds the performance of existing state-of-the-art methods in the fMRI captioning task and, for the first time, implements fMRI question answering. Additionally, BrainChat is highly flexible and can achieve high performance without image data, making it better suited for real-world scenarios with limited data.