Brain Dialogue Interface (BDI): A User-Friendly fMRI Model for Interactive Brain Decoding

TL;DR

This paper presents a user-friendly, unsupervised brain decoding model that enables interactive communication with the brain, effectively compresses brain signals, and integrates with multimodal inputs for versatile neurocognitive analysis.

Contribution

Introduces a novel interactive brain decoding model trained unsupervisedly, capable of dynamic dialogue-based decoding and efficient signal compression, surpassing traditional static models.

Findings

Successfully compresses brain signals from 185,751 voxels into 32 signals

Demonstrates feasibility of interactive, dialogue-based brain decoding

Integrates seamlessly with multimodal models for enhanced control

Abstract

Brain decoding techniques are essential for understanding the neurocognitive system. Although numerous methods have been introduced in this field, accurately aligning complex external stimuli with brain activities remains a formidable challenge. To alleviate alignment difficulties, many studies have simplified their models by employing single-task paradigms and establishing direct links between brain/world through classification strategies. Despite improvements in decoding accuracy, this strategy frequently encounters issues with generality when adapting these models to various task paradigms. To address this issue, this study introduces a user-friendly decoding model that enables dynamic communication with the brain, as opposed to the static decoding approaches utilized by traditional studies. The model functions as a brain simulator, allowing for interactive engagement with the brain and enabling the decoding of a subject's experiences through dialogue-like queries. Uniquely, our model is trained in a completely unsupervised and task-free manner. Our experiments demonstrate the feasibility and versatility of our proposed method. Notably, our model demonstrates exceptional capabilities in signal compression, successfully representing the entire brain signal of approximately 185,751 voxels with just 32 signals. Furthermore, we show how our model can integrate seamlessly with multimodal models, thus enhancing the potential for controlling brain decoding through textual or image inputs.