BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings

TL;DR

BrainFLORA introduces a multimodal neural embedding framework that integrates EEG, MEG, and fMRI data to better understand brain representations of visual concepts, advancing neuroscience and AI applications.

Contribution

It presents a novel unified framework using multimodal large language models with adapters for integrating neuroimaging data, enabling improved neural representation analysis.

Findings

Achieved state-of-the-art performance in joint-subject visual retrieval

Revealed alignment of visual concept representations across neural modalities

Bridged neural imaging data with real-world object perception

Abstract

Understanding how the brain represents visual information is a fundamental challenge in neuroscience and artificial intelligence. While AI-driven decoding of neural data has provided insights into the human visual system, integrating multimodal neuroimaging signals, such as EEG, MEG, and fMRI, remains a critical hurdle due to their inherent spatiotemporal misalignment. Current approaches often analyze these modalities in isolation, limiting a holistic view of neural representation. In this study, we introduce BrainFLORA, a unified framework for integrating cross-modal neuroimaging data to construct a shared neural representation. Our approach leverages multimodal large language models (MLLMs) augmented with modality-specific adapters and task decoders, achieving state-of-the-art performance in joint-subject visual retrieval task and has the potential to extend multitasking. Combining neuroimaging analysis methods, we further reveal how visual concept representations align across neural modalities and with real world object perception. We demonstrate that the brain's structured visual concept representations exhibit an implicit mapping to physical-world stimuli, bridging neuroscience and machine learning from different modalities of neural imaging. Beyond methodological advancements, BrainFLORA offers novel implications for cognitive neuroscience and brain-computer interfaces (BCIs). Our code is available at https://github.com/ncclab-sustech/BrainFLORA.