Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion

TL;DR

This paper introduces a contrastive learning framework to align EEG, MEG, and fMRI data for decoding, encoding, and converting neural representations of visual stimuli, advancing multimodal brain data analysis.

Contribution

It presents a novel multimodal neural alignment model that effectively integrates EEG, MEG, and fMRI data for comprehensive brain activity analysis.

Findings

Accurately decodes visual information from neural data

Successfully encodes images into neural representations

Enables conversion between different neural modalities

Abstract

This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across multimodal representationsof brain activity by leveraging contrastive learning. We used electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. Our framework's capabilities are demonstrated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities. The results highlight the model's ability to accurately capture semantic information across different brain imaging techniques, illustrating its potential in decoding, encoding, and modality conversion tasks.