Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning

TL;DR

This paper introduces MUSE, a contrastive learning framework that enables zero-shot image classification from EEG signals, achieving state-of-the-art accuracy and providing insights into neural visual processing.

Contribution

The paper presents a novel multimodal contrastive learning approach for EEG-based image recognition, with specialized encoders and extensive pretraining for improved zero-shot performance.

Findings

Achieved 19.3% top-1 accuracy in 200-way zero-shot classification.

Attained 48.8% top-5 accuracy in the same setting.

Provided neural pattern visualization to interpret visual processing in the brain.

Abstract

Decoding images from non-invasive electroencephalographic (EEG) signals has been a grand challenge in understanding how the human brain process visual information in real-world scenarios. To cope with the issues of signal-to-noise ratio and nonstationarity, this paper introduces a MUltimodal Similarity-keeping contrastivE learning (MUSE) framework for zero-shot EEG-based image classification. We develop a series of multivariate time-series encoders tailored for EEG signals and assess the efficacy of regularized contrastive EEG-Image pretraining using an extensive visual EEG dataset. Our method achieves state-of-the-art performance, with a top-1 accuracy of 19.3% and a top-5 accuracy of 48.8% in 200-way zero-shot image classification. Furthermore, we visualize neural patterns via model interpretation, shedding light on the visual processing dynamics in the human brain. The code repository for this work is available at: https://github.com/ChiShengChen/MUSE_EEG.