CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment

TL;DR

CognitionCapturerPro enhances visual stimulus reconstruction from EEG by integrating multi-modal priors and novel alignment techniques, significantly improving accuracy over previous methods.

Contribution

The paper introduces a multi-modal integration framework with an uncertainty-weighted scoring and simplified alignment, advancing EEG-based visual decoding.

Findings

Achieved 25.9% higher Top-1 accuracy on THINGS-EEG dataset.

Improved Top-5 retrieval accuracy by 10.6%.

Outperformed original CognitionCapturer significantly.

Abstract

Visual stimuli reconstruction from EEG remains challenging due to fidelity loss and representation shift. We propose CognitionCapturerPro, an enhanced framework that integrates EEG with multi-modal priors (images, text, depth, and edges) via collaborative training. Our core contributions include an uncertainty-weighted similarity scoring mechanism to quantify modality-specific fidelity and a fusion encoder for integrating shared representations. By employing a simplified alignment module and a pre-trained diffusion model, our method significantly outperforms the original CognitionCapturer on the THINGS-EEG dataset, improving Top-1 and Top-5 retrieval accuracy by 25.9% and 10.6%, respectively. Code is available at: https://github.com/XiaoZhangYES/CognitionCapturerPro.