Visual Neural Decoding via Improved Visual-EEG Semantic Consistency

TL;DR

This paper introduces a novel joint semantic space and two methods to improve visual-EEG decoding accuracy by enhancing semantic consistency and cross-modal alignment.

Contribution

It proposes a Visual-EEG Joint Semantic Space and two innovative approaches, VE-SDN and NGIC, to improve semantic alignment and decoding performance.

Findings

Achieved 38.9%/17.9% Top-1/Top-5 accuracy improvements intra-subject.

Demonstrated superior decoding performance over baseline methods.

Validated effectiveness on a large-scale Visual-EEG dataset.

Abstract

Visual neural decoding aims to extract and interpret original visual experiences directly from human brain activity. Recent studies have demonstrated the feasibility of decoding visual semantic categories from electroencephalography (EEG) signals, among which metric learning-based approaches have delivered promising results. However, these methods that directly map EEG features into a pre-trained embedding space inevitably introduce mapping bias, resulting in a modality gap and semantic inconsistency that impair cross-modal alignment. To address these issues, this work constructs a Visual-EEG Joint Semantic Space to bridge the gap between visual images and neural signals. Building upon this space, we propose two novel approaches to improve semantic consistency between cross-modal representations and facilitate optimal alignment. Specifically, (1) we introduce a Visual-EEG Semantic Decoupling Network (VE-SDN) to explicitly disentangle semantic components from modality representations, thereby achieving purely semantic-level cross-modal alignment. (2) We introduce a Neural-Guided Intra-Class Consistency (NGIC) objective, an asymmetric representation alignment strategy designed to effectively enhance the robustness of visual representations and further boost decoding performance. Extensive experiments on a large-scale Visual-EEG dataset validate the effectiveness of the proposed method. Compared to the strongest baseline, our approach demonstrates superior decoding performance, yielding relative Top-1/Top-5 accuracy improvements of 38.9%/17.9% in intra-subject and 16.1%/11.3% in inter-subject settings. The code is available at https://github.com/hzalanchen/Cross-Modal-EEG