Subject-Aware Multi-Granularity Alignment for Zero-Shot EEG-to-Image Retrieval

TL;DR

This paper introduces SAMGA, a subject-aware multi-granularity alignment framework that improves zero-shot EEG-to-image retrieval by adaptively aligning neural responses with visual representations across multiple scales.

Contribution

The proposed SAMGA framework adaptively constructs visual supervision targets and employs a coarse-to-fine alignment strategy to enhance cross-subject EEG-to-image retrieval performance.

Findings

Achieves 91.3% Top-1 accuracy intra-subject on THINGS-EEG.

Achieves 34.4% Top-1 accuracy inter-subject on THINGS-EEG.

Outperforms recent state-of-the-art methods in EEG-to-image retrieval.

Abstract

Zero-shot EEG-to-image retrieval aims to decode perceived visual content from electroencephalography (EEG) by aligning neural responses with pretrained visual representations, providing a promising route toward scalable visual neural decoding and practical brain-computer interfaces. However, robust EEG-to-image retrieval remains challenging, because prior methods usually rely on either a single fixed visual target or a subject-invariant target construction scheme. Such designs overlook two important properties of visually evoked EEG signals: they preserve information across multiple representational scales, and the visual granularity best matched to EEG may vary across subjects. To address these issues, subject-aware multi-granularity alignment (SAMGA) framework is proposed for zero-shot EEG-to-image retrieval. SAMGA first constructs a subject-aware visual supervision target by adaptively aggregating multiple intermediate representations from a pretrained vision encoder, allowing the model to absorb subject-dependent granularity deviations during training while preserving subject-agnostic inference. Building on this adaptive target construction, a coarse-to-fine cross-modal alignment strategy is further designed with a shared encoder wherein the coarse stage stabilizes the shared semantic geometry and reduces subject-induced distribution shift, and the fine stage further improves instance-level retrieval discrimination. Extensive experiments on the THINGS-EEG benchmark demonstrate that the proposed method achieves 91.3% Top-1 and 98.8% Top-5 accuracy in the intra-subject setting, and 34.4% Top-1 and 64.8% Top-5 accuracy in the inter-subject setting, outperforming recent state-of-the-art methods.