Mapping EEG Signals to Visual Stimuli: A Deep Learning Approach to Match vs. Mismatch Classification

TL;DR

This paper introduces a deep learning model that classifies whether visual stimuli induce excitatory EEG responses, effectively handling inter-subject variability and identifying brain regions involved in processing visual and language information.

Contribution

The study presents a novel match-vs-mismatch deep learning approach that improves EEG-visual stimulus association modeling and generalizes better across subjects compared to existing methods.

Findings

Achieves highest accuracy on unseen subjects.

Reduces inter-subject noise as shown by silhouette scores.

Identifies language and visual processing regions as key contributors.

Abstract

Existing approaches to modeling associations between visual stimuli and brain responses are facing difficulties in handling between-subject variance and model generalization. Inspired by the recent progress in modeling speech-brain response, we propose in this work a "match-vs-mismatch" deep learning model to classify whether a video clip induces excitatory responses in recorded EEG signals and learn associations between the visual content and corresponding neural recordings. Using an exclusive experimental dataset, we demonstrate that the proposed model is able to achieve the highest accuracy on unseen subjects as compared to other baseline models. Furthermore, we analyze the inter-subject noise using a subject-level silhouette score in the embedding space and show that the developed model is able to mitigate inter-subject noise and significantly reduce the silhouette score. Moreover, we examine the Grad-CAM activation score and show that the brain regions associated with language processing contribute most to the model predictions, followed by regions associated with visual processing. These results have the potential to facilitate the development of neural recording-based video reconstruction and its related applications.