Brain-Gen: Towards Interpreting Neural Signals for Stimulus Reconstruction Using Transformers and Latent Diffusion Models

TL;DR

This paper introduces a transformer and latent diffusion model-based framework to interpret EEG signals and reconstruct visual stimuli, improving semantic understanding and generalization in neural decoding.

Contribution

It presents a novel approach combining transformers and latent diffusion models for EEG-based stimulus reconstruction, enhancing interpretability and zero-shot generalization.

Findings

Up to 6.5% increase in latent space clustering accuracy

11.8% improvement in zero-shot generalization

Comparable Inception Score and FID with existing methods

Abstract

Advances in neuroscience and artificial intelligence have enabled preliminary decoding of brain activity. However, despite the progress, the interpretability of neural representations remains limited. A significant challenge arises from the intrinsic properties of electroencephalography (EEG) signals, including high noise levels, spatial diffusion, and pronounced temporal variability. To interpret the neural mechanism underlying thoughts, we propose a transformers-based framework to extract spatial-temporal representations associated with observed visual stimuli from EEG recordings. These features are subsequently incorporated into the attention mechanisms of Latent Diffusion Models (LDMs) to facilitate the reconstruction of visual stimuli from brain activity. The quantitative evaluations on publicly available benchmark datasets demonstrate that the proposed method excels at modeling the semantic structures from EEG signals; achieving up to 6.5% increase in latent space clustering accuracy and 11.8% increase in zero shot generalization across unseen classes while having comparable Inception Score and Fr\'echet Inception Distance with existing baselines. Our work marks a significant step towards generalizable semantic interpretation of the EEG signals.