EEG-Driven Image Reconstruction with Saliency-Guided Diffusion Models

TL;DR

This paper introduces a novel EEG-driven image reconstruction method that combines neural signals with saliency maps using diffusion models, significantly improving image quality and alignment with human attention.

Contribution

It presents a dual-conditioning framework with EEG embeddings and saliency maps, fine-tunes diffusion models, and demonstrates superior reconstruction fidelity and spatial control.

Findings

Enhanced image quality over existing methods

Strong alignment with human visual attention

Effective neural decoding for medical and neuroadaptive applications

Abstract

Existing EEG-driven image reconstruction methods often overlook spatial attention mechanisms, limiting fidelity and semantic coherence. To address this, we propose a dual-conditioning framework that combines EEG embeddings with spatial saliency maps to enhance image generation. Our approach leverages the Adaptive Thinking Mapper (ATM) for EEG feature extraction and fine-tunes Stable Diffusion 2.1 via Low-Rank Adaptation (LoRA) to align neural signals with visual semantics, while a ControlNet branch conditions generation on saliency maps for spatial control. Evaluated on THINGS-EEG, our method achieves a significant improvement in the quality of low- and high-level image features over existing approaches. Simultaneously, strongly aligning with human visual attention. The results demonstrate that attentional priors resolve EEG ambiguities, enabling high-fidelity reconstructions with applications in medical diagnostics and neuroadaptive interfaces, advancing neural decoding through efficient adaptation of pre-trained diffusion models.