Toward High-Fidelity Visual Reconstruction: From EEG-Based Conditioned Generation to Joint-Modal Guided Rebuilding

TL;DR

This paper introduces JMVR, a novel framework for high-fidelity visual reconstruction from EEG signals, leveraging joint-modal learning and multi-scale encoding to improve spatial and chromatic detail recovery.

Contribution

The paper proposes a joint-modal learning approach that treats EEG and text as independent modalities, along with multi-scale EEG encoding and image augmentation for enhanced reconstruction.

Findings

Achieves state-of-the-art performance on THINGS-EEG dataset.

Excels in modeling spatial structure and chromatic fidelity.

Outperforms six baseline methods.

Abstract

Human visual reconstruction aims to reconstruct fine-grained visual stimuli based on subject-provided descriptions and corresponding neural signals. As a widely adopted modality, Electroencephalography (EEG) captures rich visual cognition information, encompassing complex spatial relationships and chromatic details within scenes. However, current approaches are deeply coupled with an alignment framework that forces EEG features to align with text or image semantic representation. The dependency may condense the rich spatial and chromatic details in EEG that achieved mere conditioned image generation rather than high-fidelity visual reconstruction. To address this limitation, we propose a novel Joint-Modal Visual Reconstruction (JMVR) framework. It treats EEG and text as independent modalities for joint learning to preserve EEG-specific information for reconstruction. It further employs a multi-scale EEG encoding strategy to capture both fine- and coarse-grained features, alongside image augmentation to enhance the recovery of perceptual details. Extensive experiments on the THINGS-EEG dataset demonstrate that JMVR achieves SOTA performance against six baseline methods, specifically exhibiting superior capabilities in modeling spatial structure and chromatic fidelity.