EEG-Driven 3D Object Reconstruction with Style Consistency and Diffusion Prior

TL;DR

This paper introduces a novel EEG-based 3D object reconstruction method that leverages style consistency and diffusion priors, combining neural encoding, diffusion models, and NeRF optimization to improve reconstruction quality.

Contribution

It proposes a new multi-task EEG encoding and a diffusion prior-based 3D reconstruction framework with style constraints, advancing EEG-driven 3D object generation.

Findings

Effective reconstruction of 3D objects from EEG data.

Improved style consistency between stimuli and reconstructions.

Demonstrated superiority over existing EEG-based reconstruction methods.

Abstract

Electroencephalography (EEG)-based visual perception reconstruction has become an important area of research. Neuroscientific studies indicate that humans can decode imagined 3D objects by perceiving or imagining various visual information, such as color, shape, and rotation. Existing EEG-based visual decoding methods typically focus only on the reconstruction of 2D visual stimulus images and face various challenges in generation quality, including inconsistencies in texture, shape, and color between the visual stimuli and the reconstructed images. This paper proposes an EEG-based 3D object reconstruction method with style consistency and diffusion priors. The method consists of an EEG-driven multi-task joint learning stage and an EEG-to-3D diffusion stage. The first stage uses a neural EEG encoder based on regional semantic learning, employing a multi-task joint learning scheme that includes a masked EEG signal recovery task and an EEG based visual classification task. The second stage introduces a latent diffusion model (LDM) fine-tuning strategy with style-conditioned constraints and a neural radiance field (NeRF) optimization strategy. This strategy explicitly embeds semantic- and location-aware latent EEG codes and combines them with visual stimulus maps to fine-tune the LDM. The fine-tuned LDM serves as a diffusion prior, which, combined with the style loss of visual stimuli, is used to optimize NeRF for generating 3D objects. Finally, through experimental validation, we demonstrate that this method can effectively use EEG data to reconstruct 3D objects with style consistency.