3D-Telepathy: Reconstructing 3D Objects from EEG Signals

TL;DR

This paper introduces a novel method to reconstruct 3D objects from EEG signals by combining a specialized encoder, attention mechanisms, and diffusion models, advancing Brain-Computer Interface capabilities.

Contribution

It presents an innovative EEG encoder with dual self-attention and a hybrid training strategy, enabling 3D object reconstruction from noisy EEG data using diffusion prior and neural radiance fields.

Findings

Successful 3D object reconstruction from EEG signals

Enhanced neural network performance with attention mechanisms

Effective use of diffusion models for 3D generation

Abstract

Reconstructing 3D visual stimuli from Electroencephalography (EEG) data holds significant potential for applications in Brain-Computer Interfaces (BCIs) and aiding individuals with communication disorders. Traditionally, efforts have focused on converting brain activity into 2D images, neglecting the translation of EEG data into 3D objects. This limitation is noteworthy, as the human brain inherently processes three-dimensional spatial information regardless of whether observing 2D images or the real world. The neural activities captured by EEG contain rich spatial information that is inevitably lost when reconstructing only 2D images, thus limiting its practical applications in BCI. The transition from EEG data to 3D object reconstruction faces considerable obstacles. These include the presence of extensive noise within EEG signals and a scarcity of datasets that include both EEG and 3D information, which complicates the extraction process of 3D visual data. Addressing this challenging task, we propose an innovative EEG encoder architecture that integrates a dual self-attention mechanism. We use a hybrid training strategy to train the EEG Encoder, which includes cross-attention, contrastive learning, and self-supervised learning techniques. Additionally, by employing stable diffusion as a prior distribution and utilizing Variational Score Distillation to train a neural radiation field, we successfully generate 3D objects with similar content and structure from EEG data.