NECOMIMI: Neural-Cognitive Multimodal EEG-informed Image Generation with Diffusion Models

TL;DR

NECOMIMI presents a novel EEG-informed image generation framework using diffusion models, achieving state-of-the-art classification and introducing new evaluation metrics, while highlighting challenges in translating EEG signals into detailed images.

Contribution

This work introduces NECOMIMI, a pioneering EEG-to-image generation framework utilizing diffusion models, and proposes the CAT Score for evaluating semantic image quality from EEG data.

Findings

NERV EEG encoder achieves SoTA zero-shot classification performance

Model tends to generate abstract or generalized images like landscapes

Introduces the CAT Score as a new EEG-to-image evaluation metric

Abstract

NECOMIMI (NEural-COgnitive MultImodal EEG-Informed Image Generation with Diffusion Models) introduces a novel framework for generating images directly from EEG signals using advanced diffusion models. Unlike previous works that focused solely on EEG-image classification through contrastive learning, NECOMIMI extends this task to image generation. The proposed NERV EEG encoder demonstrates state-of-the-art (SoTA) performance across multiple zero-shot classification tasks, including 2-way, 4-way, and 200-way, and achieves top results in our newly proposed Category-based Assessment Table (CAT) Score, which evaluates the quality of EEG-generated images based on semantic concepts. A key discovery of this work is that the model tends to generate abstract or generalized images, such as landscapes, rather than specific objects, highlighting the inherent challenges of translating noisy and low-resolution EEG data into detailed visual outputs. Additionally, we introduce the CAT Score as a new metric tailored for EEG-to-image evaluation and establish a benchmark on the ThingsEEG dataset. This study underscores the potential of EEG-to-image generation while revealing the complexities and challenges that remain in bridging neural activity with visual representation.