Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation

TL;DR

This paper introduces a novel EEG-guided audiovisual generation framework to reconstruct autobiographical memories with affective context, demonstrating high affect decoding accuracy and personalized content generation aligned with individual memory affect trajectories.

Contribution

It presents a new task, dataset, and a three-stage framework for affective memory reconstruction using EEG signals, advancing personalized media creation and affect decoding research.

Findings

Successful decoding of affect dynamics from EEG during memory recall (F1=0.9)

Faithful reconstruction of affective audiovisual memories across subjects

Higher affect and user preference correlation for affect-driven generated content

Abstract

In this paper, we introduce RevisitAffectiveMemory, a novel task designed to reconstruct autobiographical memories through audio-visual generation guided by affect extracted from electroencephalogram (EEG) signals. To support this pioneering task, we present the EEG-AffectiveMemory dataset, which encompasses textual descriptions, visuals, music, and EEG recordings collected during memory recall from nine participants. Furthermore, we propose RYM (Revisit Your Memory), a three-stage framework for generating synchronized audio-visual contents while maintaining dynamic personal memory affect trajectories. Experimental results demonstrate our method successfully decodes individual affect dynamics trajectories from neural signals during memory recall (F1=0.9). Also, our approach faithfully reconstructs affect-contextualized audio-visual memory across all subjects, both qualitatively and quantitatively, with participants reporting strong affective concordance between their recalled memories and the generated content. Especially, contents generated from subject-reported affect dynamics showed higher correlation with participants' reported affect dynamics trajectories (r=0.265, p<.05) and received stronger user preference (preference=56%) compared to those generated from randomly reordered affect dynamics. Our approaches advance affect decoding research and its practical applications in personalized media creation via neural-based affect comprehension. Codes and the dataset are available at https://github.com/ioahKwon/Revisiting-Your-Memory.