Enhancing Affective Representations of Music-Induced EEG through Multimodal Supervision and latent Domain Adaptation

TL;DR

This paper introduces a multimodal deep learning framework that combines EEG signals and music data to improve emotion recognition and understand music-induced affective features, addressing data scarcity and variability issues.

Contribution

It proposes a novel bi-modal model with domain adaptation and emotion supervision to enhance personalized affective representations from music-induced EEG signals.

Findings

Improved emotion recognition accuracy using multimodal supervision.

Insights into the distribution and temporal dynamics of music-induced affective features.

Demonstrated potential for stimulus-based enhancement of neuronal data for emotion recognition.

Abstract

The study of Music Cognition and neural responses to music has been invaluable in understanding human emotions. Brain signals, though, manifest a highly complex structure that makes processing and retrieving meaningful features challenging, particularly of abstract constructs like affect. Moreover, the performance of learning models is undermined by the limited amount of available neuronal data and their severe inter-subject variability. In this paper we extract efficient, personalized affective representations from EEG signals during music listening. To this end, we employ music signals as a supervisory modality to EEG, aiming to project their semantic correspondence onto a common representation space. We utilize a bi-modal framework by combining an LSTM-based attention model to process EEG and a pre-trained model for music tagging, along with a reverse domain discriminator to align the distributions of the two modalities, further constraining the learning process with emotion tags. The resulting framework can be utilized for emotion recognition both directly, by performing supervised predictions from either modality, and indirectly, by providing relevant music samples to EEG input queries. The experimental findings show the potential of enhancing neuronal data through stimulus information for recognition purposes and yield insights into the distribution and temporal variance of music-induced affective features.