Hypergraph Multi-Modal Learning for EEG-based Emotion Recognition in Conversation

TL;DR

This paper introduces Hyper-MML, a novel multi-modal learning framework that effectively integrates EEG, audio, and video data for improved emotion recognition in conversation, with potential healthcare applications.

Contribution

The paper presents a new Hypergraph Multi-Modal Learning framework with specialized modules for EEG processing and higher-order signal fusion, advancing emotion recognition technology.

Findings

Hyper-MML outperforms existing methods on EAV and AFFEC datasets.

The ABEMA module effectively captures emotion-relevant EEG features.

The AHFM module models complex relationships among multi-modal signals.

Abstract

Emotional Recognition in Conversation (ERC) is valuable for diagnosing health conditions such as autism and depression, and for understanding the emotions of individuals who struggle to express their feelings. Current ERC methods primarily rely on semantic, audio and visual data but face significant challenges in integrating physiological signals such as Electroencephalography (EEG). This research proposes Hypergraph Multi-Modal Learning (Hyper-MML), a novel framework for identifying emotions in conversation. Hyper-MML effectively integrates EEG with audio and video information to capture complex emotional dynamics. Firstly, we introduce an Adaptive Brain Encoder with Mutual-cross Attention (ABEMA) module for processing EEG signals. This module captures emotion-relevant features across different frequency bands and adapts to subject-specific variations through hierarchical mutual-cross attention mechanisms. Secondly, we propose an Adaptive Hypergraph Fusion Module (AHFM) to actively model the higher-order relationships among multi-modal signals in ERC. Experimental results on the EAV and AFFEC datasets demonstrate that our Hyper-MML model significantly outperforms current state-of-the-art methods. The proposed Hyper-MML can serve as an effective communication tool for healthcare professionals, enabling better engagement with patients who have difficulty expressing their emotions. The official implementation codes are available at https://github.com/NZWANG/Hyper-MML.