Multimodal Fusion via Hypergraph Autoencoder and Contrastive Learning for Emotion Recognition in Conversation

TL;DR

This paper introduces a novel multimodal emotion recognition framework that uses a variational hypergraph autoencoder and contrastive learning to improve context modeling and fusion, outperforming existing methods on benchmark datasets.

Contribution

It proposes a dynamic hypergraph connection adjustment via VHGAE and incorporates contrastive learning to enhance multimodal emotion recognition accuracy.

Findings

Outperforms state-of-the-art on IEMOCAP and MELD datasets.

Effectively models long-distance conversational context.

Reduces redundancy and smoothing in hypergraph-based context modeling.

Abstract

Multimodal emotion recognition in conversation (MERC) seeks to identify the speakers' emotions expressed in each utterance, offering significant potential across diverse fields. The challenge of MERC lies in balancing speaker modeling and context modeling, encompassing both long-distance and short-distance contexts, as well as addressing the complexity of multimodal information fusion. Recent research adopts graph-based methods to model intricate conversational relationships effectively. Nevertheless, the majority of these methods utilize a fixed fully connected structure to link all utterances, relying on convolution to interpret complex context. This approach can inherently heighten the redundancy in contextual messages and excessive graph network smoothing, particularly in the context of long-distance conversations. To address this issue, we propose a framework that dynamically adjusts hypergraph connections by variational hypergraph autoencoder (VHGAE), and employs contrastive learning to mitigate uncertainty factors during the reconstruction process. Experimental results demonstrate the effectiveness of our proposal against the state-of-the-art methods on IEMOCAP and MELD datasets. We release the code to support the reproducibility of this work at https://github.com/yzjred/-HAUCL.