Adaptive Graph Learning with Multimodal Fusion for Emotion Recognition in Conversation

TL;DR

This paper introduces GASMER, a new model that improves emotion recognition in conversations by combining adaptive graph learning with multimodal data.

Contribution

The novelty lies in the unified architecture that adaptively learns graph structures for modeling conversation dependencies while fusing multimodal data.

Findings

GASMER outperforms existing graph-based approaches in emotion recognition.

It achieves a 2.7% accuracy improvement on IEMOCAP and 1.2% on MOSEI.

The model remains competitive against recent multimodal fusion models.

Abstract

Robust emotion recognition is a prerequisite for natural, fluid human–computer interaction, yet conversational settings remain challenging because emotions are shaped simultaneously by global topic flow and local speaker-to-speaker dependencies. Here, we introduce GASMER—Graph-Adaptive Structure for Multimodal Emotion Recognition—a unified architecture that tackles both issues. It uses the correlation structure based on graph neural networks (GNNs) to model the complex dependencies in the conversation, while adaptively learning the graph structure for GNNs. The experiments indicate that our model has strong performance that outperforms all existing graph-based approaches, and remains competitive when compared to recent multimodal fusion models, underscoring the importance of combining fine-grained multimodal fusion with adaptive graph learning for conversational emotion recognition. On the IEMOCAP dataset, GASMER improves accuracy by 2.7% and the weighted F1-score by 3.6% compared to the best baseline. On the MOSEI dataset, it achieves a 1.2% gain in binary classification accuracy (ACC-2).

Figures