Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum

TL;DR

This paper introduces GS-MCC, a graph-spectrum-based framework that enhances multimodal emotion recognition in conversations by effectively capturing long-distance and frequency-specific semantic features, outperforming existing GNN methods.

Contribution

The paper proposes a novel graph-spectrum approach using Fourier graph operators and contrastive learning to better model semantic consistency and complementarity in multimodal dialogue data.

Findings

GS-MCC outperforms existing methods on benchmark datasets.

Effective extraction of high and low-frequency information improves emotion recognition.

Contrastive learning enhances the collaboration of semantic features.

Abstract

Efficiently capturing consistent and complementary semantic features in a multimodal conversation context is crucial for Multimodal Emotion Recognition in Conversation (MERC). Existing methods mainly use graph structures to model dialogue context semantic dependencies and employ Graph Neural Networks (GNN) to capture multimodal semantic features for emotion recognition. However, these methods are limited by some inherent characteristics of GNN, such as over-smoothing and low-pass filtering, resulting in the inability to learn long-distance consistency information and complementary information efficiently. Since consistency and complementarity information correspond to low-frequency and high-frequency information, respectively, this paper revisits the problem of multimodal emotion recognition in conversation from the perspective of the graph spectrum. Specifically, we propose a Graph-Spectrum-based Multimodal Consistency and Complementary collaborative learning framework GS-MCC. First, GS-MCC uses a sliding window to construct a multimodal interaction graph to model conversational relationships and uses efficient Fourier graph operators to extract long-distance high-frequency and low-frequency information, respectively. Then, GS-MCC uses contrastive learning to construct self-supervised signals that reflect complementarity and consistent semantic collaboration with high and low-frequency signals, thereby improving the ability of high and low-frequency information to reflect real emotions. Finally, GS-MCC inputs the collaborative high and low-frequency information into the MLP network and softmax function for emotion prediction. Extensive experiments have proven the superiority of the GS-MCC architecture proposed in this paper on two benchmark data sets.