Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations

TL;DR

This paper introduces CBERL, a novel multimodal emotion recognition model that effectively handles imbalanced data, improving accuracy especially on minority emotion classes through data augmentation, feature fusion, and boundary learning techniques.

Contribution

The paper proposes a comprehensive framework combining data augmentation, feature fusion, and boundary learning to address class imbalance in multimodal emotion recognition.

Findings

Improved accuracy on minority classes by 10-20%.

Enhanced feature representations through multimodal fusion.

Effective handling of imbalanced emotion datasets.

Abstract

The main task of Multimodal Emotion Recognition in Conversations (MERC) is to identify the emotions in modalities, e.g., text, audio, image and video, which is a significant development direction for realizing machine intelligence. However, many data in MERC naturally exhibit an imbalanced distribution of emotion categories, and researchers ignore the negative impact of imbalanced data on emotion recognition. To tackle this problem, we systematically analyze it from three aspects: data augmentation, loss sensitivity, and sampling strategy, and propose the Class Boundary Enhanced Representation Learning (CBERL) model. Concretely, we first design a multimodal generative adversarial network to address the imbalanced distribution of {emotion} categories in raw data. Secondly, a deep joint variational autoencoder is proposed to fuse complementary semantic information across modalities and obtain discriminative feature representations. Finally, we implement a multi-task graph neural network with mask reconstruction and classification optimization to solve the problem of overfitting and underfitting in class boundary learning, and achieve cross-modal emotion recognition. We have conducted extensive experiments on the IEMOCAP and MELD benchmark datasets, and the results show that CBERL has achieved a certain performance improvement in the effectiveness of emotion recognition. Especially on the minority class fear and disgust emotion labels, our model improves the accuracy and F1 value by 10% to 20%.