RAMer: Reconstruction-based Adversarial Model for Multi-party Multi-modal Multi-label Emotion Recognition

TL;DR

RAMer is a novel model that enhances multi-modal emotion recognition in multi-party settings by reconstructing features, leveraging contrastive learning, and using auxiliary tasks to handle incomplete data and improve inter-modal and label correlations.

Contribution

It introduces a reconstruction-based adversarial framework with contrastive learning and auxiliary tasks to address modality incompleteness and interdependency in multi-party multi-modal emotion recognition.

Findings

Achieves state-of-the-art results on MEmoR, CMU-MOSEI, and M3ED benchmarks.

Effectively handles incomplete modalities in multi-party scenarios.

Improves emotion recognition accuracy through novel feature reconstruction and correlation strategies.

Abstract

Conventional Multi-modal multi-label emotion recognition (MMER) assumes complete access to visual, textual, and acoustic modalities. However, real-world multi-party settings often violate this assumption, as non-speakers frequently lack acoustic and textual inputs, leading to a significant degradation in model performance. Existing approaches also tend to unify heterogeneous modalities into a single representation, overlooking each modality's unique characteristics. To address these challenges, we propose RAMer (Reconstruction-based Adversarial Model for Emotion Recognition), which refines multi-modal representations by not only exploring modality commonality and specificity but crucially by leveraging reconstructed features, enhanced by contrastive learning, to overcome data incompleteness and enrich feature quality. RAMer also introduces a personality auxiliary task to complement missing modalities using modality-level attention, improving emotion reasoning. To further strengthen the model's ability to capture label and modality interdependency, we propose a stack shuffle strategy to enrich correlations between labels and modality-specific features. Experiments on three benchmarks, i.e., MEmoR, CMU-MOSEI, and $M^3ED$, demonstrate that RAMer achieves state-of-the-art performance in dyadic and multi-party MMER scenarios.