Calibrating Multimodal Consensus for Emotion Recognition

TL;DR

This paper introduces Calibrated Multimodal Consensus (CMC), a novel model for emotion recognition that addresses semantic inconsistencies and modality dominance issues, achieving superior performance across multiple datasets.

Contribution

The paper proposes CMC, which uses pseudo unimodal labels and a consensus-guided fusion process to improve multimodal emotion recognition accuracy.

Findings

CMC outperforms state-of-the-art methods on four datasets.

It shows robustness in scenarios with semantic inconsistencies.

The approach mitigates text modality dominance.

Abstract

In recent years, Multimodal Emotion Recognition (MER) has made substantial progress. Nevertheless, most existing approaches neglect the semantic inconsistencies that may arise across modalities, such as conflicting emotional cues between text and visual inputs. Besides, current methods are often dominated by the text modality due to its strong representational capacity, which can compromise recognition accuracy. To address these challenges, we propose a model termed Calibrated Multimodal Consensus (CMC). CMC introduces a Pseudo Label Generation Module (PLGM) to produce pseudo unimodal labels, enabling unimodal pretraining in a self-supervised fashion. It then employs a Parameter-free Fusion Module (PFM) and a Multimodal Consensus Router (MCR) for multimodal finetuning, thereby mitigating text dominance and guiding the fusion process toward a more reliable consensus. Experimental results demonstrate that CMC achieves performance on par with or superior to state-of-the-art methods across four datasets, CH-SIMS, CH-SIMS v2, CMU-MOSI, and CMU-MOSEI, and exhibits notable advantages in scenarios with semantic inconsistencies on CH-SIMS and CH-SIMS v2. The implementation of this work is publicly accessible at https://github.com/gw-zhong/CMC.