Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning

TL;DR

This paper introduces CA-MER, a benchmark for emotion conflict scenarios in multimodal emotion reasoning, and proposes MoSEAR, a framework that balances modality contributions to improve emotion recognition accuracy.

Contribution

The paper presents CA-MER for evaluating emotion conflicts and proposes MoSEAR, a novel, parameter-efficient framework that mitigates modality bias in multimodal emotion reasoning.

Findings

MoSEAR reduces modality bias during emotion conflicts.

MoSEAR achieves state-of-the-art results on multiple benchmarks.

Balanced modality integration improves emotion recognition accuracy.

Abstract

Despite their strong performance in multimodal emotion reasoning, existing Multimodal Large Language Models (MLLMs) often overlook the scenarios involving emotion conflicts, where emotional cues from different modalities are inconsistent. To fill this gap, we first introduce CA-MER, a new benchmark designed to examine MLLMs under realistic emotion conflicts. It consists of three subsets: video-aligned, audio-aligned, and consistent, where only one or all modalities reflect the true emotion. However, evaluations on our CA-MER reveal that current state-of-the-art emotion MLLMs systematically over-rely on audio signal during emotion conflicts, neglecting critical cues from visual modality. To mitigate this bias, we propose MoSEAR, a parameter-efficient framework that promotes balanced modality integration. MoSEAR consists of two modules: (1)MoSE, modality-specific experts with a regularized gating mechanism that reduces modality bias in the fine-tuning heads; and (2)AR, an attention reallocation mechanism that rebalances modality contributions in frozen backbones during inference. Our framework offers two key advantages: it mitigates emotion conflicts and improves performance on consistent samples-without incurring a trade-off between audio and visual modalities. Experiments on multiple benchmarks-including MER2023, EMER, DFEW, and our CA-MER-demonstrate that MoSEAR achieves state-of-the-art performance, particularly under modality conflict conditions.