TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognition

TL;DR

TiCAL introduces a novel framework for multimodal emotion recognition that dynamically assesses and leverages inter-modal consistency, embedding features in hyperbolic space to improve accuracy, especially on inconsistent samples.

Contribution

The paper proposes TiCAL, a framework that addresses inter-modal conflicts by estimating typicality and consistency, and embeds features in hyperbolic space for better emotion differentiation.

Findings

Achieves about 2.6% accuracy improvement over state-of-the-art methods.

Effectively mitigates inter-modal emotional conflicts.

Enhances emotion recognition on benchmark datasets.

Abstract

Multimodal Emotion Recognition (MER) aims to accurately identify human emotional states by integrating heterogeneous modalities such as visual, auditory, and textual data. Existing approaches predominantly rely on unified emotion labels to supervise model training, often overlooking a critical challenge: inter-modal emotion conflicts, wherein different modalities within the same sample may express divergent emotional tendencies. In this work, we address this overlooked issue by proposing a novel framework, Typicality-based Consistent-aware Multimodal Emotion Recognition (TiCAL), inspired by the stage-wise nature of human emotion perception. TiCAL dynamically assesses the consistency of each training sample by leveraging pseudo unimodal emotion labels alongside a typicality estimation. To further enhance emotion representation, we embed features in a hyperbolic space, enabling the capture of fine-grained distinctions among emotional categories. By incorporating consistency estimates into the learning process, our method improves model performance, particularly on samples exhibiting high modality inconsistency. Extensive experiments on benchmark datasets, e.g, CMU-MOSEI and MER2023, validate the effectiveness of TiCAL in mitigating inter-modal emotional conflicts and enhancing overall recognition accuracy, e.g., with about 2.6% improvements over the state-of-the-art DMD.