DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognition

TL;DR

This paper introduces DRKF, a novel multimodal emotion recognition method that decouples shared and specific features, fuses modalities with attention, and preserves emotional inconsistency cues to improve accuracy.

Contribution

DRKF combines contrastive mutual information estimation, a self-attention fusion encoder, and an emotion discrimination submodule to address modality heterogeneity and inconsistency in MER.

Findings

Achieves state-of-the-art results on IEMOCAP, MELD, and M3ED datasets.

Effectively handles emotional inconsistency through the ED submodule.

Outperforms existing methods in multimodal emotion recognition.

Abstract

Multimodal emotion recognition (MER) aims to identify emotional states by integrating and analyzing information from multiple modalities. However, inherent modality heterogeneity and inconsistencies in emotional cues remain key challenges that hinder performance. To address these issues, we propose a Decoupled Representations with Knowledge Fusion (DRKF) method for MER. DRKF consists of two main modules: an Optimized Representation Learning (ORL) Module and a Knowledge Fusion (KF) Module. ORL employs a contrastive mutual information estimation method with progressive modality augmentation to decouple task-relevant shared representations and modality-specific features while mitigating modality heterogeneity. KF includes a lightweight self-attention-based Fusion Encoder (FE) that identifies the dominant modality and integrates emotional information from other modalities to enhance the fused representation. To handle potential errors from incorrect dominant modality selection under emotionally inconsistent conditions, we introduce an Emotion Discrimination Submodule (ED), which enforces the fused representation to retain discriminative cues of emotional inconsistency. This ensures that even if the FE selects an inappropriate dominant modality, the Emotion Classification Submodule (EC) can still make accurate predictions by leveraging preserved inconsistency information. Experiments show that DRKF achieves state-of-the-art (SOTA) performance on IEMOCAP, MELD, and M3ED. The source code is publicly available at https://github.com/PANPANKK/DRKF.