Angle-Optimized Partial Disentanglement for Multimodal Emotion Recognition in Conversation

TL;DR

This paper introduces AO-FL, a novel framework for multimodal emotion recognition in conversation that employs angle-optimized partial disentanglement to better capture subtle emotional cues and improve recognition accuracy.

Contribution

The paper proposes an adaptive angular optimization method for partial feature disentanglement in MERC, enhancing the integration of shared and modality-specific features.

Findings

AO-FL outperforms state-of-the-art methods in MERC tasks.

The framework effectively preserves modality-specific cues like micro-expressions and sarcasm.

AO-FL generalizes well to other multimodal tasks such as MER.

Abstract

Multimodal Emotion Recognition in Conversation (MERC) aims to enhance emotion understanding by integrating complementary cues from text, audio, and visual modalities. Existing MERC approaches predominantly focus on cross-modal shared features, often overlooking modality-specific features that capture subtle yet critical emotional cues such as micro-expressions, prosodic variations, and sarcasm. Although related work in multimodal emotion recognition (MER) has explored disentangling shared and modality-specific features, these methods typically employ rigid orthogonal constraints to achieve full disentanglement, which neglects the inherent complementarity between feature types and may limit recognition performance. To address these challenges, we propose Angle-Optimized Feature Learning (AO-FL), a framework tailored for MERC that achieves partial disentanglement of shared and specific features within each modality through adaptive angular optimization. Specifically, AO-FL aligns shared features across modalities to ensure semantic consistency, and within each modality it adaptively models the angular relationship between its shared and modality-specific features to preserve both distinctiveness and complementarity. An orthogonal projection refinement further removes redundancy in specific features and enriches shared features with contextual information, yielding more discriminative multimodal representations. Extensive experiments confirm the effectiveness of AO-FL for MERC, demonstrating superior performance over state-of-the-art approaches. Moreover, AO-FL can be seamlessly integrated with various unimodal feature extractors and extended to other multimodal fusion tasks, such as MER, thereby highlighting its strong generalization beyond MERC.