Orthogonal Disentanglement with Projected Feature Alignment for Multimodal Emotion Recognition in Conversation

TL;DR

This paper introduces OD-PFA, a novel framework for multimodal emotion recognition that effectively captures shared and modality-specific emotional cues by disentangling features and aligning shared semantics across modalities.

Contribution

The paper proposes a new orthogonal disentanglement and feature alignment framework that explicitly models both shared and modality-specific emotional information.

Findings

Outperforms state-of-the-art methods on IEMOCAP and MELD datasets.

Effectively captures modality-specific nuances like micro-expressions and tone.

Enhances semantic coherence across modalities through projected feature alignment.

Abstract

Multimodal Emotion Recognition in Conversation (MERC) significantly enhances emotion recognition performance by integrating complementary emotional cues from text, audio, and visual modalities. While existing methods commonly utilize techniques such as contrastive learning and cross-attention mechanisms to align cross-modal emotional semantics, they typically overlook modality-specific emotional nuances like micro-expressions, tone variations, and sarcastic language. To overcome these limitations, we propose Orthogonal Disentanglement with Projected Feature Alignment (OD-PFA), a novel framework designed explicitly to capture both shared semantics and modality-specific emotional cues. Our approach first decouples unimodal features into shared and modality-specific components. An orthogonal disentanglement strategy (OD) enforces effective separation between these components, aided by a reconstruction loss to maintain critical emotional information from each modality. Additionally, a projected feature alignment strategy (PFA) maps shared features across modalities into a common latent space and applies a cross-modal consistency alignment loss to enhance semantic coherence. Extensive evaluations on widely-used benchmark datasets, IEMOCAP and MELD, demonstrate effectiveness of our proposed OD-PFA multimodal emotion recognition tasks, as compared with the state-of-the-art approaches.