Cross-Space Synergy: A Unified Framework for Multimodal Emotion Recognition in Conversation

TL;DR

This paper introduces Cross-Space Synergy, a unified framework for multimodal emotion recognition in conversations that effectively captures complex cross-modal interactions and stabilizes training.

Contribution

It proposes a novel framework combining Polynomial Fusion and Pareto Gradient Modulation to enhance multimodal emotion recognition performance and training stability.

Findings

Outperforms existing methods on IEMOCAP and MELD datasets

Improves training stability in complex multimodal scenarios

Effectively captures high-order cross-modal interactions

Abstract

Multimodal Emotion Recognition in Conversation (MERC) aims to predict speakers' emotions by integrating textual, acoustic, and visual cues. Existing approaches either struggle to capture complex cross-modal interactions or experience gradient conflicts and unstable training when using deeper architectures. To address these issues, we propose Cross-Space Synergy (CSS), which couples a representation component with an optimization component. Synergistic Polynomial Fusion (SPF) serves the representation role, leveraging low-rank tensor factorization to efficiently capture high-order cross-modal interactions. Pareto Gradient Modulator (PGM) serves the optimization role, steering updates along Pareto-optimal directions across competing objectives to alleviate gradient conflicts and improve stability. Experiments show that CSS outperforms existing representative methods on IEMOCAP and MELD in both accuracy and training stability, demonstrating its effectiveness in complex multimodal scenarios.