Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation

TL;DR

This paper introduces a two-phase Divide and Refine framework that explicitly decomposes and enhances multimodal representations for emotion recognition in conversation, improving performance across multiple models.

Contribution

It presents a novel Divide and Refine approach that explicitly decomposes multimodal signals into unique, redundant, and synergistic components, and refines them for better emotion recognition.

Findings

Consistent performance improvements on IEMOCAP and MELD datasets.

Effective enhancement of multimodal representations across various MERC models.

Demonstrated the importance of explicit decomposition and refinement in multimodal learning.

Abstract

Multimodal emotion recognition in conversation (MERC) requires representations that effectively integrate signals from multiple modalities. These signals include modality-specific cues, information shared across modalities, and interactions that emerge only when modalities are combined. In information-theoretic terms, these correspond to \emph{unique}, \emph{redundant}, and \emph{synergistic} contributions. An ideal representation should leverage all three, yet achieving such balance remains challenging. Recent advances in contrastive learning and augmentation-based methods have made progress, but they often overlook the role of data preparation in preserving these components. In particular, applying augmentations directly to raw inputs or fused embeddings can blur the boundaries between modality-unique and cross-modal signals. To address this challenge, we propose a two-phase framework \emph{\textbf{D}ivide and \textbf{R}efine} (\textbf{DnR}). In the \textbf{Divide} phase, each modality is explicitly decomposed into uniqueness, pairwise redundancy, and synergy. In the \textbf{Refine} phase, tailored objectives enhance the informativeness of these components while maintaining their distinct roles. The refined representations are plug-and-play compatible with diverse multimodal pipelines. Extensive experiments on IEMOCAP and MELD demonstrate consistent improvements across multiple MERC backbones. These results highlight the effectiveness of explicitly dividing, refining, and recombining multimodal representations as a principled strategy for advancing emotion recognition. Our implementation is available at https://github.com/mattam301/DnR-WACV2026