PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysis

TL;DR

This paper introduces PaSE, a novel framework for multimodal sentiment analysis that improves modality collaboration and reduces competition among modalities through prototype alignment and Shapley-based optimization.

Contribution

PaSE combines prototype-guided calibration, optimal transport, and Shapley-based gradient modulation to enhance multimodal fusion and mitigate modality competition.

Findings

PaSE outperforms existing methods on IEMOCAP, MOSI, and MOSEI datasets.

It effectively alleviates modality competition in multimodal sentiment analysis.

Experimental results show improved accuracy and robustness.

Abstract

Multimodal Sentiment Analysis (MSA) seeks to understand human emotions by integrating textual, acoustic, and visual signals. Although multimodal fusion is designed to leverage cross-modal complementarity, real-world scenarios often exhibit modality competition: dominant modalities tend to overshadow weaker ones, leading to suboptimal performance. In this paper, we propose PaSE, a novel Prototype-aligned Calibration and Shapley-optimized Equilibrium framework, which enhances collaboration while explicitly mitigating modality competition. PaSE first applies Prototype-guided Calibration Learning (PCL) to refine unimodal representations and align them through an Entropic Optimal Transport mechanism that ensures semantic consistency. To further stabilize optimization, we introduce a Dual-Phase Optimization strategy. A prototype-gated fusion module is first used to extract shared representations, followed by Shapley-based Gradient Modulation (SGM), which adaptively adjusts gradients according to the contribution of each modality. Extensive experiments on IEMOCAP, MOSI, and MOSEI confirm that PaSE achieves the superior performance and effectively alleviates modality competition.