Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities

TL;DR

This paper introduces PRLF, a novel framework for multimodal sentiment analysis that effectively handles incomplete modalities by dynamically estimating modality reliability and progressively aligning modalities, improving robustness and accuracy.

Contribution

The paper proposes PRLF, which includes an Adaptive Modality Reliability Estimator and a Progressive Interaction module, to improve multimodal sentiment analysis with missing data.

Findings

PRLF outperforms state-of-the-art methods on multiple datasets.

PRLF effectively handles both inter- and intra-modality missing scenarios.

PRLF demonstrates robustness and generalization in real-world noisy conditions.

Abstract

Multimodal Sentiment Analysis (MSA) seeks to infer human emotions by integrating textual, acoustic, and visual cues. However, existing approaches often rely on all modalities are completeness, whereas real-world applications frequently encounter noise, hardware failures, or privacy restrictions that result in missing modalities. There exists a significant feature misalignment between incomplete and complete modalities, and directly fusing them may even distort the well-learned representations of the intact modalities. To this end, we propose PRLF, a Progressive Representation Learning Framework designed for MSA under uncertain missing-modality conditions. PRLF introduces an Adaptive Modality Reliability Estimator (AMRE), which dynamically quantifies the reliability of each modality using recognition confidence and Fisher information to determine the dominant modality. In addition, the Progressive Interaction (ProgInteract) module iteratively aligns the other modalities with the dominant one, thereby enhancing cross-modal consistency while suppressing noise. Extensive experiments on CMU-MOSI, CMU-MOSEI, and SIMS verify that PRLF outperforms state-of-the-art methods across both inter- and intra-modality missing scenarios, demonstrating its robustness and generalization capability.