Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities

TL;DR

This paper introduces a two-level reference alignment framework to enhance multimodal sentiment analysis robustness when dealing with missing or unreliable modalities, achieving state-of-the-art results.

Contribution

It proposes a novel two-level reference alignment method that stabilizes sentiment predictions across diverse missing-modality scenarios in multimodal sentiment analysis.

Findings

Improved accuracy on CMU-MOSI and CMU-MOSEI datasets.

State-of-the-art performance with ACC of 86.28% and 85.88%.

Enhanced robustness under various missing-modality patterns.

Abstract

Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from available ones, but differences in expression mechanisms and sentiment dynamics across modalities may cause the generated features to deviate from true distributions and mislead prediction. In addition, unreliable modalities may dominate fusion, resulting in representation shift across modality combinations and unstable sentiment representations. To address these challenges, we propose a two-level reference alignment framework. The framework introduces stable references at the feature representation and sentiment decision levels to improve robustness under modality missing. First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space. Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting. As a result, the framework maintains stable and reliable sentiment predictions under diverse missing-modality patterns. Experiments on CMU-MOSI and CMU-MOSEI show consistent improvements across various missing-modality settings. Under full-modality input, the proposed method achieves state-of-the-art performance, with ACC of 86.28% and 85.88%, and F1 of 86.24% and 85.86%.