TMDC: A Two-Stage Modality Denoising and Complementation Framework for Multimodal Sentiment Analysis with Missing and Noisy Modalities

TL;DR

This paper introduces TMDC, a two-stage framework for multimodal sentiment analysis that effectively handles missing and noisy data by denoising and complementing modalities, leading to improved robustness and state-of-the-art results.

Contribution

The paper proposes a novel two-stage framework that jointly addresses modality noise and missing data in multimodal sentiment analysis, enhancing robustness and accuracy.

Findings

TMDC outperforms existing methods on MOSI, MOSEI, and IEMOCAP datasets.

The framework achieves new state-of-the-art results in multimodal sentiment analysis.

Extensive evaluations validate the effectiveness of the two-stage denoising and complementation approach.

Abstract

Multimodal Sentiment Analysis (MSA) aims to infer human sentiment by integrating information from multiple modalities such as text, audio, and video. In real-world scenarios, however, the presence of missing modalities and noisy signals significantly hinders the robustness and accuracy of existing models. While prior works have made progress on these issues, they are typically addressed in isolation, limiting overall effectiveness in practical settings. To jointly mitigate the challenges posed by missing and noisy modalities, we propose a framework called Two-stage Modality Denoising and Complementation (TMDC). TMDC comprises two sequential training stages. In the Intra-Modality Denoising Stage, denoised modality-specific and modality-shared representations are extracted from complete data using dedicated denoising modules, reducing the impact of noise and enhancing representational robustness. In the Inter-Modality Complementation Stage, these representations are leveraged to compensate for missing modalities, thereby enriching the available information and further improving robustness. Extensive evaluations on MOSI, MOSEI, and IEMOCAP demonstrate that TMDC consistently achieves superior performance compared to existing methods, establishing new state-of-the-art results.