Confidence-Aware Self-Distillation for Multimodal Sentiment Analysis with Incomplete Modalities

TL;DR

This paper introduces a confidence-aware self-distillation method for multimodal sentiment analysis that effectively handles incomplete modalities by modeling uncertainty with Student's t-distributions, leading to improved robustness and state-of-the-art results.

Contribution

It proposes a novel confidence-aware self-distillation strategy using probabilistic embeddings and a reparameterization module for robust multimodal sentiment analysis.

Findings

Achieves state-of-the-art performance on benchmark datasets.

Effectively models uncertainty in multimodal data.

Improves robustness to modality missingness.

Abstract

Multimodal sentiment analysis (MSA) aims to understand human sentiment through multimodal data. In real-world scenarios, practical factors often lead to uncertain modality missingness. Existing methods for handling modality missingness are based on data reconstruction or common subspace projections. However, these methods neglect the confidence in multimodal combinations and impose constraints on intra-class representation, hindering the capture of modality-specific information and resulting in suboptimal performance. To address these challenges, we propose a Confidence-Aware Self-Distillation (CASD) strategy that effectively incorporates multimodal probabilistic embeddings via a mixture of Student's $t$-distributions, enhancing its robustness by incorporating confidence and accommodating heavy-tailed properties. This strategy estimates joint distributions with uncertainty scores and reduces uncertainty in the student network by consistency distillation. Furthermore, we introduce a reparameterization representation module that facilitates CASD in robust multimodal learning by sampling embeddings from the joint distribution for the prediction module to calculate the task loss. As a result, the directional constraint from the loss minimization is alleviated by the sampled representation. Experimental results on three benchmark datasets demonstrate that our method achieves state-of-the-art performance.