Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis

TL;DR

This paper proposes MITR-DNet, a novel framework for missing multimodal sentiment analysis that uses distillation and bidirectional temporal learning to handle incomplete data and modality heterogeneity effectively.

Contribution

The paper introduces MITR-DNet, combining a distillation approach with bidirectional temporal representation learning to improve robustness in missing modality scenarios.

Findings

Outperforms existing methods on multimodal sentiment analysis benchmarks.

Effectively handles missing modalities with high accuracy.

Reduces heterogeneity impact in multimodal data.

Abstract

Multimodal Sentiment Analysis (MSA) integrates diverse modalities(text, audio, and video) to comprehensively analyze and understand individuals' emotional states. However, the real-world prevalence of incomplete data poses significant challenges to MSA, mainly due to the randomness of modality missing. Moreover, the heterogeneity issue in multimodal data has yet to be effectively addressed. To tackle these challenges, we introduce the Modality-Invariant Bidirectional Temporal Representation Distillation Network (MITR-DNet) for Missing Multimodal Sentiment Analysis. MITR-DNet employs a distillation approach, wherein a complete modality teacher model guides a missing modality student model, ensuring robustness in the presence of modality missing. Simultaneously, we developed the Modality-Invariant Bidirectional Temporal Representation Learning Module (MIB-TRL) to mitigate heterogeneity.