Contrastive Knowledge Distillation for Robust Multimodal Sentiment Analysis

TL;DR

This paper proposes a novel contrastive knowledge distillation approach for multimodal sentiment analysis that effectively handles incomplete modalities with lower computational costs compared to traditional imputation methods.

Contribution

It introduces Multi-Modal Contrastive Knowledge Distillation (MM-CKD), a non-imputation-based method using multi-view supervised contrastive learning for robust video sentiment analysis.

Findings

Achieves competitive performance with reduced computational costs.

Effectively handles incomplete modalities without data imputation.

Improves model robustness through cross-modal and cross-sample knowledge transfer.

Abstract

Multimodal sentiment analysis (MSA) systems leverage information from different modalities to predict human sentiment intensities. Incomplete modality is an important issue that may cause a significant performance drop in MSA systems. By generative imputation, i.e., recovering the missing data from available data, systems may achieve robust performance but will lead to high computational costs. This paper introduces a knowledge distillation method, called `Multi-Modal Contrastive Knowledge Distillation' (MM-CKD), to address the issue of incomplete modality in video sentiment analysis with lower computation cost, as a novel non-imputation-based method. We employ Multi-view Supervised Contrastive Learning (MVSC) to transfer knowledge from a teacher model to student models. This approach not only leverages cross-modal knowledge but also introduces cross-sample knowledge with supervision, jointly improving the performance of both teacher and student models through online learning. Our method gives competitive results with significantly lower computational costs than state-of-the-art imputation-based methods.