CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations

TL;DR

CMSBERT-CLR is a novel multimodal sentiment analysis model that integrates verbal and non-verbal context with contrastive learning to improve modality alignment and achieve state-of-the-art results.

Contribution

The paper introduces CMSBERT-CLR, combining context-driven modality shifting with contrastive learning for better multimodal representation alignment.

Findings

Achieves state-of-the-art performance on multimodal sentiment analysis tasks.

Effectively aligns linguistic, visual, and acoustic modalities within a shared embedding space.

Stabilizes training with exponential moving average and label smoothing.

Abstract

Multimodal sentiment analysis has become an increasingly popular research area as the demand for multimodal online content is growing. For multimodal sentiment analysis, words can have different meanings depending on the linguistic context and non-verbal information, so it is crucial to understand the meaning of the words accordingly. In addition, the word meanings should be interpreted within the whole utterance context that includes nonverbal information. In this paper, we present a Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations (CMSBERT-CLR), which incorporates the whole context's non-verbal and verbal information and aligns modalities more effectively through contrastive learning. First, we introduce a Context-driven Modality Shifting (CMS) to incorporate the non-verbal and verbal information within the whole context of the sentence utterance. Then, for improving the alignment of different modalities within a common embedding space, we apply contrastive learning. Furthermore, we use an exponential moving average parameter and label smoothing as optimization strategies, which can make the convergence of the network more stable and increase the flexibility of the alignment. In our experiments, we demonstrate that our approach achieves state-of-the-art results.