TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis

TL;DR

This paper introduces TCAN, a multimodal sentiment analysis model that emphasizes the dominant role of text by using cross-attention mechanisms, improving performance by addressing modality heterogeneity.

Contribution

The paper proposes a novel Text-oriented Cross-Attention Network (TCAN) that prioritizes text modality and incorporates gated control and joint learning to enhance multimodal sentiment analysis.

Findings

TCAN outperforms state-of-the-art methods on CMU-MOSI and CMU-MOSEI datasets.

The model effectively mitigates noise and redundancy through gated control.

Focusing on text improves sentiment prediction accuracy.

Abstract

Multimodal Sentiment Analysis (MSA) endeavors to understand human sentiment by leveraging language, visual, and acoustic modalities. Despite the remarkable performance exhibited by previous MSA approaches, the presence of inherent multimodal heterogeneities poses a challenge, with the contribution of different modalities varying considerably. Past research predominantly focused on improving representation learning techniques and feature fusion strategies. However, many of these efforts overlooked the variation in semantic richness among different modalities, treating each modality uniformly. This approach may lead to underestimating the significance of strong modalities while overemphasizing the importance of weak ones. Motivated by these insights, we introduce a Text-oriented Cross-Attention Network (TCAN), emphasizing the predominant role of the text modality in MSA. Specifically, for each multimodal sample, by taking unaligned sequences of the three modalities as inputs, we initially allocate the extracted unimodal features into a visual-text and an acoustic-text pair. Subsequently, we implement self-attention on the text modality and apply text-queried cross-attention to the visual and acoustic modalities. To mitigate the influence of noise signals and redundant features, we incorporate a gated control mechanism into the framework. Additionally, we introduce unimodal joint learning to gain a deeper understanding of homogeneous emotional tendencies across diverse modalities through backpropagation. Experimental results demonstrate that TCAN consistently outperforms state-of-the-art MSA methods on two datasets (CMU-MOSI and CMU-MOSEI).