Multimodal Sentiment Analysis based on Multi-channel and Symmetric Mutual Promotion Feature Fusion

TL;DR

This paper introduces a novel multimodal sentiment analysis approach that employs multi-channel feature extraction and a symmetric mutual promotion fusion method, enhancing inter- and intra-modal feature interactions for improved emotion recognition.

Contribution

It proposes a dual-channel feature extraction framework combined with a symmetric mutual promotion fusion technique, addressing limitations of feature richness and inter-modal information utilization.

Findings

Enhanced feature representation through multi-channel extraction.

Improved inter-modal fusion via symmetric mutual promotion.

Outperforms existing methods on benchmark datasets.

Abstract

Multimodal sentiment analysis is a key technology in the fields of human-computer interaction and affective computing. Accurately recognizing human emotional states is crucial for facilitating smooth communication between humans and machines. Despite some progress in multimodal sentiment analysis research, numerous challenges remain. The first challenge is the limited and insufficiently rich features extracted from single modality data. Secondly, most studies focus only on the consistency of inter-modal feature information, neglecting the differences between features, resulting in inadequate feature information fusion. In this paper, we first extract multi-channel features to obtain more comprehensive feature information. We employ dual-channel features in both the visual and auditory modalities to enhance intra-modal feature representation. Secondly, we propose a symmetric mutual promotion (SMP) inter-modal feature fusion method. This method combines symmetric cross-modal attention mechanisms and self-attention mechanisms, where the cross-modal attention mechanism captures useful information from other modalities, and the self-attention mechanism models contextual information. This approach promotes the exchange of useful information between modalities, thereby strengthening inter-modal interactions. Furthermore, we integrate intra-modal features and inter-modal fused features, fully leveraging the complementarity of inter-modal feature information while considering feature information differences. Experiments conducted on two benchmark datasets demonstrate the effectiveness and superiority of our proposed method.