Multi-Modality Collaborative Learning for Sentiment Analysis

TL;DR

This paper introduces a Multi-Modality Collaborative Learning framework for sentiment analysis that enhances cross-modal feature interaction and improves performance by capturing both shared and unique modality features.

Contribution

The paper proposes a novel MMCL framework with a parameter-free decoupling module and reinforcement learning-inspired policy models for adaptive feature mining across modalities.

Findings

Outperforms existing methods on four benchmark datasets.

Effectively captures complementary features across modalities.

Modules are validated to improve sentiment analysis accuracy.

Abstract

Multimodal sentiment analysis (MSA) identifies individuals' sentiment states in videos by integrating visual, audio, and text modalities. Despite progress in existing methods, the inherent modality heterogeneity limits the effective capture of interactive sentiment features across modalities. In this paper, by introducing a Multi-Modality Collaborative Learning (MMCL) framework, we facilitate cross-modal interactions and capture enhanced and complementary features from modality-common and modality-specific representations, respectively. Specifically, we design a parameter-free decoupling module and separate uni-modality into modality-common and modality-specific components through semantics assessment of cross-modal elements. For modality-specific representations, inspired by the act-reward mechanism in reinforcement learning, we design policy models to adaptively mine complementary sentiment features under the guidance of a joint reward. For modality-common representations, intra-modal attention is employed to highlight crucial components, playing enhanced roles among modalities. Experimental results, including superiority evaluations on four databases, effectiveness verification of each module, and assessment of complementary features, demonstrate that MMCL successfully learns collaborative features across modalities and significantly improves performance. The code can be available at https://github.com/smwanghhh/MMCL.