MInD: Improving Multimodal Sentiment Analysis via Multimodal Information Disentanglement

TL;DR

This paper introduces MInD, a novel method for multimodal sentiment analysis that disentangles modality-invariant and modality-specific information, leading to improved performance by simplifying fusion and reducing noise.

Contribution

The paper proposes a disentanglement approach with shared and private encoders, and adversarial noise training, to better fuse multi-modal signals for sentiment analysis.

Findings

MInD outperforms existing methods on benchmark datasets.

Disentanglement improves multi-modal emotion recognition.

Effective noise isolation enhances representation quality.

Abstract

Learning effective joint representations has been a central task in multi-modal sentiment analysis. Previous works addressing this task focus on exploring sophisticated fusion techniques to enhance performance. However, the inherent heterogeneity of distinct modalities remains a core problem that brings challenges in fusing and coordinating the multi-modal signals at both the representational level and the informational level, impeding the full exploitation of multi-modal information. To address this problem, we propose the Multi-modal Information Disentanglement (MInD) method, which decomposes the multi-modal inputs into modality-invariant and modality-specific components through a shared encoder and multiple private encoders. Furthermore, by explicitly training generated noise in an adversarial manner, MInD is able to isolate uninformativeness, thus improves the learned representations. Therefore, the proposed disentangled decomposition allows for a fusion process that is simpler than alternative methods and results in improved performance. Experimental evaluations conducted on representative benchmark datasets demonstrate MInD's effectiveness in both multi-modal emotion recognition and multi-modal humor detection tasks. Code will be released upon acceptance of the paper.