Mutual Information-based Representations Disentanglement for Unaligned Multimodal Language Sequences

TL;DR

This paper introduces a mutual information-based method for disentangling and integrating unaligned multimodal language sequences, reducing redundancy and improving model generalization by jointly learning modality-agnostic representations.

Contribution

It proposes a novel disentanglement framework that minimizes mutual information to eliminate redundancy and leverages unlabeled data to enhance performance and prevent overfitting.

Findings

Effective disentanglement of representations demonstrated on benchmark datasets.

Reduced information redundancy leads to better generalization.

Improved performance over existing methods in multimodal sequence tasks.

Abstract

The key challenge in unaligned multimodal language sequences lies in effectively integrating information from various modalities to obtain a refined multimodal joint representation. Recently, the disentangle and fuse methods have achieved the promising performance by explicitly learning modality-agnostic and modality-specific representations and then fusing them into a multimodal joint representation. However, these methods often independently learn modality-agnostic representations for each modality and utilize orthogonal constraints to reduce linear correlations between modality-agnostic and modality-specific representations, neglecting to eliminate their nonlinear correlations. As a result, the obtained multimodal joint representation usually suffers from information redundancy, leading to overfitting and poor generalization of the models. In this paper, we propose a Mutual Information-based Representations Disentanglement (MIRD) method for unaligned multimodal language sequences, in which a novel disentanglement framework is designed to jointly learn a single modality-agnostic representation. In addition, the mutual information minimization constraint is employed to ensure superior disentanglement of representations, thereby eliminating information redundancy within the multimodal joint representation. Furthermore, the challenge of estimating mutual information caused by the limited labeled data is mitigated by introducing unlabeled data. Meanwhile, the unlabeled data also help to characterize the underlying structure of multimodal data, consequently further preventing overfitting and enhancing the performance of the models. Experimental results on several widely used benchmark datasets validate the effectiveness of our proposed approach.