Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

TL;DR

This paper introduces the Bi-Bimodal Fusion Network (BBFN), an end-to-end model that dynamically balances relevance and independence among modalities for improved multimodal sentiment analysis, outperforming state-of-the-art methods.

Contribution

The paper proposes a novel end-to-end bimodal fusion network with a relevance-separation mechanism and gated Transformer control for better multimodal sentiment analysis.

Findings

BBFN significantly outperforms SOTA on three datasets

The relevance-separation mechanism improves fusion quality

Gated Transformer enhances final sentiment prediction accuracy

Abstract

Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data. This research area's major concern lies in developing an extraordinary fusion scheme that can extract and integrate key information from various modalities. However, one issue that may restrict previous work to achieve a higher level is the lack of proper modeling for the dynamics of the competition between the independence and relevance among modalities, which could deteriorate fusion outcomes by causing the collapse of modality-specific feature space or introducing extra noise. To mitigate this, we propose the Bi-Bimodal Fusion Network (BBFN), a novel end-to-end network that performs fusion (relevance increment) and separation (difference increment) on pairwise modality representations. The two parts are trained simultaneously such that the combat between them is simulated. The model takes two bimodal pairs as input due to the known information imbalance among modalities. In addition, we leverage a gated control mechanism in the Transformer architecture to further improve the final output. Experimental results on three datasets (CMU-MOSI, CMU-MOSEI, and UR-FUNNY) verifies that our model significantly outperforms the SOTA. The implementation of this work is available at https://github.com/declare-lab/multimodal-deep-learning.