Improving Multimodal fusion via Mutual Dependency Maximisation

TL;DR

This paper introduces new dependency-based loss functions for multimodal fusion in sentiment analysis, significantly improving accuracy and robustness across multiple models and datasets.

Contribution

It proposes novel dependency maximisation penalties for multimodal fusion, leading to state-of-the-art results and more interpretable high-dimensional representations.

Findings

Achieved up to 4.3% accuracy improvement

Set new SOTA on CMU-MOSI and CMU-MOSEI datasets

Produced more robust and interpretable multimodal representations

Abstract

Multimodal sentiment analysis is a trending area of research, and the multimodal fusion is one of its most active topic. Acknowledging humans communicate through a variety of channels (i.e visual, acoustic, linguistic), multimodal systems aim at integrating different unimodal representations into a synthetic one. So far, a consequent effort has been made on developing complex architectures allowing the fusion of these modalities. However, such systems are mainly trained by minimising simple losses such as $L_1$ or cross-entropy. In this work, we investigate unexplored penalties and propose a set of new objectives that measure the dependency between modalities. We demonstrate that our new penalties lead to a consistent improvement (up to $4.3$ on accuracy) across a large variety of state-of-the-art models on two well-known sentiment analysis datasets: \texttt{CMU-MOSI} and \texttt{CMU-MOSEI}. Our method not only achieves a new SOTA on both datasets but also produces representations that are more robust to modality drops. Finally, a by-product of our methods includes a statistical network which can be used to interpret the high dimensional representations learnt by the model.