Learn to Combine Modalities in Multimodal Deep Learning

TL;DR

This paper introduces a novel deep learning method that multiplicatively combines multiple modalities to enhance classification accuracy by focusing on more reliable sources and capturing cross-modal correlations.

Contribution

It proposes a multiplicative fusion technique for multimodal data that automatically emphasizes reliable modalities and models cross-modal interactions, improving performance.

Findings

Consistent accuracy improvements across three multimodal classification tasks.

Effective filtering of noise and conflicts between modalities.

Enhanced modeling of cross-modal signal correlations.

Abstract

Combining complementary information from multiple modalities is intuitively appealing for improving the performance of learning-based approaches. However, it is challenging to fully leverage different modalities due to practical challenges such as varying levels of noise and conflicts between modalities. Existing methods do not adopt a joint approach to capturing synergies between the modalities while simultaneously filtering noise and resolving conflicts on a per sample basis. In this work we propose a novel deep neural network based technique that multiplicatively combines information from different source modalities. Thus the model training process automatically focuses on information from more reliable modalities while reducing emphasis on the less reliable modalities. Furthermore, we propose an extension that multiplicatively combines not only the single-source modalities, but a set of mixtured source modalities to better capture cross-modal signal correlations. We demonstrate the effectiveness of our proposed technique by presenting empirical results on three multimodal classification tasks from different domains. The results show consistent accuracy improvements on all three tasks.