Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

TL;DR

This paper provides a theoretical explanation for why joint training of multi-modal neural networks often underperforms compared to uni-modal networks, highlighting the phenomenon of modality competition where some modalities are ignored.

Contribution

It introduces a theoretical framework demonstrating modality competition in joint training, explaining the performance gap in multi-modal deep learning.

Findings

Multi-modal networks with joint training exhibit modality competition.

Some modalities are ignored during training, leading to sub-optimal performance.

Experimental results confirm the theoretical predictions.

Abstract

Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network outperforms the jointly trained multi-modal network, which is counter-intuitive since multiple signals generally bring more information. This work provides a theoretical explanation for the emergence of such performance gap in neural networks for the prevalent joint training framework. Based on a simplified data distribution that captures the realistic property of multi-modal data, we prove that for the multi-modal late-fusion network with (smoothed) ReLU activation trained jointly by gradient descent, different modalities will compete with each other. The encoder networks will learn only a subset of modalities. We refer to this phenomenon as modality competition. The losing modalities, which fail to be discovered, are the origins where the sub-optimality of joint training comes from. Experimentally, we illustrate that modality competition matches the intrinsic behavior of late-fusion joint training.