Understanding Unimodal Bias in Multimodal Deep Linear Networks

TL;DR

This paper develops a theoretical framework to understand unimodal bias in multimodal deep linear networks, revealing how architecture and data influence the duration of unimodal reliance during training, with implications for generalization.

Contribution

First to analytically characterize the unimodal bias phase duration in multimodal deep linear networks based on fusion layer depth, data, and initialization.

Findings

Deeper fusion layers extend the unimodal phase duration.

Long unimodal phases can cause generalization issues and permanent bias.

Results extend to certain nonlinear network settings.

Abstract

Using multiple input streams simultaneously to train multimodal neural networks is intuitively advantageous but practically challenging. A key challenge is unimodal bias, where a network overly relies on one modality and ignores others during joint training. We develop a theory of unimodal bias with multimodal deep linear networks to understand how architecture and data statistics influence this bias. This is the first work to calculate the duration of the unimodal phase in learning as a function of the depth at which modalities are fused within the network, dataset statistics, and initialization. We show that the deeper the layer at which fusion occurs, the longer the unimodal phase. A long unimodal phase can lead to a generalization deficit and permanent unimodal bias in the overparametrized regime. Our results, derived for multimodal linear networks, extend to nonlinear networks in certain settings. Taken together, this work illuminates pathologies of multimodal learning under joint training, showing that late and intermediate fusion architectures can give rise to long unimodal phases and permanent unimodal bias. Our code is available at: https://yedizhang.github.io/unimodal-bias.html.