A Closer Look at Multimodal Representation Collapse

TL;DR

This paper investigates the phenomenon of modality collapse in multimodal models, revealing its causes, proposing a disentanglement solution via cross-modal distillation, and demonstrating its effectiveness through extensive experiments.

Contribution

It provides a theoretical understanding of modality collapse, introduces an algorithm to prevent it, and validates the approach on multiple benchmarks.

Findings

Modality collapse occurs due to entangled noisy features in the fusion head.

Cross-modal knowledge distillation helps disentangle representations and mitigate collapse.

The proposed algorithm effectively prevents modality collapse in various benchmarks.

Abstract

We aim to develop a fundamental understanding of modality collapse, a recently observed empirical phenomenon wherein models trained for multimodal fusion tend to rely only on a subset of the modalities, ignoring the rest. We show that modality collapse happens when noisy features from one modality are entangled, via a shared set of neurons in the fusion head, with predictive features from another, effectively masking out positive contributions from the predictive features of the former modality and leading to its collapse. We further prove that cross-modal knowledge distillation implicitly disentangles such representations by freeing up rank bottlenecks in the student encoder, denoising the fusion-head outputs without negatively impacting the predictive features from either modality. Based on the above findings, we propose an algorithm that prevents modality collapse through explicit basis reallocation, with applications in dealing with missing modalities. Extensive experiments on multiple multimodal benchmarks validate our theoretical claims. Project page: https://abhrac.github.io/mmcollapse/.