Diagnosing and Re-learning for Balanced Multimodal Learning

TL;DR

This paper introduces a Diagnosing mp; Re-learning approach that estimates each modality's learning state and adaptively re-initializes encoders to balance and enhance multimodal learning, addressing modality capacity limitations.

Contribution

It proposes a novel method that diagnoses modality learning states and re-initializes encoders to improve balance and performance in multimodal learning.

Findings

Outperforms existing methods across multiple modalities and frameworks.

Effectively balances learning by re-initializing encoders based on modality separability.

Demonstrates superior results on diverse multimodal datasets.

Abstract

To overcome the imbalanced multimodal learning problem, where models prefer the training of specific modalities, existing methods propose to control the training of uni-modal encoders from different perspectives, taking the inter-modal performance discrepancy as the basis. However, the intrinsic limitation of modality capacity is ignored. The scarcely informative modalities can be recognized as ``worse-learnt'' ones, which could force the model to memorize more noise, counterproductively affecting the multimodal model ability. Moreover, the current modality modulation methods narrowly concentrate on selected worse-learnt modalities, even suppressing the training of others. Hence, it is essential to consider the intrinsic limitation of modality capacity and take all modalities into account during balancing. To this end, we propose the Diagnosing \& Re-learning method. The learning state of each modality is firstly estimated based on the separability of its uni-modal representation space, and then used to softly re-initialize the corresponding uni-modal encoder. In this way, the over-emphasizing of scarcely informative modalities is avoided. In addition, encoders of worse-learnt modalities are enhanced, simultaneously avoiding the over-training of other modalities. Accordingly, multimodal learning is effectively balanced and enhanced. Experiments covering multiple types of modalities and multimodal frameworks demonstrate the superior performance of our simple-yet-effective method for balanced multimodal learning. The source code and dataset are available at \url{https://github.com/GeWu-Lab/Diagnosing_Relearning_ECCV2024}.