On-the-fly Modulation for Balanced Multimodal Learning

TL;DR

This paper introduces on-the-fly modulation strategies to balance the optimization of different modalities in multimodal learning, improving performance by dynamically adjusting the influence of dominant modalities during training.

Contribution

It proposes novel On-the-fly Prediction and Gradient Modulation techniques to address modality imbalance in joint training, enhancing multimodal model performance.

Findings

Significant performance improvements across various multimodal tasks.

Effective balancing of modalities leads to better representation learning.

Strategies are flexible and applicable to complex multimodal models.

Abstract

Multimodal learning is expected to boost model performance by integrating information from different modalities. However, its potential is not fully exploited because the widely-used joint training strategy, which has a uniform objective for all modalities, leads to imbalanced and under-optimized uni-modal representations. Specifically, we point out that there often exists modality with more discriminative information, e.g., vision of playing football and sound of blowing wind. They could dominate the joint training process, resulting in other modalities being significantly under-optimized. To alleviate this problem, we first analyze the under-optimized phenomenon from both the feed-forward and the back-propagation stages during optimization. Then, On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies are proposed to modulate the optimization of each modality, by monitoring the discriminative discrepancy between modalities during training. Concretely, OPM weakens the influence of the dominant modality by dropping its feature with dynamical probability in the feed-forward stage, while OGM mitigates its gradient in the back-propagation stage. In experiments, our methods demonstrate considerable improvement across a variety of multimodal tasks. These simple yet effective strategies not only enhance performance in vanilla and task-oriented multimodal models, but also in more complex multimodal tasks, showcasing their effectiveness and flexibility. The source code is available at \url{https://github.com/GeWu-Lab/BML_TPAMI2024}.