Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion

TL;DR

This paper introduces a novel multimodal learning method that dynamically balances the classification abilities of different modalities using boosting, effectively addressing modality imbalance and improving performance over existing methods.

Contribution

It proposes a sustained boosting algorithm with adaptive classifier assignment to balance modality classification abilities, backed by theoretical convergence analysis.

Findings

Outperforms state-of-the-art multimodal learning baselines

Effectively mitigates modality imbalance in classification tasks

Demonstrates improved accuracy on widely used datasets

Abstract

Multimodal learning (MML) is significantly constrained by modality imbalance, leading to suboptimal performance in practice. While existing approaches primarily focus on balancing the learning of different modalities to address this issue, they fundamentally overlook the inherent disproportion in model classification ability, which serves as the primary cause of this phenomenon. In this paper, we propose a novel multimodal learning approach to dynamically balance the classification ability of weak and strong modalities by incorporating the principle of boosting. Concretely, we first propose a sustained boosting algorithm in multimodal learning by simultaneously optimizing the classification and residual errors. Subsequently, we introduce an adaptive classifier assignment strategy to dynamically facilitate the classification performance of the weak modality. Furthermore, we theoretically analyze the convergence property of the cross-modal gap function, ensuring the effectiveness of the proposed boosting scheme. To this end, the classification ability of strong and weak modalities is expected to be balanced, thereby mitigating the imbalance issue. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines. The source code is available at https://github.com/njustkmg/NeurIPS25-AUG.