ReconBoost: Boosting Can Achieve Modality Reconcilement
Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, Qingming Huang

TL;DR
ReconBoost introduces a modality-alternating learning approach for multi-modal models, addressing modality competition by dynamically updating fixed modalities and employing regularization techniques inspired by gradient boosting, leading to improved multi-modal learning performance.
Contribution
The paper proposes ReconBoost, a novel modality-reconcilement method that alternates learning between modalities and incorporates regularization, inspired by gradient boosting, to enhance multi-modal learning.
Findings
Outperforms existing multi-modal methods on six benchmarks.
Effectively mitigates modality competition and overfitting.
Demonstrates the benefit of alternating modality updates with regularization.
Abstract
This paper explores a novel multi-modal alternating learning paradigm pursuing a reconciliation between the exploitation of uni-modal features and the exploration of cross-modal interactions. This is motivated by the fact that current paradigms of multi-modal learning tend to explore multi-modal features simultaneously. The resulting gradient prohibits further exploitation of the features in the weak modality, leading to modality competition, where the dominant modality overpowers the learning process. To address this issue, we study the modality-alternating learning paradigm to achieve reconcilement. Specifically, we propose a new method called ReconBoost to update a fixed modality each time. Herein, the learning objective is dynamically adjusted with a reconcilement regularization against competition with the historical models. By choosing a KL-based reconcilement, we show that the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling
