PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization

TL;DR

This paper introduces PDMP, a strategy that prioritizes the performance-dominant modality in multimodal learning, improving optimization by asymmetric gradient modulation based on unimodal performance rankings.

Contribution

It proposes a novel modality prioritization method that enhances multimodal learning by focusing on the best-performing unimodal modality, independent of model structure.

Findings

PDMP outperforms existing methods on various datasets.

Prioritizing the performance-dominant modality improves multimodal optimization.

The method is flexible and applicable across different multimodal models.

Abstract

Multimodal learning has attracted increasing attention due to its practicality. However, it often suffers from insufficient optimization, where the multimodal model underperforms even compared to its unimodal counterparts. Existing methods attribute this problem to the imbalanced learning between modalities and solve it by gradient modulation. This paper argues that balanced learning is not the optimal setting for multimodal learning. On the contrary, imbalanced learning driven by the performance-dominant modality that has superior unimodal performance can contribute to better multimodal performance. And the under-optimization problem is caused by insufficient learning of the performance-dominant modality. To this end, we propose the Performance-Dominant Modality Prioritization (PDMP) strategy to assist multimodal learning. Specifically, PDMP firstly mines the performance-dominant modality via the performance ranking of the independently trained unimodal model. Then PDMP introduces asymmetric coefficients to modulate the gradients of each modality, enabling the performance-dominant modality to dominate the optimization. Since PDMP only relies on the unimodal performance ranking, it is independent of the structures and fusion methods of the multimodal model and has great potential for practical scenarios. Finally, extensive experiments on various datasets validate the superiority of PDMP.