MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance

TL;DR

This paper introduces MMPareto, a novel algorithm that mitigates gradient conflicts in multimodal learning by ensuring aligned gradient directions and magnitudes, leading to improved generalization across various modalities and tasks.

Contribution

We propose MMPareto, an algorithm that addresses gradient conflicts in multimodal learning by analyzing loss discrepancies and applying Pareto integration for better gradient alignment.

Findings

MMPareto improves performance across multiple modalities.

The method enhances generalization and scalability.

Experiments confirm superior results over existing approaches.

Abstract

Multimodal learning methods with targeted unimodal learning objectives have exhibited their superior efficacy in alleviating the imbalanced multimodal learning problem. However, in this paper, we identify the previously ignored gradient conflict between multimodal and unimodal learning objectives, potentially misleading the unimodal encoder optimization. To well diminish these conflicts, we observe the discrepancy between multimodal loss and unimodal loss, where both gradient magnitude and covariance of the easier-to-learn multimodal loss are smaller than the unimodal one. With this property, we analyze Pareto integration under our multimodal scenario and propose MMPareto algorithm, which could ensure a final gradient with direction that is common to all learning objectives and enhanced magnitude to improve generalization, providing innocent unimodal assistance. Finally, experiments across multiple types of modalities and frameworks with dense cross-modal interaction indicate our superior and extendable method performance. Our method is also expected to facilitate multi-task cases with a clear discrepancy in task difficulty, demonstrating its ideal scalability. The source code and dataset are available at https://github.com/GeWu-Lab/MMPareto_ICML2024.