Balancing Multimodal Domain Generalization via Gradient Modulation and Projection

TL;DR

This paper introduces Gradient Modulation Projection (GMP), a novel method to balance gradient contributions across modalities in Multimodal Domain Generalization, improving model generalization to unseen domains by dynamically adjusting gradients based on confidence levels.

Contribution

The paper proposes GMP, a unified gradient balancing strategy that decouples and modulates gradients for classification and domain-invariance, enhancing cross-domain generalization in MMDG.

Findings

GMP achieves state-of-the-art results on multiple benchmarks.

GMP effectively balances modality contributions during training.

GMP is compatible with various MMDG methods.

Abstract

Multimodal Domain Generalization (MMDG) leverages the complementary strengths of multiple modalities to enhance model generalization on unseen domains. A central challenge in multimodal learning is optimization imbalance, where modalities converge at different speeds during training. This imbalance leads to unequal gradient contributions, allowing some modalities to dominate the learning process while others lag behind. Existing balancing strategies typically regulate each modality's gradient contribution based on its classification performance on the source domain to alleviate this issue. However, relying solely on source-domain accuracy neglects a key insight in MMDG: modalities that excel on the source domain may generalize poorly to unseen domains, limiting cross-domain gains. To overcome this limitation, we propose Gradient Modulation Projection (GMP), a unified strategy that promotes balanced optimization in MMDG. GMP first decouples gradients associated with classification and domain-invariance objectives. It then modulates each modality's gradient based on semantic and domain confidence. Moreover, GMP dynamically adjusts gradient projections by tracking the relative strength of each task, mitigating conflicts between classification and domain-invariant learning within modality-specific encoders. Extensive experiments demonstrate that GMP achieves state-of-the-art performance and integrates flexibly with diverse MMDG methods, significantly improving generalization across multiple benchmarks.