Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization

TL;DR

This paper introduces MBCD, a collaborative distillation framework that improves multi-modal domain generalization by balancing modalities, promoting flatter minima, and enhancing cross-modal knowledge transfer for better out-of-distribution performance.

Contribution

MBCD combines adaptive modality dropout, gradient consistency constraints, and WA-based cross-modal distillation to address modality imbalance in multi-modal domain generalization.

Findings

MBCD outperforms existing methods on MMDG benchmarks.

It achieves higher accuracy across diverse unseen domains.

The framework enhances robustness and generalization in multi-modal tasks.

Abstract

Weight Averaging (WA) has emerged as a powerful technique for enhancing generalization by promoting convergence to a flat loss landscape, which correlates with stronger out-of-distribution performance. However, applying WA directly to multi-modal domain generalization (MMDG) is challenging: differences in optimization speed across modalities lead WA to overfit to faster-converging ones in early stages, suppressing the contribution of slower yet complementary modalities, thereby hindering effective modality fusion and skewing the loss surface toward sharper, less generalizable minima. To address this issue, we propose MBCD, a unified collaborative distillation framework that retains WA's flatness-inducing advantages while overcoming its shortcomings in multi-modal contexts. MBCD begins with adaptive modality dropout in the student model to curb early-stage bias toward dominant modalities. A gradient consistency constraint then aligns learning signals between uni-modal branches and the fused representation, encouraging coordinated and smoother optimization. Finally, a WA-based teacher conducts cross-modal distillation by transferring fused knowledge to each uni-modal branch, which strengthens cross-modal interactions and steer convergence toward flatter solutions. Extensive experiments on MMDG benchmarks show that MBCD consistently outperforms existing methods, achieving superior accuracy and robustness across diverse unseen domains.