Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representations

TL;DR

This paper introduces a unified representation approach for multi-modal domain generalization, effectively aligning different modalities to improve model robustness across unseen target domains in multi-modal tasks.

Contribution

The paper proposes a novel unified representation framework and a supervised disentanglement method to enhance multi-modal domain generalization, addressing limitations of existing single-modal DG techniques.

Findings

Outperforms existing methods on benchmark datasets like EPIC-Kitchens.

Effectively aligns multi-modal data within a unified space for better generalization.

Demonstrates robustness in unseen target domains across multiple modalities.

Abstract

Domain Generalization (DG) aims to enhance model robustness in unseen or distributionally shifted target domains through training exclusively on source domains. Although existing DG techniques, such as data manipulation, learning strategies, and representation learning, have shown significant progress, they predominantly address single-modal data. With the emergence of numerous multi-modal datasets and increasing demand for multi-modal tasks, a key challenge in Multi-modal Domain Generalization (MMDG) has emerged: enabling models trained on multi-modal sources to generalize to unseen target distributions within the same modality set. Due to the inherent differences between modalities, directly transferring methods from single-modal DG to MMDG typically yields sub-optimal results. These methods often exhibit randomness during generalization due to the invisibility of target domains and fail to consider inter-modal consistency. Applying these methods independently to each modality in the MMDG setting before combining them can lead to divergent generalization directions across different modalities, resulting in degraded generalization capabilities. To address these challenges, we propose a novel approach that leverages Unified Representations to map different paired modalities together, effectively adapting DG methods to MMDG by enabling synchronized multi-modal improvements within the unified space. Additionally, we introduce a supervised disentanglement framework that separates modal-general and modal-specific information, further enhancing the alignment of unified representations. Extensive experiments on benchmark datasets, including EPIC-Kitchens and Human-Animal-Cartoon, demonstrate the effectiveness and superiority of our method in enhancing multi-modal domain generalization.