Unknown Domain Inconsistency Minimization for Domain Generalization

TL;DR

This paper proposes Unknown Domain Inconsistency Minimization (UDIM), a novel approach that improves domain generalization by aligning loss landscapes between source and perturbed domains, outperforming existing methods.

Contribution

UDIM introduces a combined parameter and data perturbation objective that reduces loss landscape inconsistency, enhancing generalization to unseen domains in domain generalization tasks.

Findings

UDIM outperforms SAM variants on multiple benchmarks.

UDIM shows significant improvements in restrictive domain scenarios.

Theoretically, UDIM establishes an upper bound for the DG objective.

Abstract

The objective of domain generalization (DG) is to enhance the transferability of the model learned from a source domain to unobserved domains. To prevent overfitting to a specific domain, Sharpness-Aware Minimization (SAM) reduces source domain's loss sharpness. Although SAM variants have delivered significant improvements in DG, we highlight that there's still potential for improvement in generalizing to unknown domains through the exploration on data space. This paper introduces an objective rooted in both parameter and data perturbed regions for domain generalization, coined Unknown Domain Inconsistency Minimization (UDIM). UDIM reduces the loss landscape inconsistency between source domain and unknown domains. As unknown domains are inaccessible, these domains are empirically crafted by perturbing instances from the source domain dataset. In particular, by aligning the loss landscape acquired in the source domain to the loss landscape of perturbed domains, we expect to achieve generalization grounded on these flat minima for the unknown domains. Theoretically, we validate that merging SAM optimization with the UDIM objective establishes an upper bound for the true objective of the DG task. In an empirical aspect, UDIM consistently outperforms SAM variants across multiple DG benchmark datasets. Notably, UDIM shows statistically significant improvements in scenarios with more restrictive domain information, underscoring UDIM's generalization capability in unseen domains. Our code is available at \url{https://github.com/SJShin-AI/UDIM}.