Towards Optimization and Model Selection for Domain Generalization: A Mixup-guided Solution

TL;DR

This paper introduces Mixup-guided optimization and model selection methods to improve domain generalization, addressing the challenges of distribution shifts and lack of target domain knowledge, leading to significant performance gains.

Contribution

It proposes novel Mixup-based techniques for optimizing and selecting models in domain generalization, incorporating theoretical insights and practical validation.

Findings

Significant performance improvements over existing DG algorithms

Achieved new state-of-the-art results in domain generalization

Effective generation of out-of-distribution datasets for optimization

Abstract

The distribution shifts between training and test data typically undermine the performance of models. In recent years, lots of work pays attention to domain generalization (DG) where distribution shifts exist, and target data are unseen. Despite the progress in algorithm design, two foundational factors have long been ignored: 1) the optimization for regularization-based objectives, and 2) the model selection for DG since no knowledge about the target domain can be utilized. In this paper, we propose Mixup guided optimization and selection techniques for DG. For optimization, we utilize an adapted Mixup to generate an out-of-distribution dataset that can guide the preference direction and optimize with Pareto optimization. For model selection, we generate a validation dataset with a closer distance to the target distribution, and thereby it can better represent the target data. We also present some theoretical insights behind our proposals. Comprehensive experiments demonstrate that our model optimization and selection techniques can largely improve the performance of existing domain generalization algorithms and even achieve new state-of-the-art results.