Crafting Distribution Shifts for Validation and Training in Single Source Domain Generalization

TL;DR

This paper introduces a novel validation approach using augmented source images to better estimate model generalization in single-source domain generalization, and proposes methods to increase shape bias for improved robustness.

Contribution

It presents a new validation set construction method with diverse augmentations and a k-fold process to separate training and validation augmentations, along with a novel shape bias enhancement technique.

Findings

Validation correlates well with test performance across datasets.

Proposed validation improves accuracy by up to 15.4%.

State-of-the-art results achieved on benchmarks.

Abstract

Single-source domain generalization attempts to learn a model on a source domain and deploy it to unseen target domains. Limiting access only to source domain data imposes two key challenges - how to train a model that can generalize and how to verify that it does. The standard practice of validation on the training distribution does not accurately reflect the model's generalization ability, while validation on the test distribution is a malpractice to avoid. In this work, we construct an independent validation set by transforming source domain images with a comprehensive list of augmentations, covering a broad spectrum of potential distribution shifts in target domains. We demonstrate a high correlation between validation and test performance for multiple methods and across various datasets. The proposed validation achieves a relative accuracy improvement over the standard validation equal to 15.4% or 1.6% when used for method selection or learning rate tuning, respectively. Furthermore, we introduce a novel family of methods that increase the shape bias through enhanced edge maps. To benefit from the augmentations during training and preserve the independence of the validation set, a k-fold validation process is designed to separate the augmentation types used in training and validation. The method that achieves the best performance on the augmented validation is selected from the proposed family. It achieves state-of-the-art performance on various standard benchmarks. Code at: https://github.com/NikosEfth/crafting-shifts