Failure Modes of Domain Generalization Algorithms

TL;DR

This paper introduces a new evaluation framework for domain generalization algorithms that decomposes errors to better understand their failure modes, revealing diverse challenges and suggesting new directions for improving generalization.

Contribution

It proposes an error decomposition framework for domain generalization, extending it to analyze failures in domain-invariant representation learning, and highlights the variability of error sources across methods and datasets.

Findings

Most algorithms fail to achieve true domain invariance on training data.

Domain invariance often degrades representation quality on unseen domains.

Error sources vary significantly across datasets and training conditions.

Abstract

Domain generalization algorithms use training data from multiple domains to learn models that generalize well to unseen domains. While recently proposed benchmarks demonstrate that most of the existing algorithms do not outperform simple baselines, the established evaluation methods fail to expose the impact of various factors that contribute to the poor performance. In this paper we propose an evaluation framework for domain generalization algorithms that allows decomposition of the error into components capturing distinct aspects of generalization. Inspired by the prevalence of algorithms based on the idea of domain-invariant representation learning, we extend the evaluation framework to capture various types of failures in achieving invariance. We show that the largest contributor to the generalization error varies across methods, datasets, regularization strengths and even training lengths. We observe two problems associated with the strategy of learning domain-invariant representations. On Colored MNIST, most domain generalization algorithms fail because they reach domain-invariance only on the training domains. On Camelyon-17, domain-invariance degrades the quality of representations on unseen domains. We hypothesize that focusing instead on tuning the classifier on top of a rich representation can be a promising direction.