Model-Based Domain Generalization

TL;DR

This paper introduces a theoretically grounded approach called Model-Based Domain Generalization to improve out-of-distribution generalization in deep learning, achieving significant performance gains on multiple benchmarks.

Contribution

It formulates domain generalization as an infinite-dimensional constrained learning problem and develops a novel algorithm with convergence guarantees based on nonconvex duality theory.

Findings

Up to 30 percentage points improvement over baselines

Theoretical formulation of domain generalization as a constrained problem

Effective algorithm with convergence guarantees

Abstract

Despite remarkable success in a variety of applications, it is well-known that deep learning can fail catastrophically when presented with out-of-distribution data. Toward addressing this challenge, we consider the domain generalization problem, wherein predictors are trained using data drawn from a family of related training domains and then evaluated on a distinct and unseen test domain. We show that under a natural model of data generation and a concomitant invariance condition, the domain generalization problem is equivalent to an infinite-dimensional constrained statistical learning problem; this problem forms the basis of our approach, which we call Model-Based Domain Generalization. Due to the inherent challenges in solving constrained optimization problems in deep learning, we exploit nonconvex duality theory to develop unconstrained relaxations of this statistical problem with tight bounds on the duality gap. Based on this theoretical motivation, we propose a novel domain generalization algorithm with convergence guarantees. In our experiments, we report improvements of up to 30 percentage points over state-of-the-art domain generalization baselines on several benchmarks including ColoredMNIST, Camelyon17-WILDS, FMoW-WILDS, and PACS.