Domain-Specific Risk Minimization for Out-of-Distribution Generalization

TL;DR

This paper introduces Domain-specific Risk Minimization (DRM), a novel approach for out-of-distribution generalization that models source domains separately and adapts online to unseen target samples, effectively reducing the adaptivity gap.

Contribution

The paper proposes DRM, a new method that explicitly considers the adaptivity gap in domain generalization, combining ensemble-based gap estimation and online model adaptation for improved robustness.

Findings

DRM outperforms baselines across various distribution shifts

Achieves comparable or better accuracy on source domains

Remains simple, efficient, and complementary to invariant learning

Abstract

Recent domain generalization (DG) approaches typically use the hypothesis learned on source domains for inference on the unseen target domain. However, such a hypothesis can be arbitrarily far from the optimal one for the target domain, induced by a gap termed ``adaptivity gap''. Without exploiting the domain information from the unseen test samples, adaptivity gap estimation and minimization are intractable, which hinders us to robustify a model to any unknown distribution. In this paper, we first establish a generalization bound that explicitly considers the adaptivity gap. Our bound motivates two strategies to reduce the gap: the first one is ensembling multiple classifiers to enrich the hypothesis space, then we propose effective gap estimation methods for guiding the selection of a better hypothesis for the target. The other method is minimizing the gap directly by adapting model parameters using online target samples. We thus propose \textbf{Domain-specific Risk Minimization (DRM)}. During training, DRM models the distributions of different source domains separately; for inference, DRM performs online model steering using the source hypothesis for each arriving target sample. Extensive experiments demonstrate the effectiveness of the proposed DRM for domain generalization with the following advantages: 1) it significantly outperforms competitive baselines on different distributional shift settings; 2) it achieves either comparable or superior accuracies on all source domains compared to vanilla empirical risk minimization; 3) it remains simple and efficient during training, and 4) it is complementary to invariant learning approaches.