When is invariance useful in an Out-of-Distribution Generalization problem ?

TL;DR

This paper develops new theoretical conditions for invariant predictors to be optimal in Out-of-Distribution generalization, extending previous theories and proposing an algorithm that performs well on benchmark datasets.

Contribution

It introduces a novel set of theoretical conditions for invariant predictors in OOD problems and derives the Inter Gradient Alignment algorithm from these conditions.

Findings

Theoretical conditions extend previous invariance hypotheses.

Inter Gradient Alignment algorithm is competitive on benchmark datasets.

The theory applies to non-linear cases and generalizes prior necessary conditions.

Abstract

The goal of Out-of-Distribution (OOD) generalization problem is to train a predictor that generalizes on all environments. Popular approaches in this field use the hypothesis that such a predictor shall be an \textit{invariant predictor} that captures the mechanism that remains constant across environments. While these approaches have been experimentally successful in various case studies, there is still much room for the theoretical validation of this hypothesis. This paper presents a new set of theoretical conditions necessary for an invariant predictor to achieve the OOD optimality. Our theory not only applies to non-linear cases, but also generalizes the necessary condition used in \citet{rojas2018invariant}. We also derive Inter Gradient Alignment algorithm from our theory and demonstrate its competitiveness on MNIST-derived benchmark datasets as well as on two of the three \textit{Invariance Unit Tests} proposed by \citet{aubinlinear}.