Transformation-Invariant Learning and Theoretical Guarantees for OOD Generalization

TL;DR

This paper develops a theoretical framework for learning under distribution shifts caused by data transformations, providing guarantees and bounds on sample complexity, and introduces a game-theoretic perspective on the problem.

Contribution

It introduces a novel theoretical approach for distribution shifts via transformation classes, with learning guarantees and a game-theoretic interpretation.

Findings

Established ERM-based learning rules with guarantees.

Derived upper bounds on sample complexity related to VC dimension.

Provided a game-theoretic view of distribution shift challenges.

Abstract

Learning with identical train and test distributions has been extensively investigated both practically and theoretically. Much remains to be understood, however, in statistical learning under distribution shifts. This paper focuses on a distribution shift setting where train and test distributions can be related by classes of (data) transformation maps. We initiate a theoretical study for this framework, investigating learning scenarios where the target class of transformations is either known or unknown. We establish learning rules and algorithmic reductions to Empirical Risk Minimization (ERM), accompanied with learning guarantees. We obtain upper bounds on the sample complexity in terms of the VC dimension of the class composing predictors with transformations, which we show in many cases is not much larger than the VC dimension of the class of predictors. We highlight that the learning rules we derive offer a game-theoretic viewpoint on distribution shift: a learner searching for predictors and an adversary searching for transformation maps to respectively minimize and maximize the worst-case loss.