Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift

TL;DR

This paper critically examines the theoretical understanding of distribution shift in machine learning, proposing alternative assumptions to discrepancy-based bounds and providing rigorous guarantees for classification performance under various data availability scenarios.

Contribution

It introduces an IRM-like assumption to analyze distribution shift, offering new theoretical guarantees and conditions for when source, unlabeled, or labeled target data suffice for accurate classification.

Findings

Invariant-Risk-Minimization-like assumption connects source and target distributions.

Conditions identified where source data alone suffices for target classification.

When source data is insufficient, unlabeled or labeled target data are necessary.

Abstract

Many machine learning models appear to deploy effortlessly under distribution shift, and perform well on a target distribution that is considerably different from the training distribution. Yet, learning theory of distribution shift bounds performance on the target distribution as a function of the discrepancy between the source and target, rarely guaranteeing high target accuracy. Motivated by this gap, this work takes a closer look at the theory of distribution shift for a classifier from a source to a target distribution. Instead of relying on the discrepancy, we adopt an Invariant-Risk-Minimization (IRM)-like assumption connecting the distributions, and characterize conditions under which data from a source distribution is sufficient for accurate classification of the target. When these conditions are not met, we show when only unlabeled data from the target is sufficient, and when labeled target data is needed. In all cases, we provide rigorous theoretical guarantees in the large sample regime.