General and Estimable Learning Bound Unifying Covariate and Concept Shifts

TL;DR

This paper introduces a unified, support-agnostic framework for quantifying and estimating covariate and concept shifts in machine learning, bridging theory and practice with new bounds and algorithms.

Contribution

It proposes new definitions for distribution shifts using entropic optimal transport, along with estimators and the DataShifts algorithm for practical shift quantification and error estimation.

Findings

New support-agnostic shift definitions improve robustness.

The DataShifts algorithm accurately estimates distribution shifts.

The unified error bound applies broadly across loss functions and label spaces.

Abstract

Generalization under distribution shift remains a core challenge in modern machine learning, yet existing learning bound theory is limited to narrow, idealized settings and is non-estimable from samples. In this paper, we bridge the gap between theory and practical applications. We first show that existing bounds become loose and non-estimable because their concept shift definition breaks when the source and target supports mismatch. Leveraging entropic optimal transport, we propose new support-agnostic definitions for covariate and concept shifts, and derive a novel unified error bound that applies to broad loss functions, label spaces, and stochastic labeling. We further develop estimators for these shifts with concentration guarantees, and the DataShifts algorithm, which can quantify distribution shifts and estimate the error bound in most applications -- a rigorous and general tool for analyzing learning error under distribution shift.