Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization

TL;DR

This paper investigates conditions under which model performance on different data distributions maintains a monotonic relationship, providing theoretical proofs for linear relations in regularized models under distribution shifts.

Contribution

It establishes theoretical conditions for monotonic performance relationships under distribution shifts, including exact and approximate linear relations for specific models.

Findings

Proves asymptotic linear relation for squared error in ridge-regularized models.

Shows monotonic relation for misclassification error under covariate shift.

Derives approximate linear relation for linear inverse problems.

Abstract

Machine learning systems are often applied to data that is drawn from a different distribution than the training distribution. Recent work has shown that for a variety of classification and signal reconstruction problems, the out-of-distribution performance is strongly linearly correlated with the in-distribution performance. If this relationship or more generally a monotonic one holds, it has important consequences. For example, it allows to optimize performance on one distribution as a proxy for performance on the other. In this paper, we study conditions under which a monotonic relationship between the performances of a model on two distributions is expected. We prove an exact asymptotic linear relation for squared error and a monotonic relation for misclassification error for ridge-regularized general linear models under covariate shift, as well as an approximate linear relation for linear inverse problems.