Covariate Shift in High-Dimensional Random Feature Regression

TL;DR

This paper provides a theoretical analysis of covariate shift in high-dimensional random feature regression, revealing how overparameterized models can be more robust and establishing a linear relationship between in- and out-of-distribution performance.

Contribution

It offers the first precise high-dimensional asymptotic characterization of test error under covariate shift and explains the robustness of overparameterized models.

Findings

Overparameterized models show increased robustness to covariate shift.

A partial order over covariate shifts predicts when shifts harm or help performance.

Linear relationship between in-distribution and out-of-distribution generalization.

Abstract

A significant obstacle in the development of robust machine learning models is covariate shift, a form of distribution shift that occurs when the input distributions of the training and test sets differ while the conditional label distributions remain the same. Despite the prevalence of covariate shift in real-world applications, a theoretical understanding in the context of modern machine learning has remained lacking. In this work, we examine the exact high-dimensional asymptotics of random feature regression under covariate shift and present a precise characterization of the limiting test error, bias, and variance in this setting. Our results motivate a natural partial order over covariate shifts that provides a sufficient condition for determining when the shift will harm (or even help) test performance. We find that overparameterized models exhibit enhanced robustness to covariate shift, providing one of the first theoretical explanations for this intriguing phenomenon. Additionally, our analysis reveals an exact linear relationship between in-distribution and out-of-distribution generalization performance, offering an explanation for this surprising recent empirical observation.