Learning under random distributional shifts

TL;DR

This paper investigates prediction strategies under random distributional shifts, demonstrating that a hybrid approach combining long-term and proxy outcomes offers robustness and improved accuracy in real-world applications like asylum-seeker assignment and childhood education.

Contribution

It introduces a novel framework for modeling random distribution shifts and proposes a hybrid prediction method that outperforms existing strategies in complex, real-world scenarios.

Findings

Hybrid approach reduces mean-squared error significantly.

Method is robust to the strength of distribution shifts.

Applied successfully to asylum-seeker and childhood education datasets.

Abstract

Many existing approaches for generating predictions in settings with distribution shift model distribution shifts as adversarial or low-rank in suitable representations. In various real-world settings, however, we might expect shifts to arise through the superposition of many small and random changes in the population and environment. Thus, we consider a class of random distribution shift models that capture arbitrary changes in the underlying covariate space, and dense, random shocks to the relationship between the covariates and the outcomes. In this setting, we characterize the benefits and drawbacks of several alternative prediction strategies: the standard approach that directly predicts the long-term outcome of interest, the proxy approach that directly predicts a shorter-term proxy outcome, and a hybrid approach that utilizes both the long-term policy outcome and (shorter-term) proxy outcome(s). We show that the hybrid approach is robust to the strength of the distribution shift and the proxy relationship. We apply this method to datasets in two high-impact domains: asylum-seeker assignment and early childhood education. In both settings, we find that the proposed approach results in substantially lower mean-squared error than current approaches.