Adapting to Shifting Correlations with Unlabeled Data Calibration

TL;DR

This paper introduces GPA, a flexible method that adjusts model predictions to account for shifting correlations between targets and confounders using unlabeled data, improving accuracy across different sites.

Contribution

GPA is a novel approach that leverages unlabeled data to adapt to correlation shifts, overcoming limitations of previous methods that ignore unstable features or make unrealistic assumptions.

Findings

GPA outperforms baseline methods on real and synthetic datasets.

It effectively exploits unstable features by adjusting for correlation shifts.

GPA scales well to multiple confounding features.

Abstract

Distribution shifts between sites can seriously degrade model performance since models are prone to exploiting unstable correlations. Thus, many methods try to find features that are stable across sites and discard unstable features. However, unstable features might have complementary information that, if used appropriately, could increase accuracy. More recent methods try to adapt to unstable features at the new sites to achieve higher accuracy. However, they make unrealistic assumptions or fail to scale to multiple confounding features. We propose Generalized Prevalence Adjustment (GPA for short), a flexible method that adjusts model predictions to the shifting correlations between prediction target and confounders to safely exploit unstable features. GPA can infer the interaction between target and confounders in new sites using unlabeled samples from those sites. We evaluate GPA on several real and synthetic datasets, and show that it outperforms competitive baselines.