Robust Estimation of Loss-Based Measures of Model Performance under Covariate Shift

TL;DR

This paper introduces robust, data-adaptive estimators for assessing model performance under covariate shift, especially when only covariate data are available from the target population, improving upon traditional weighting methods.

Contribution

It develops new estimators for target population risk that are more robust and compatible with machine learning-based nuisance parameter estimation, extending to complex survey data.

Findings

New estimators demonstrate improved robustness in simulations.

Application to NHANES data shows practical utility.

Methods effectively handle covariate shift with complex survey design.

Abstract

We present methods for estimating loss-based measures of the performance of a prediction model in a target population that differs from the source population in which the model was developed, in settings where outcome and covariate data are available from the source population but only covariate data are available on a simple random sample from the target population. Prior work adjusting for differences between the two populations has used various weighting estimators with inverse odds or density ratio weights. Here, we develop more robust estimators for the target population risk (expected loss) that can be used with data-adaptive (e.g., machine learning-based) estimation of nuisance parameters. We examine the large-sample properties of the estimators and evaluate finite sample performance in simulations. Last, we apply the methods to data from lung cancer screening using nationally representative data from the National Health and Nutrition Examination Survey (NHANES) and extend our methods to account for the complex survey design of the NHANES.