An Estimand-Focused Approach for AUC Estimation, Generalization, and Comparison: From Non-representative Samples to Target Population

TL;DR

This paper introduces a new framework for accurately estimating and comparing the AUC of biomarkers across different populations, addressing biases caused by covariate shifts and non-representative samples.

Contribution

It develops an estimand-focused methodology that extends calibration weighting to the AUC estimation, incorporating double robustness and asymptotic properties for better generalization.

Findings

Proposed estimators perform well under various covariate shifts.

Demonstrated utility in lung cancer trial data analysis.

Provided a toolkit for fair biomarker AUC comparison across studies.

Abstract

The area under the ROC curve (AUC) is the standard measure of a biomarker's discriminatory accuracy; however, naive AUC estimates can be misleading when validation cohorts differ from the intended target population. Such covariate shifts commonly arise under biased or non-random sampling, distorting AUC estimations and thus impeding both generalization and cross-study comparison of AUC. We develop an estimand-focused framework for valid AUC estimation and benchmarking under covariate shift. Leveraging balancing ideas from causal inference, we extend calibration weighting to the U-statistic framework for AUC estimation and introduce a family of estimators that accommodate both summary-level and patient-level information; in certain specifications, some of these estimators attain double robustness. Furthermore, we establish asymptotic properties and study their performances across a spectrum of covariate shift severities and calibration choices in comprehensive simulations. Finally, we demonstrate practical utility in the POWER trials by evaluating how baseline stair-climb power (SCP) predicts 6-month survival among advanced non-small-cell lung cancer (NSCLC) patients. Together, the results provide a principled toolkit for anchoring biomarker AUCs to clinically relevant target populations and for comparing them fairly across studies despite distributional differences.