Evaluating discriminatory accuracy of models using partial risk-scores in two-phase studies

TL;DR

This paper introduces an efficient method to evaluate the discriminatory accuracy of risk prediction models in two-phase studies using partial risk-scores, enabling validation with incomplete covariate data.

Contribution

The authors develop a non-parametric approach leveraging partial risk-scores for model evaluation, along with an influence function based variance estimation, applicable in complex two-phase study designs.

Findings

Method performs well in finite samples

Outperforms inverse probability weighted estimators in simulations

Successfully applied to lung cancer risk model data

Abstract

Prior to clinical applications, it is critical that risk prediction models are evaluated in independent studies that did not contribute to model development. While prospective cohort studies provide a natural setting for model validation, they often ascertain information on some risk factors (e.g., an expensive biomarker) in a nested sub-study of the original cohort, typically selected based on case-control status, and possibly some additional covariates. In this article, we propose an efficient approach for evaluating discriminatory ability of models using data from all individuals in a cohort study irrespective of whether they were sampled in the nested sub-study for measuring the complete set of risk factors. For evaluation of the Area Under the Curve (AUC) statistics, we estimate probabilities of risk-scores for cases being larger than those in controls conditional on partial risk-scores, the component of the risk-score that could be defined based on partial covariate information. The use of partial risk-scores, as opposed to actual multivariate risk-factor profiles, allows estimation of the underlying conditional expectations using subjects with complete covariate information in a non-parametric fashion even when numerous covariates are involved. We propose an influence function based approach for estimation of the variance of the resulting AUC statistics. We evaluate finite sample performance of the proposed method and compare it to an inverse probability weighted (IPW) estimator through extensive simulation studies. Finally, we illustrate an application of the proposed method for evaluating performance of a lung cancer risk prediction model using data from the Prostate, Lung, Colorectal and Ovarian Cancer (PLCO) trial.