Evaluating Health Risk Models

TL;DR

This paper discusses methods to evaluate and compare health risk models, emphasizing the importance of population risk distribution, and introduces a new approach to identify individuals who benefit most from additional covariates.

Contribution

It introduces a new method to identify individuals who gain the most from expanded risk models and discusses the limitations of traditional performance measures.

Findings

Model performance is limited by true risk distribution in the population.

Brier Score and IDI outperform concordance statistic in measuring precision gains.

Individual gains from additional covariates can be significant for some, guiding targeted data collection.

Abstract

Interest in targeted disease prevention has stimulated development of models that assign risks to individuals, using their personal covariates. We need to evaluate these models, and to quantify the gains achieved by expanding a model with additional covariates. We describe several performance measures for risk models, and show how they are related. Application of the measures to risk models for hypothetical populations and for postmenopausal US women illustrate several points. First, model performance is constrained by the distribution of true risks in the population. This complicates the comparison of two models if they are applied to populations with different covariate distributions. Second, the Brier Score and the Integrated Discrimination Improvement (IDI) are more useful than the concordance statistic for quantifying precision gains obtained from model expansion. Finally, these precision gains are apt to be small, although they may be large for some individuals. We propose a new way to identify these individuals, and show how to quantify how much they gain by measuring the additional covariates. Those with largest gains could be targeted for cost-efficient covariate assessment.