Prediction with Differential Covariate Classification: Illustrated by Covariate Classification in Medical Risk Assessment

TL;DR

This paper introduces a formal framework for prediction when covariate classification differs between research and application settings, highlighting the challenges and limitations in clinical and policy contexts.

Contribution

It develops a formal framework for differential covariate classification (DCC) and analyzes partial identification of predictions under this framework, emphasizing the impact of assumptions.

Findings

Bounds on P(y|x) can be very wide.

Additional information is often needed to narrow these bounds.

Differential classification significantly affects predictive accuracy in practice.

Abstract

A common practice in evidence-based decision-making uses estimates of conditional probabilities P(y|x) obtained from research studies to predict outcomes y on the basis of observed covariates x. Given this information, decisions are then based on the predicted outcomes. Researchers commonly assume that the predictors used in the generation of the evidence are the same as those used in applying the evidence: i.e., the meaning of x in the two circumstances is the same. This may not be the case in real-world settings. Across a wide range of settings, ranging from clinical practice to education policy, demographic attributes (e.g., age, race, ethnicity) are often classified differently in research studies than in decision settings. This paper studies identification in such settings. We propose a formal framework for prediction with what we term differential covariate classification (DCC). Using this framework, we analyze partial identification of probabilistic predictions and assess how various assumptions influence the identification regions. We apply the findings to a range of settings, focusing mainly on differential classification of individuals' race and ethnicity in clinical medicine. We find that bounds on P(y|x) can be wide, and the information needed to narrow them available only in special cases. These findings highlight an important problem in using evidence in decision making, a problem that has not yet been fully appreciated in debates on classification in public policy and medicine.