Estimating and evaluating counterfactual prediction models

TL;DR

This paper discusses methods for estimating and evaluating counterfactual prediction models, addressing challenges in assessing models where outcomes are not observed under all treatment scenarios, and provides practical tools for model performance measurement.

Contribution

It introduces identification and estimation techniques for counterfactual prediction models, including performance measures that remain valid even if models are misspecified.

Findings

Provides methods for valid performance evaluation under counterfactual settings

Demonstrates estimation techniques through simulation studies

Applies methods to develop a cardiovascular risk prediction model

Abstract

Counterfactual prediction methods are required when a model will be deployed in a setting where treatment policies differ from the setting where the model was developed, or when a model provides predictions under hypothetical interventions to support decision-making. However, estimating and evaluating counterfactual prediction models is challenging because, unlike traditional (factual) prediction, one does not observe the potential outcomes for all individuals under all treatment strategies of interest. Here, we discuss how to estimate a counterfactual prediction model, how to assess the model's performance, and how to perform model and tuning parameter selection. We provide identification and estimation results for counterfactual prediction models and for multiple measures of counterfactual model performance, including loss-based measures, the area under the receiver operating characteristic curve, and the calibration curve. Importantly, our results allow valid estimates of model performance under counterfactual intervention even if the candidate prediction model is misspecified, permitting a wider array of use cases. We illustrate these methods using simulation and apply them to the task of developing a statin-naive risk prediction model for cardiovascular disease.