Evaluating individualized treatment effect predictions: a model-based perspective on discrimination and calibration assessment

TL;DR

This paper develops and evaluates model-based metrics for assessing the discrimination and calibration of individualized treatment effect prediction models, emphasizing their validation in randomized trial data.

Contribution

It introduces novel model-based discrimination and calibration measures grounded in outcome risk prediction, enhancing validation of treatment effect models.

Findings

Model-based metrics outperform traditional measures in bias and accuracy.

Resampling methods have high variance, limiting their reliability.

Independent data validation is recommended for better assessment.

Abstract

In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined as based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of measures of discrimination for benefit (variations of the c-for-benefit), and propose model-based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model-based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R.