Calibration plots for multistate risk predictions models: an overview and simulation comparing novel approaches

TL;DR

This paper reviews and compares methods for assessing calibration of multistate risk prediction models, introducing simulation studies and a real-world application to healthcare data.

Contribution

It presents novel techniques for calibration assessment of multistate models, including the MLR-IPCW approach and a comprehensive simulation study.

Findings

Pseudo-value, BLR-IPCW, and MLR-IPCW provide unbiased calibration estimates under non-informative censoring.

Methods remain unbiased with informative censoring unless the mechanism is strongly informative.

The MLR-IPCW approach offers additional insights through calibration scatter plots.

Abstract

Introduction. There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of non-informative and informative censoring through a simulation. Methods. We studied pseudo-values based on the Aalen-Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR-IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW). The MLR-IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records. Results. The pseudo-value, BLR-IPCW and MLR-IPCW approaches give unbiased estimates of the calibration curves under non-informative censoring. These methods remained unbiased in the presence of informative censoring, unless the mechanism was strongly informative, with bias concentrated in the areas of predicted transition probabilities of low density. Conclusions. We recommend implementing either the pseudo-value or BLR-IPCW approaches to produce a calibration curve, combined with the MLR-IPCW approach to produce a calibration scatter plot, which provides additional information over either of the other methods.