A-calibration: assessment of prediction models for survival data under censoring

TL;DR

This paper introduces A-calibration, a new method to evaluate how well survival prediction models are calibrated, especially when data is censored.

Contribution

A-calibration is proposed as a more powerful and less conservative alternative to D-calibration for assessing model calibration in survival analysis.

Findings

A-calibration showed similar or better power than D-calibration in detecting miscalibration across different censoring scenarios.

D-calibration was found to be more sensitive to censoring, leading to potential loss of power.

Theoretical and empirical evidence supports A-calibration as a superior calibration assessment method.

Abstract

Evaluating the performance of predictive models for survival is essential before they can be trusted for real-world applications and decision making. While good measures such as the C-index are available for model discrimination, the toolbox for model calibration is much more limited in the time-to-event setting.

The method of D-calibration was therefore an important contribution that yields a single numeric value for calibration across the available follow-up time. D-calibration consists of performing a Pearson’s goodness-of-fit test on transformed survival times. Censored survival times are handled using an imputation approach which however tends to yield a conservative test and loss of power.

In this paper, we introduce A-calibration based on Akritas’s goodness-of-fit test which is designed specifically for censored time-to-event data. Through theoretical arguments, simulations, and a case study, we compare A- and D-calibration as measures of calibration. In the simulation study, the power of each test to reject a false null-hypothesis was assessed for varying censoring mechanisms (memoryless, uniform and zero censoring), censoring rates, and parameter values of the predictive model considered.

The simulation study demonstrated that A-calibration had similar or superior power to D-calibration in all considered cases, and that D-calibration, unlike A-calibration, was particularly sensitive to censoring.

Advantages of A-calibration compared to D-calibration have been demonstrated through theoretical considerations, a simulation study, and a case study, while no disadvantages relative to D-calibration were identified.

The online version contains supplementary material available at 10.1186/s12874-025-02671-6.

Figures

MC estimates of the power of A- and D-calibration with a censoring percentage of $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q=20$$\end{document}$ across varying $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{docume

MC estimates of the power of A- and D-calibration with a censoring percentage of $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q=20$$\end{document}$ across varying $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{docume

MC estimates of the power of A- and D-calibration with a misspecification of $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda = 0.85$$\end{document}$ on the shape parameter of the model across varying censoring percentages *q*. Estimates are based on 20,000 MC simulations for different validation data sizes and censoring schemes

MC estimates of the power of A- and D-calibration with no censoring across varying $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document}$ -values controlling the misspecification of the shape of the model, with $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlengt

Full text

Background

Evaluating predictive performance of survival models is essential to validate the models for use in real-world settings. Predictive performance is usually measured in terms of discrimination and calibration. Discrimination is the degree to which the model is able to distinguish between high and low-risk cases, whereas calibration measures the accuracy of outcome predictions in comparison with the true outcome probabilities. The integrated Brier score (IBS) [1] and the Concordance index (C-index) [2] are commonly utilized performance measures. The Murphy decomposition of the Brier score shows that this score measures a combination of calibration, discrimination, and the probabilistic uncertainty of the outcome [3]. In contrast, the C-index is specifically a measure of discrimination. The IBS and the C-index can be estimated consistently using inverse probability of censoring weighting (IPCW) schemes [4, 5].

However, since the IBS and the C-index do not measure calibration specifically, calibration plots are often used for this purpose. These are visual tools rather than numerical quantities, making e.g. comparisons and optimization difficult. From calibration plots, certain summary statistics are sometimes extracted, including calibration intercept and slope. However, these quantities have been criticized with regards to interpretability [6] and have been characterized as weak measures of calibration, as they only measure average effects [7]. Therefore, other calibration measures have been developed based on the calibration curves, including the estimated calibration index [8]. Since calibration curves are not designed for time-to-event data, these calibration measures require a fixed time-point unlike the IBS and C-index which measure performance across the available follow-up.

Calibration has been described as the Achilles heel of predictive analytics, due to a combination of its great importance and the general lack of attention to calibration [9]. Recently, in the context of censored survival data, Haider et al. [10] introduced D-calibration that is not based on calibration curves and measures the calibration across the available follow-up. The main idea of D-calibration is to utilize the known distribution of event times transformed by the survival function, an idea previously used in the Cox-Snell residuals [11]. However, D-calibration uses an imputation approach to adjust for censoring that relies heavily on the null hypothesis considered. This can lead to considerable loss of power in the presence of censoring. In this paper we introduce a new method where we combine the transformation idea of D-calibration with a goodness of fit (GOF) test introduced by Akritas [12]. The new method which we coin A-calibration does not require data imputation under the null hypothesis. This can lead to greater power because dilution of data information due to imputation is avoided. The advantages of A-calibration are demonstrated by theoretical considerations, simulations, and a case study.

Methods

Notation

In this paper we consider independent and identically distributed (i.i.d.) randomly right-censored survival outcomes and predictors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(T_i,\Delta _i, Z_i)_{i=1}^n$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n\ge 1$$\end{document} subjects. Here \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_i$$\end{document} is a vector of risk predictors for the ith subject, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_i$$\end{document} is the survival time with conditional survival function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S(\cdot \ | \ Z_i)$$\end{document} given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_i$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_i$$\end{document} is the censoring time, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_i = \min (X_i,C_i)$$\end{document} is the censored survival time, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta_i=1\lbrack X_i<C_i\rbrack$$\end{document} is the censoring indicator (0 if censoring occurs and 1 otherwise). We also consider predictive models \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S}$$\end{document} of the survival functions, as well as observed survival outcomes and predictors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf {t}}=(t_i)_{i=1}^n$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{\delta }=(\delta _{i})_{i=1}^n$$\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf {z}} = ({\textbf {z}}_i)_{i=1}^n$$\end{document} .

Basic idea and assumptions

Throughout the paper, A- and D-calibration are presented and compared. The two approaches are very similar, as they both transform the observed survival data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({\textbf {t}}, \varvec{\delta }, {\textbf {z}})$$\end{document} into a sample of known distribution under a null hypothesis, and then test whether the sample adheres to this distribution. For both methods, the null hypothesis more precisely means that the survival times are hypothesized to arise from a specific predictive model \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S}$$\end{document} . For clarity, we emphasize that the null hypothesis is not a composite hypothesis that the true survival function belongs to some specific model class.

Assuming a continuous survival function, we employ the probability integral transform (PIT), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S(X_i \ | \ Z_i)$$\end{document} for the survival times \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_i$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1,\ldots ,n$$\end{document} . If S is strictly decreasing and thus possesses an inverse, it is trivial to show that the PIT transformed survival times follow the standard uniform distribution on [0, 1]. However, this also holds without strict monotonicity, which follows by utilizing the generalized inverse \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{\text {gen}}^{-1}(y \ | \ z) = \inf \{t \ | \ S(t \ | \ z) \le y \}$$\end{document} . Thus, throughout the paper, a continuous but not necessarily strictly decreasing survival function is assumed.

Consider the uncensored PIT residuals \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S}(x_i \ | \ {\textbf {z}}_i)$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1,\ldots ,n,$$\end{document} which approximately form a standard uniform sample if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S}$$\end{document} is well calibrated. The central idea behind both A- and D-calibration is to test whether the PIT residuals adhere to the standard uniform distribution using goodness-of-fit (GOF) tests of the form

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sum \limits _{k=1}^K \frac{(N_k-E_k)^2}{E_k}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} constitute observed and expected counts of PIT residuals (under the null hypothesis) in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_k$$\end{document} , respectively, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{I_k\}_{k=1,\dots ,K}$$\end{document} represents a partition of [0, 1] (or a subset thereof) into \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K \ge 1$$\end{document} intervals.

However, as the survival data is right-censored, the observed PIT residuals \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S}(t_i \ | \ {\textbf {z}}_i)$$\end{document} constitute a left-censored standard uniform sample under the null hypothesis.

The difference between A- and D-calibration lies in how the censoring of the PIT residuals is handled, which in turn influences how the terms \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} in (1) are defined as elaborated in the following sections.

Regarding the dependence between survival times and censoring times it is sufficient for both methods to assume conditional independence, that is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X \perp \!\!\!\!\perp C\ |\ Z$$\end{document} where X, C, and Z is generic notation for a survival time, a censoring time and a predictor. This is because dependence between X and C through Z vanishes after the transformation to PIT residuals, as shown in Proposition 1.

Proposition 1

Let X be a survival time with survival function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S( \cdot \ | \ Z)$$\end{document} depending on a predictor Z and let C be a right-censoring time such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X \perp \!\!\!\!\perp C \ | \ Z$$\end{document} . Then the transformed survival and censoring times are independent, i.e. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S(X \ | \ Z) \perp \!\!\!\!\perp S(C \ | \ Z)$$\end{document} .

Proof

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A,B \subseteq [0,1]$$\end{document} . Then

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P\left( S(X \ | \ Z) \in A, S(C \ | \ Z) \in B\right)= & \mathbb {E}\left[ P\left( S(X \ | \ Z) \in A, S(C \ | \ Z) \in B\right) \ | \ Z \right] \\= & \mathbb {E}\left[ P\left( S(X \ | \ Z) \in A \ | \ Z \right) P\left( S(C \ | \ Z) \in B \ | \ Z \right) \right] \\= & P\left( S(X \ | \ Z) \in A \right) \mathbb {E}\left[ P\left( S(C \ | \ Z) \in B \ | \ Z \right) \right] \\= & P\left( S(X \ | \ Z) \in A \right) P\left( S(C \ | \ Z) \in B \right) \end{aligned}$$\end{document}

where the first and last equalities follow from the Law of Total Expectation, the second equality follows since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X \perp \!\!\!\!\perp C \ | \ Z$$\end{document} , and the third equality follows from the fact that the PIT residuals are independent of the predictors.

In the following two sections, the particularities of each of the methods are discussed.

D-calibration

For D-calibration, in the case of no censoring, the idea would simply be to use a Pearson’s GOF test for standard uniformity, corresponding to a test statistic such as (1) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k=\sum_{i=1}^n1\lbrack\widehat S(x_i\;\vert\;{\textbf{z}}_i)\in I_k\rbrack$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k = |I_k|$$\end{document} , i.e.

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi^2=\sum_{k=1}^K\frac{\left(\left(\sum_{i=1}^n1\lbrack\widehat S(x_i\;\vert\;{\textbf{z}}_i)\in I_k\rbrack\right)-n\vert I_k\vert\right)^2}{n\vert I_k\vert},$$\end{document}

As mentioned above, due to right-censoring, the PIT residuals form a left-censored sample, and hence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum_{i=1}^n1\lbrack\widehat S(x_i\;\vert\;{\textbf{z}}_i)\in I_k\rbrack$$\end{document} is unknown. Assuming

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{S}(X_i \ | \ Z_i ) \ | \ (\Delta _i = 0, T_i = t_i,Z_i=z_i) \sim \text {unif}(0,\hat{S}(t_i \ | \ z_i)),i=1,\dots ,n, \end{aligned}$$\end{document}

which follows under the null hypothesis due to Proposition 1, the contribution to the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k'th$$\end{document} interval of the i’th subject censored at time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_i$$\end{document} is modified to ([10], Appendix B.5)

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{array}{ll} 0 & \text {for } \hat{S}(t_i\ | \ {\textbf {z}}_i) \le \inf (I_k), \\ \frac{\hat{S}(t_i\ | \ {\textbf {z}}_i) - \inf (I_k)}{\hat{S}(t_i \ | \ {\textbf {z}}_i)} & \text {for } \hat{S}(t_i\ | \ {\textbf {z}}_i) \in I_k, \\ \frac{|I_k|}{\hat{S}(t_i\ | \ {\textbf {z}}_i)} & \text {for } \hat{S}(t_i\ | \ {\textbf {z}}_i) \ge \sup (I_k), \end{array}\right. \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\inf (I_k)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sup (I_k)$$\end{document} represents the infimum and supremum of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_k$$\end{document} . That is, for a subject censored at time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_i$$\end{document} , the contribution is evenly distributed among the intervals in accordance with the interval lengths in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[0,\hat{S}(t_i \ | \ z_i))$$\end{document} where the unobserved \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S}(x_i \ | \ z_i)$$\end{document} belongs. Using this approach to handle censoring, the expected proportion of PIT residuals within each interval equals the length of the intervals if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S} = S$$\end{document} , provided the true survival function is strictly decreasing ([10], Theorem 2B). A predictive model is considered D-calibrated if its associated p-value exceeds a specified significance level [10]. Furthermore, the performance of multiple predictive models can be compared on validation data by contrasting the test statistics.

The approach outlined above is in essence an imputation strategy, where the unknown \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k, k =1,\dots ,K$$\end{document} are imputed using the observed PIT residuals and the null hypothesis. This can make the imputed transformed sample appear close to a uniform sample despite possible differences between the true data distribution and the hypothesized one. As the GOF assesses the constrast between the observed \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} and the expected \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} under the null hypothesis, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} itself is then imputed using the very same null hypothesis tested for, D-calibration can become a conservative test (low Type I error) under censoring at the cost of reduced power (greater Type II error). This issue was also identified by Haider et al. [10]. It was in particular highlighted how heavy zero censoring could be problematic. The impact of several different censoring schemes, including zero censoring, is considered in the simulation study (“Simulation study” and “Simulation study” sections).

A-calibration

Given the discussion in the previous section, better handling of censoring is of key interest in connection with GOF testing. Our suggestion is to use Akritas’ Pearson-type GOF test introduced by Akritas [12] and developed specifically for randomly right-censored i.i.d. samples. Consider independent survival and censoring times \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_i$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V_i$$\end{document} with distribution functions F and G, and let H denote the distribution function of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\min (U_i, V_i)$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1,\dots ,n$$\end{document} . The supports of F and G could be the entire positive real line or subsets thereof. The idea is to construct a test statistic on the form of (1) where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k=\sum_{i=1}^n1\lbrack u_i<v_i,u_i\in I_k\rbrack$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} are the observed and expected (under the null hypothesis) number of non-censored events occurring in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_k$$\end{document} , and where the partitioning \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{I_k\}_{k=1,\dots ,K}$$\end{document} is over the support of H, for the sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(u_i,v_i)$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1,\dots ,n$$\end{document} . The expected number of non-censored event-times in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_k$$\end{document} is

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_k = n\int _{I_k}(1-G)dF, \end{aligned}$$\end{document}

which in particular depends on the censoring distribution G. To leave the censoring distribution unspecified and only assume random censoring, Akritas observes that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(1-G)(1-F)=1-H$$\end{document} due to the independence of survival and censoring times, and proposes to estimate the censoring survival function as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 1-\hat{G} =\frac{(1-\hat{H})}{(1-F_0)}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_0$$\end{document} is the distribution function under the null-hypothesis and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{H}$$\end{document} is the empirical distribution function of the censored survival times. Inserting this estimator of the censoring distribution, the test statistic given in (1) follows a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} -distribution with K degrees of freedom under the null hypothesis.

We suggest using Akritas’ Pearsons-type GOF test on one minus the PIT residuals, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1-\hat{S}(t_1 \ | \ {\textbf {z}}_1), \dots , 1-\hat{S}(t_n \ | \ {\textbf {z}}_n)$$\end{document} , which constitutes a right-censored i.i.d. sample. With a slight abuse of wording, this sample will also be referred to as the PIT residuals. More precisely, we let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_i=S(X_i \ | \ Z_i)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V_i=S(C_i \ | \ Z_i)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_i$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V_i$$\end{document} are independent by Proposition 1. Regarding the corresponding distribution functions, F has support [0, 1] while the common support of G and H is of the form [0, a] for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \le a \le 1$$\end{document} . Furthermore, under the null-hypothesis of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S}=S$$\end{document} , the sample follows a censored uniform distribution. Therefore, A-calibration is tested using (1), where

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k=\sum_{i=1}^n1\lbrack\delta_i=1,1-\widehat S(t_i\;\vert\;{\textbf{z}}_i)\in I_k\rbrack\quad\text{and}\quad E_k=n\int_{I_k}\frac{1-\widehat H(t)}{1-t}\mathrm dt,$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{H}$$\end{document} is the empirical distribution function of the PIT residuals.

Depending on the scientific context the upper limit a of the support of H may be known or unknown. If the censoring times \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_i$$\end{document} are unbounded then \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a=1$$\end{document} . If the censoring times are bounded, a may be less than one and we pragmatically choose \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a=\sup \{t\in [0,1]: 1-\hat{H}(t)>0\}$$\end{document} , i.e. we use the empirical support of the transformed censored survival times. This modification is not covered by the theory in [12] but works well in the simulation studies. A-calibration thus proceeds as follows.**Algorithm 1 **A-Calibration

Compared to D-calibration, A-calibration is advantageous because it avoids loss of power due to imputing data under the null hypothesis. Instead, the expected number of cases in each interval \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_k$$\end{document} is adjusted to take censoring into account. Advantages of A-calibration compared to D-calibration are investigated in the subsequent simulation study.

Simulation study

This simulation study compares the power, i.e. the probability of rejecting the null-hypothesis, of D-calibration and A-calibration in various circumstances. Throughout the study, a Weibull model is used as the true survival model, with survival function

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} S(t; \alpha , \sigma ) = \exp \left( -\left( \frac{t}{\sigma } \right) ^\alpha \right) \end{aligned}$$\end{document}

for shape parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha> 0$$\end{document} and scale parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma> 0$$\end{document} . The scale parameter depends on subject predictors and is given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma = \exp (\varvec{\beta }^\top {\textbf {z}}^*)$$\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{\beta }$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf {z}}^*$$\end{document} are the vectors of true scale coefficients and subject-specific predictors.

Given predictors, the true survival model is used to simulate subject specific event-times, which are then right-censored according to one of three censoring schemes. The three censoring schemes considered are memoryless, uniform and zero censoring, where the censoring distribution G is either exponential, uniform with lower bound of zero, or dichotomous with outcomes 0 or \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\infty$$\end{document} . For each scheme, parameters are chosen (rate, upper bound, or probability of zero) to achieve some specified censoring percentage q among the simulated subjects.

Simulated subjects form validation datasets used to evaluate calibration across different predictive models, all of which are misspecified Weibull models. Models are misspecified through missing predictors or misspecified shape or scale parameters. In the latter cases, misspecified are handled using a misspecification parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document} , such that e.g. the scale is misspecified if scale coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda \varvec{\beta }$$\end{document} are utilized for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda \ne 1$$\end{document} . The simulation study is conducted as follows.**Algorithm 2 **Simulation study

Case study

In this case study the Rotterdam dataset from the R survival package [13] is considered. The Rotterdam data first described by Foekens et al. [14] concerns breast cancer patients registered in the Rotterdam tumor bank. The dataset consists of records for 2,982 patients of which 43% died during the available follow-up. Hence the censoring rate is 57%. The dataset contains information on the survival outcome, treatment received, year of surgery (year), age at surgery (age), and numerous clinical characteristics of the cancer and the patients.

In this case study we consider the predictors age, year, menopausal status, tumor size, cancer grade, number of involved lymph nodes, dosis received of progesterone and estrogen receptors, and whether the patients received hormones or chemotherapy. We perform repeated random splitting, dividing patients into a training cohort (70% of the data) and a validation cohort (30% of the data).

From the training cohort three predictive models are fitted: a simple Weibull regression considering only age as a predictor, a Weibull regression based on all predictors and a random survival forest (RSF) [15] based on all predictors. The minimal allowed node size for the RSF was treated as the tuning parameter and was selected over a grid of possible choices to minimize the out-of-bag (OOB) IBS. Finally, the predictive performance of all models was tested on the validation cohort using the C-index, the IBS, calibration plots, calibration intercept and calibration slope, as well as A- and D-calibration. These performance measures were averaged across the repeated training-test splits, and MC estimates along with standard errors were reported for all measures, except for the calibration plots, which were shown for a single training-test split only. The calibration plots, as well as the calibration intercept and slope, were evaluated at the 5-year follow-up mark. With regard to interpretation, C-index values closer to 1 indicate better discrimination; IBS values closer to 0 indicate better overall predictive performance; and calibration intercepts closer to 0 and slopes closer to 1 indicate better calibration. The R packages randomForestSRC [16] and pec [17] were used for training RSFs and evaluating predictive performance, respectively. All implementations, package usage, simulations, and the case study can be found in the supplementary R code.

Results

Simulation study

The simulation study is conducted with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m = 20,000$$\end{document} MC repetitions, true shape parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 1$$\end{document} (corresponding to an exponential model), and the scale coefficient vector is given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{\beta } = \left(\log (2), \log \left(\frac{1}{2}\right), \log (2), \log (1.5)\right)$$\end{document} . Furthermore, the subject specific predictors are independently simulated as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_1,z_2 \sim N(0,1)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_3, z_4\sim \text {Bernoulli(0.5)}$$\end{document} . The study varies the censoring scheme (memoryless, uniform, or zero censoring), censoring percentage q (0%−50%), misspecification parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document} (0.6-1.4) and validation data size n (100 and 1,000). The number of partitioning intervals, K, is set to 10, and evenly sized intervals are used. All MC standard errors for the power estimation were 0.1% or lower. For instance, if the power was estimated at 50% with an MC standard error of 0.1%, the corresponding 95% confidence interval would be [49.8%,50.2%]. As the number of MC repetitions was chosen to ensure that the MC standard errors are negligible compared to the observed differences between A- and D-calibration, we abstain from indicating MC standard errors in the simulation study plots. Figures 1 and S1 show the power of D-calibration and A-calibration for different \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document} values, misspecifying the shape and scale, respectively, with a censoring percentage of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q = 20\%$$\end{document} .Fig. 1MC estimates of the power of A- and D-calibration with a censoring percentage of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q=20$$\end{document} across varying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document} -values controlling the misspecification of shape of the model, with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda = 1$$\end{document} yielding the true model. Estimates are based on 20,000 MC simulations for different validation data sizes and censoring schemes. The horizontal dashed line shows the 5% nominal significance level

Across all conditions, regardless of misspecification, data size n, and censoring scheme, A-calibration always has a similar or superior power compared to D-calibration. For both tests, as the data size increases, the power curves get increasingly steep around the value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda =1$$\end{document} , indicating that with sufficient validation data, either test can reliably reject a false null hypothesis. Although both tests use a nominal significance level of 5%, only the A-calibration power converges for increasing n towards the nominal level of 5% in the case of the true predictive model \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda =1$$\end{document} , while the power of D-calibration converges to a lower, censoring dependent value. For memoryless censoring with a 20% censoring rate, a data size of 1,000, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda =1$$\end{document} , D-calibration achieves a power, and hence an actual significance level, of 1.7% (Fig. 1). Figure 2 shows a similar plot, where the nominal significance level for A-calibration has been adjusted to the actual power of D-calibration (1.7%) under the null hypothesis. Even with this adjustment, A-calibration maintains a superior power throughout the range of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document} values misspecifying the shape.Fig. 2MC estimates of the power of A- and D-calibration with a censoring percentage of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q=20$$\end{document} across varying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document} -values controlling the misspecification of the shape of the model, with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda = 1$$\end{document} yielding the true model. Estimates are based on 20,000 MC simulations with validation data of size 1,000 and using the memoryless censoring scheme. In this simulation the significance level used for A-calibration was reduced to the nominal level of D-calibration at 1.3%

Figures 3, S2, and S3 illustrate that the reduced power of D-calibration is directly related to the censoring rate. Here we use misspecifications on shape, scale, or through missing predictor that correspond to models that are almost always rejected in the case of no censoring for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=1,000$$\end{document} .

The power is reduced for both tests as the censoring percentage is increased but A-calibration is less affected by the increasing censoring percentage and maintains superior power, except for Fig. S2 where the power of D-calibration is slightly above that of A-calibration when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=100$$\end{document} and the censoring percentage is small. For larger data sizes, the censoring percentage has a diminishing impact on power. In particular, with a misspecification of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda =0.8$$\end{document} on the shape parameter and a datasize of 1,000, A-calibration maintains a power near 100% for all all censoring schemes, even when the censoring percentage reaches the extreme level of 50%. Finally, which censoring scheme had the greatest impact on power varied widely, depending on the misspecification of the model.Fig. 3MC estimates of the power of A- and D-calibration with a misspecification of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda = 0.85$$\end{document} on the shape parameter of the model across varying censoring percentages q. Estimates are based on 20,000 MC simulations for different validation data sizes and censoring schemes

Figures 4 and S4 compared A- and D-calibration in the case of no censoring. Notably, the actual Type I error for D-calibration here coincides with the nominal level in both figures. For Fig. 4, where the model’s shape was misspecified, A-calibration still maintained similar or superior power. For Fig. S4 however, where the model’s scale was misspecified, the powers of the methods were similar, with D-calibration occasionally exhibiting greater power.Fig. 4MC estimates of the power of A- and D-calibration with no censoring across varying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document} -values controlling the misspecification of the shape of the model, with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda = 1$$\end{document} yielding the true model. Estimates are based on 20,000 MC simulations with validation data of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n = 100$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n = 1,000$$\end{document}

Case study

Considering the Rotterdam data, the average predictive performance of the three models trained on the training cohorts when predicting on the validation cohorts across 5,000 repeated training-test splits is found in Table 1. All MC standard errors are of order 0.001 or below, and thus, the uncertainty does not affect the conclusions drawn below.

Across all considered measures, the simple Weibull regression consistently performs the worst, while the random survival forest performs the best. However, while the other measures give us direct information regarding the quality of the predictions, A- and D-calibration only answer whether the given model could be the true model. Here, A-calibration would reject the simple Weibull model, whereas D-calibration would accept all of the considered models, even though the other performance measures clearly demonstrate the superior calibration (and model fit in general) of the random survival forest relative to the simple Weibull model.Table 1MC estimates of the predictive performance on the validation cohorts of the two Weibull regression models and the random survival forest trained on the training data through 5,000 repeated training-test splitsMeasureSimple WeibullWeibullRandom survival forestAgeAll predictorsAll predictorsC-index0.5780.6620.672IBS0.1870.1730.168Intercept−0.147−0.1340.002Slope0.5921.1061.086A-calibration0.01940.1140.336D-calibration0.3410.4980.702

Furthermore, performances for a particular split (Table S1), including calibration plots for each of the models (Figs. S5-S7) show the same tendencies. However in this case, D-calibration resulted in a greater p-value for the simple Weibull model compared to the p-value for the full Weibull model. This violates the ranking of the models observed according to the other performance measures.

Discussion

While the authors introducing D-calibration identified a weakness of the method in case of heavy zero censoring ([10], Appendix B.5), our simulation study has shown that other types of censoring can be problematic too, with the method becoming increasingly conservative depending on the censoring rate accompanied by loss of power. This emphasizes the need for improved calibration methodology.

Both A- and D-calibration are based on Pearson-type tests that can be written on the general form of Eq. (1). That is, both test statistics assumes a partitioning of [0, 1] into K intervals and constitute a sum of contrasts on each interval between what is observed \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} and what is expected under the null hypothesis \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} , for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k=1,\dots ,K$$\end{document} . The primary difference between the two approaches is how censoring is incorporated into \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} . For A-calibration, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} simply counts the number of non-censored (1-) PIT residuals belonging to the kth interval. The expected count is estimated using a non-parametric estimate of the censoring distribution based on the empirical distribution function of (1-) PIT residuals and the uniform distribution of the non-censored (1-) PIT residuals under the null hypothesis. For D-calibration, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} is given as n times the length of the kth interval which is the expected value under the null hypothesis ignoring censoring. However, this means that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} must be adjusted for censoring which is done using a kind of imputation leveraging the null hypothesis that is tested for. That is, the null hypothesis only influences \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} for A-calibration, whereas it influences both \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_k$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} for D-calibration, and the degree to which \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_k$$\end{document} is influenced depends on the censoring.

A-calibration offers several advantages. First, A-calibration has similar or superior power compared to D-calibration in all considered cases, and is notably less affected by censoring. The fact that A-calibration generally exhibited superior power in the absence of censoring is not particularly surprising. Although Akritas’ Pearson-type goodness-of-fit (GOF) test was primarily adopted for its ability to handle censoring, Akritas himself noted that in simulations without censoring, his test often outperformed Pearson’s traditional GOF test. This was likely due to the higher degrees of freedom in the test statistic [12]. Secondly, the actual significance level of A-calibration coincides with the nominal level as opposed to D-calibration where the true significance level depends greatly on censoring and is usually smaller than the nominal level. While Haider et al. [10] referred to the too frequent acceptance with D-calibration of the true model as “p-value boosting”, this may be considered unreliable because it depends on the censoring scheme and censoring rate.

A-calibration can be used to formally test if the suggested predictive model is the true model. However, in practice, the predictive model would never be the true underlying survival model, and hence even good models would have a risk of rejection greater than the chosen significance level. Therefore, if a binary accept/reject of the predictive model is demanded, it appears reasonable to choose a smaller significance level, for example, at 1%. Alternatively, the p-value can be interpreted as a calibration measure on a continuous scale, where higher p-values indicate better calibration. This also allows for comparison between multiple predictive models, where the model with the highest p-value would be considered to be the most well calibrated. However, one must be cautious with this approach, as comparing models based on the ranking of p-values requires that the validation datasets have the same size, follow the same distribution, and use the same partitioning. Therefore, using p-values, it is not straightforward to compare the performance of a new model to other models considered in previous studies. This is a clear limitation compared to other performance measures like the C-index and the IBS, which can be compared much more straightforwardly between models.

One issue of D-calibration which A-calibration has not solved is the arbitrary choice of partitioning intervals \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{I_k\}_{k=1,\dots ,K}$$\end{document} that is left to the investigator to determine. While the impact of the choice of intervals can be investigated through a sensitivity analysis, it is still a disadvantage of the approach. Future research could involve applying a different GOF test to the PIT residuals which does not depend on such an partition, for instance by adapting Kolmogorov-Smirnov [18] or Anderson-Darling [19] tests to censored samples.

“Furthermore, a new issue has emerged with A-calibration that was not present with D-calibration. Specifically, in the case of bounded censoring, the censoring distribution H has support on the interval [0, a] for some \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \le a \le 1$$\end{document} . We have proposed using the empirical support of the transformed censored survival times as a workaround, although this approach lacks theoretical justification. Despite this, in our simulations involving uniform censoring — even under heavy censoring scenarios where the upper support limit a is particularly small — A-calibration consistently outperformed D-calibration”.

An assumption underpinning the use of the probability integral transform is that the predictive survival model \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{S}$$\end{document} is independent of the validation data. This means that the A-calibration of a model should be computed on a validation dataset independent of the data set used for training the predictive model.

Conclusion

This paper introduces A-calibration as a new GOF testing method for predictive models in the context of censored survival data. Through theoretical considerations, a simulation study, and a case study, the method is shown to be superior to the existing alternative of D-calibration in terms of power under censoring.

Supplementary Information

Supplementary Material 1.