# A jackknife approach to estimate the prediction uncertainty from binary classifiers under right-censoring

## TL;DR

This paper introduces a new method to estimate prediction uncertainty in binary classifiers when dealing with time-to-event data that includes censored observations.

## Contribution

The novel contribution is an adjustment to the jackknife estimator that incorporates inverse-probability-of-censoring-weighting for uncertainty estimation in machine learning models.

## Key findings

- The adjusted jackknife estimator provides unbiased standard error estimates in simple parametric models.
- Prediction uncertainty is higher when using binary classifiers on dichotomized data compared to survival models.
- The method is demonstrated on kidney transplant survival data and validated through simulations.

## Abstract

Clinical prediction models are developed to estimate a patient’s risk for a specific outcome, and machine learning is frequently employed to improve prediction accuracy. When the outcome is some event that happens over time, binary classifiers can predict the risk at specific time points if right-censoring is addressed by inverse-probability-of-censoring-weighting . Assessing prediction uncertainty is crucial for interpreting individual risks, but there is limited knowledge on how to consider inverse-probability-of-censoring-weighting when estimating this uncertainty. We propose an adjustment of the infinitesimal jackknife estimator for the standard error of predictions that incorporates inverse-probability-of-censoring-weighting. By using a nonparametric approach, it is broadly applicable, especially to machine learning classifiers. For a simple tractable example, we show that the proposed adjustment reveals unbiased standard error estimates. For other situations, we evaluate performance through simulation studies under both parametric models with inverse-probability-of-censoring-weighting-customized log-likelihood and machine learning with inverse-probability-of-censoring-weighting-customized loss function. We illustrate the methods by predicting post-transplant survival probabilities, using national kidney transplant registry data. Our findings show that the proposed estimator is useful for quantifying prediction uncertainty of inverse-probability-of-censoring-weighting classifiers. Applications to simulated and real data show that prediction uncertainty increases when employing binary classifiers on dichotomized data compared to predictions from survival models.