On the evaluation of time-to-event, survival time and first passage time forecasts

TL;DR

This paper introduces a framework for evaluating time-to-event forecasts under right-censoring, ensuring truthful probabilistic assessments and providing diagnostic tools, with applications to flood and wind event predictions.

Contribution

It develops a novel evaluation framework for censored time-to-event forecasts, including proper scoring rules and elicitable point forecasts, extending existing methods to censored data scenarios.

Findings

Proper scoring rules induce truthful probabilistic forecasts.

Quantile forecasts are provably elicitable under censoring.

The proposed methods effectively rank forecasters in synthetic experiments.

Abstract

Time-to-event forecasts are essential when decisions depend on event timing. This article develops a framework for evaluating such forecasts when the event has not yet occurred or is not predicted within the forecast horizon. We introduce a theory of provisional evaluation, in which each forecast is assessed against its right-censored realization, defined as the minimum of the event time and the evaluation time. For probabilistic forecasts, we show that strictly proper scoring rules induce provisionally strictly proper scoring rules, whose expected score, computed from the right-censored realization, is optimized under truthful forecasting. Threshold-weighted versions of the continuous ranked probability score and the logarithmic score satisfy this property. We also develop a theory for scoring point (single-valued) forecasts under right-censoring. Quantile and interquartile range forecasts are shown to be provisionally elicitable, meaning that scoring functions exist for which these functionals uniquely minimize the expected score, whereas the expectation functional is not provisionally elicitable. A synthetic experiment demonstrates that the proposed scores correctly rank forecasters. Diagnostic tools, including Murphy diagrams and reliability diagrams, extend naturally. Applications to operational time-to-flood and time-to-strong-wind forecasts illustrate the approach.