Effective Ways to Build and Evaluate Individual Survival Distributions

TL;DR

This paper discusses the importance of individual survival distribution models for personalized prognosis, compares various methods, and introduces a new evaluation metric called D-Calibration to assess their reliability.

Contribution

It introduces the concept of individual survival distribution models, compares existing evaluation metrics, and proposes D-Calibration as a novel way to assess model probability accuracy.

Findings

ISD models provide survival probabilities across all times.

D-Calibration effectively evaluates the meaningfulness of probability estimates.

Comparison of multiple ISD models on various datasets shows differing performance.

Abstract

An accurate model of a patient's individual survival distribution can help determine the appropriate treatment for terminal patients. Unfortunately, risk scores (e.g., from Cox Proportional Hazard models) do not provide survival probabilities, single-time probability models (e.g., the Gail model, predicting 5 year probability) only provide for a single time point, and standard Kaplan-Meier survival curves provide only population averages for a large class of patients meaning they are not specific to individual patients. This motivates an alternative class of tools that can learn a model which provides an individual survival distribution which gives survival probabilities across all times - such as extensions to the Cox model, Accelerated Failure Time, an extension to Random Survival Forests, and Multi-Task Logistic Regression. This paper first motivates such "individual survival distribution" (ISD) models, and explains how they differ from standard models. It then discusses ways to evaluate such models - namely Concordance, 1-Calibration, Brier score, and various versions of L1-loss - and then motivates and defines a novel approach "D-Calibration", which determines whether a model's probability estimates are meaningful. We also discuss how these measures differ, and use them to evaluate several ISD prediction tools, over a range of survival datasets.