Stop Chasing the C-index: This Is How We Should Evaluate Our Survival Models

TL;DR

This paper critiques the overreliance on the C-index for evaluating survival models, emphasizing the need for metrics that assess calibration and time-to-event accuracy, and proposes a framework for proper evaluation.

Contribution

It highlights the limitations of the C-index in survival analysis and offers a set of criteria and methods for more comprehensive model evaluation.

Findings

C-index only measures discriminative ability, not calibration or time-to-event accuracy.

Current survival metrics often conflate model validity with metric validity.

Proper evaluation requires considering calibration, censoring, and other aspects beyond discrimination.

Abstract

We argue that many survival analysis and time-to-event models are incorrectly evaluated. First, we survey many examples of evaluation approaches in the literature and find that most rely on concordance (C-index). However, the C-index only measures a model's discriminative ability and does not assess other important aspects, such as the accuracy of the time-to-event predictions or the calibration of the model's probabilistic estimates. Next, we present a set of key desiderata for choosing the right evaluation metric and discuss their pros and cons. These are tailored to the challenges in survival analysis, such as sensitivity to miscalibration and various censoring assumptions. We hypothesize that the current development of survival metrics conforms to a double-helix ladder, and that model validity and metric validity must stand on the same rung of the assumption ladder. Finally, we discuss the appropriate methods for evaluating a survival model in practice and summarize various viewpoints opposing our analysis.