Beyond validation loss: Clinically-tailored optimization metrics improve a model's clinical performance

TL;DR

This paper demonstrates that optimizing machine learning models using clinically-tailored metrics, rather than traditional validation loss, leads to better clinical performance in healthcare applications.

Contribution

It introduces the concept of using clinically-relevant, non-differentiable metrics for model optimization and provides experimental evidence of their superiority over validation loss.

Findings

Clinically-tailored metrics improve model performance on clinical tasks.

Using these metrics yields models better aligned with healthcare goals.

Extra effort in defining and coding metrics enhances clinical relevance.

Abstract

A key task in ML is to optimize models at various stages, e.g. by choosing hyperparameters or picking a stopping point. A traditional ML approach is to use validation loss, i.e. to apply the training loss function on a validation set to guide these optimizations. However, ML for healthcare has a distinct goal from traditional ML: Models must perform well relative to specific clinical requirements, vs. relative to the loss function used for training. These clinical requirements can be captured more precisely by tailored metrics. Since many optimization tasks do not require the driving metric to be differentiable, they allow a wider range of options, including the use of metrics tailored to be clinically-relevant. In this paper we describe two controlled experiments which show how the use of clinically-tailored metrics provide superior model optimization compared to validation loss, in the sense of better performance on the clinical task. The use of clinically-relevant metrics for optimization entails some extra effort, to define the metrics and to code them into the pipeline. But it can yield models that better meet the central goal of ML for healthcare: strong performance in the clinic.