Is the new model better? One metric says yes, but the other says no. Which metric do I use?

TL;DR

This paper compares two incremental value metrics, IncV-AUC and IncV-AP, analyzing their differences, relationships, and implications for model evaluation, especially when metrics conflict in medical risk prediction.

Contribution

It provides an analytical comparison of IncV-AUC and IncV-AP, revealing their weighting schemes and relationships with proper scoring rules, aiding better metric selection.

Findings

IncV-AP emphasizes high-risk group changes, IncV-AUC weights all risk groups equally.

IncV-AP and IncV-sBrS are highly consistent, IncV-AUC shows negative correlation with them.

Differences between metrics increase as the event rate decreases.

Abstract

Incremental value (IncV) evaluates the performance change from an existing risk model to a new model. It is one of the key considerations in deciding whether a new risk model performs better than the existing one. Problems arise when different IncV metrics contradict each other. For example, compared with a prescribed-dose model, an ovarian-dose model for predicting acute ovarian failure has a slightly lower area under the receiver operating characteristic curve (AUC) but increases the area under the precision-recall curve (AP) by 48%. This phenomenon of conflicting conclusions is not uncommon, and it creates a dilemma in medical decision making. In this article, we examine the analytical connections and differences between two IncV metrics: IncV in AUC (IncV-AUC) and IncV in AP (IncV-AP). Additionally, since they are both semi-proper scoring rules, we compare them with a strictly proper scoring rule: the IncV of the scaled Brier score (IncV-sBrS), via a numerical study. We demonstrate that both IncV-AUC and IncV-AP are weighted averages of the changes (from the existing model to the new one) in separating the risk score distributions between events and non-events. However, IncV-AP assigns heavier weights to the changes in the high-risk group, whereas IncV-AUC weights the changes equally. In the numerical study, we find that IncV-AP has a wide range, from negative to positive, but the size of IncV-AUC is much smaller. In addition, IncV-AP and IncV-sBr Sare highly consistent, but IncV-AUC is negatively correlated with IncV-sBrS and IncV-AP at a low event rate. IncV-AUC and IncV-AP are the least consistent among the three pairs, and their differences are more pronounced as the event rate decreases.