Measures of predictive accuracy, miscalibration and discrimination

TL;DR

This paper analyzes evaluation metrics for point predictors, highlighting limitations of popular scores like ABC and Gini, and advocates for mean-consistent loss functions for honest model assessment.

Contribution

It introduces a new Murphy's decomposition variant, relates it to Lorenz-based measures, and demonstrates the shortcomings of existing scores for model evaluation.

Findings

ABC and ABC$^2$ depend on predictor-dependent weights, leading to biased evaluation.

Gini score also fails to align with mean-consistent scoring functions.

Using mean-consistent loss functions improves the honesty of model comparisons.

Abstract

We study the evaluation of real-valued point predictors under the decision-theoretic framework of mean-consistent loss functions given by the Bregman divergences. We first derive a new version of Murphy's decomposition of the expected loss which does not directly include the response itself but only its predictors. We then relate the miscalibration and the discrimination component of the Murphy's decomposition to Lorenz-curve-based accuracy measures that are widely used in practice. Besides the usual area between the concentration and Lorenz curves, ABC, we introduce a mean-squared version ABC$^2$ that mitigates some of the weaknesses of the original ABC in identifying mean-calibration. More importantly, both ABC and ABC$^2$ are shown to rely on predictor-dependent weights, so they fail to align with the class of mean-consistent scoring functions. In the same spirit, we derive a similar result for the widely used Gini score. These results indicate that ABC, ABC$^2$ and Gini scores may lead to dishonest evaluation of point predictions when used for model selection; this gives support to use mean-consistent loss functions as well as the miscalibration and the discrimination measure from the Murphy's decomposition of the expected loss for model evaluation. Finally, we study forecast dominance when Lorenz curves intersect. We show that Lorenz and Murphy's curves have the same number of crossings and, in the one-crossing case, we establish weaker dominance criteria for subclasses of Bregman divergences through third-degree stochastic dominance.