Second-order accuracy metrics for scoring models and their practical use

TL;DR

This paper introduces second-order accuracy metrics for scoring models that help diagnose whether models better identify good or bad objects, aiding in validation and calibration.

Contribution

It proposes new integral and numerical second-order metrics, including binary event and default probability-based versions, for improved model validation.

Findings

Metrics validate calibration settings effectively.

Application to rating agencies demonstrates practical utility.

Metrics reveal model distortions and calibration issues.

Abstract

The paper proposes new second-order accuracy metrics for scoring or rating models, which show the target preference of the model, it is better to diagnose good objects or better to diagnose bad ones for a constant generally accepted predictive power determined by the first order metric that is known as the Gini index. There are two metrics, they have both an integral representation and a numerical one. The numerical representation of metrics is of two types, the first of which is based on binary events to evaluate the model, the second on the default probability given by the model. Comparison of the results of calculating the metrics allows you to validate the calibration settings of the scoring or rating model and reveals its distortions. The article provides examples of calculating second-order accuracy metrics for ratings of several rating agencies, as well as for the well known approach to calibration based on van der Burg's ROC curves.