Recalibrating binary probabilistic classifiers

TL;DR

This paper introduces new recalibration methods for binary classifiers based on distribution shift assumptions, demonstrating their effectiveness in credit risk evaluation scenarios.

Contribution

The paper proposes two novel recalibration techniques, CSPD and QMM, grounded in distribution shift theory, enhancing classifier calibration under prior probability constraints.

Findings

QMM provides conservative calibration results

Methods perform well under distribution shift assumptions

Improved calibration for credit risk applications

Abstract

Recalibration of binary probabilistic classifiers to a target prior probability is an important task in areas like credit risk management. However, recalibration of a classifier learned on a training dataset to a target on a test dataset in general is not a well-defined problem because there might be more than one way to transform the original posterior probabilities such that the target is matched. In this paper, methods for recalibration are analysed from a distribution shift perspective. Distribution shift assumptions linked to the area under the curve (AUC) of a probabilistic classifier are found to be useful for the design of meaningful recalibration methods. Two new methods called parametric covariate shift with posterior drift (CSPD) and ROC-based quasi moment matching (QMM) are proposed and tested together with some other methods in an example setting. The outcomes of the test suggest that the QMM methods discussed in the paper can provide appropriately conservative results in evaluations with concave functions like for instance risk weights functions for credit risk.