The PAV algorithm optimizes binary proper scoring rules

TL;DR

The paper demonstrates that the PAV algorithm optimally calibrates binary classifiers under all regular proper scoring rules, extending previous convexity limitations and applying to probabilities and log-likelihood ratios.

Contribution

It proves the optimality of the PAV algorithm for binary calibration under all regular proper scoring rules, beyond convex cases, including log-likelihood ratios.

Findings

PAV solves non-parametric calibration with monotonicity constraints.

Optimality holds for all regular binary proper scoring rules.

Results apply to probabilities and log-likelihood ratios.

Abstract

There has been much recent interest in application of the pool-adjacent-violators (PAV) algorithm for the purpose of calibrating the probabilistic outputs of automatic pattern recognition and machine learning algorithms. Special cost functions, known as proper scoring rules form natural objective functions to judge the goodness of such calibration. We show that for binary pattern classifiers, the non-parametric optimization of calibration, subject to a monotonicity constraint, can be solved by PAV and that this solution is optimal for all regular binary proper scoring rules. This extends previous results which were limited to convex binary proper scoring rules. We further show that this result holds not only for calibration of probabilities, but also for calibration of log-likelihood-ratios, in which case optimality holds independently of the prior probabilities of the pattern classes.