Calibrating sufficiently

TL;DR

This paper explores the concept of grouping loss in probabilistic classifier calibration, linking it to sufficiency and proposing methods to reduce this loss for better calibration accuracy.

Contribution

It identifies comonotonicity as a criterion for sufficiency, revisits the probing reduction approach, and discusses Brier curves to improve classifier calibration.

Findings

Grouping loss can be minimized using comonotonicity.

Probing reduction produces estimators that reduce grouping loss.

Brier curves assist in training and achieving sufficient calibration.

Abstract

When probabilistic classifiers are trained and calibrated, the so-called grouping loss component of the calibration loss can easily be overlooked. Grouping loss refers to the gap between observable information and information actually exploited in the calibration exercise. We investigate the relation between grouping loss and the concept of sufficiency, identifying comonotonicity as a useful criterion for sufficiency. We revisit the probing reduction approach of Langford & Zadrozny (2005) and find that it produces an estimator of probabilistic classifiers that reduces grouping loss. Finally, we discuss Brier curves as tools to support training and 'sufficient' calibration of probabilistic classifiers.