Reassessing How to Compare and Improve the Calibration of Machine Learning Models

TL;DR

This paper critically examines calibration metrics in machine learning, introduces a new visualization method to jointly assess calibration and generalization, and provides theoretical insights into calibration errors and estimator consistency.

Contribution

It reveals limitations of current calibration evaluation practices, proposes an extended reliability diagram for joint visualization, and proves new theoretical results on calibration errors and estimator consistency.

Findings

Trivial recalibration methods can appear optimal without proper metrics.

The new visualization detects calibration-generalization trade-offs.

Theoretical proofs relate calibration errors and establish estimator consistency.

Abstract

A machine learning model is calibrated if its predicted probability for an outcome matches the observed frequency for that outcome conditional on the model prediction. This property has become increasingly important as the impact of machine learning models has continued to spread to various domains. As a result, there are now a dizzying number of recent papers on measuring and improving the calibration of (specifically deep learning) models. In this work, we reassess the reporting of calibration metrics in the recent literature. We show that there exist trivial recalibration approaches that can appear seemingly state-of-the-art unless calibration and prediction metrics (i.e. test accuracy) are accompanied by additional generalization metrics such as negative log-likelihood. We then use a calibration-based decomposition of Bregman divergences to develop a new extension to reliability diagrams that jointly visualizes calibration and generalization error, and show how our visualization can be used to detect trade-offs between calibration and generalization. Along the way, we prove novel results regarding the relationship between full calibration error and confidence calibration error for Bregman divergences. We also establish the consistency of the kernel regression estimator for calibration error used in our visualization approach, which generalizes existing consistency results in the literature.