Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errors

TL;DR

This paper introduces a method for consistent and asymptotically unbiased estimation of proper calibration errors and refinement in probabilistic models, addressing a key gap in the evaluation of model calibration.

Contribution

It proposes a novel estimator for calibration errors and refinement that has proven statistical properties, including consistency and asymptotic unbiasedness, and introduces the Kullback--Leibler calibration error.

Findings

The estimator is consistent and asymptotically unbiased.

The relation between refinement and f-divergences is established.

Empirical validation confirms the estimator's properties and guides calibration method choice.

Abstract

Proper scoring rules evaluate the quality of probabilistic predictions, playing an essential role in the pursuit of accurate and well-calibrated models. Every proper score decomposes into two fundamental components -- proper calibration error and refinement -- utilizing a Bregman divergence. While uncertainty calibration has gained significant attention, current literature lacks a general estimator for these quantities with known statistical properties. To address this gap, we propose a method that allows consistent, and asymptotically unbiased estimation of all proper calibration errors and refinement terms. In particular, we introduce Kullback--Leibler calibration error, induced by the commonly used cross-entropy loss. As part of our results, we prove the relation between refinement and f-divergences, which implies information monotonicity in neural networks, regardless of which proper scoring rule is optimized. Our experiments validate empirically the claimed properties of the proposed estimator and suggest that the selection of a post-hoc calibration method should be determined by the particular calibration error of interest.