A Variational Estimator for $L_p$ Calibration Errors

TL;DR

This paper introduces a variational method to accurately estimate $L_p$ calibration errors in machine learning, improving over existing techniques by reducing overestimation and distinguishing confidence levels.

Contribution

It extends a recent variational framework to cover $L_p$ calibration errors, enabling more precise and reliable calibration assessment in multiclass settings.

Findings

The method effectively separates over- and under-confidence.

It avoids overestimation common in non-variational approaches.

Extensive experiments validate the approach's accuracy.

Abstract

Calibration$\unicode{x2014}$the problem of ensuring that predicted probabilities align with observed class frequencies$\unicode{x2014}$is a basic desideratum for reliable prediction with machine learning systems. Calibration error is traditionally assessed via a divergence function, using the expected divergence between predictions and empirical frequencies. Accurately estimating this quantity is challenging, especially in the multiclass setting. Here, we show how to extend a recent variational framework for estimating calibration errors beyond divergences induced induced by proper losses, to cover a broad class of calibration errors induced by $L_p$ divergences. Our method can separate over- and under-confidence and, unlike non-variational approaches, avoids overestimation. We provide extensive experiments and integrate our code in the open-source package probmetrics (https://github.com/dholzmueller/probmetrics) for evaluating calibration errors.