A Confidence Interval for the $\ell_2$ Expected Calibration Error

TL;DR

This paper introduces a statistical method to construct confidence intervals for the $ ext{l}_2$ Expected Calibration Error in machine learning models, enabling more reliable calibration assessment.

Contribution

It develops asymptotically valid confidence intervals for the $ ext{l}_2$ ECE, accounting for different convergence behaviors in calibrated and miscalibrated models.

Findings

Confidence intervals are valid and shorter than resampling methods.

Asymptotic normality of the debiased ECE estimator is established.

Methods apply to top-1-to-$k$ calibration, including full calibration.

Abstract

Recent advances in machine learning have significantly improved prediction accuracy in various applications. However, ensuring the calibration of probabilistic predictions remains a significant challenge. Despite efforts to enhance model calibration, the rigorous statistical evaluation of model calibration remains less explored. In this work, we develop confidence intervals the $\ell_2$ Expected Calibration Error (ECE). We consider top-1-to-$k$ calibration, which includes both the popular notion of confidence calibration as well as full calibration. For a debiased estimator of the ECE, we show asymptotic normality, but with different convergence rates and asymptotic variances for calibrated and miscalibrated models. We develop methods to construct asymptotically valid confidence intervals for the ECE, accounting for this behavior as well as non-negativity. Our theoretical findings are supported through extensive experiments, showing that our methods produce valid confidence intervals with shorter lengths compared to those obtained by resampling-based methods.