Information-theoretic Generalization Analysis for Expected Calibration Error

TL;DR

This paper provides a theoretical analysis of the estimation bias in expected calibration error (ECE) using binning strategies, offering bounds and insights into optimal bin choices, with implications for understanding model calibration.

Contribution

It introduces the first comprehensive bias bounds for common binning strategies in ECE estimation and extends the analysis to generalization error using information theory.

Findings

Established upper bounds on bias with improved convergence rates

Identified the optimal number of bins to minimize bias

Bounds are nonvacuous in deep learning model experiments

Abstract

While the expected calibration error (ECE), which employs binning, is widely adopted to evaluate the calibration performance of machine learning models, theoretical understanding of its estimation bias is limited. In this paper, we present the first comprehensive analysis of the estimation bias in the two common binning strategies, uniform mass and uniform width binning. Our analysis establishes upper bounds on the bias, achieving an improved convergence rate. Moreover, our bounds reveal, for the first time, the optimal number of bins to minimize the estimation bias. We further extend our bias analysis to generalization error analysis based on the information-theoretic approach, deriving upper bounds that enable the numerical evaluation of how small the ECE is for unknown data. Experiments using deep learning models show that our bounds are nonvacuous thanks to this information-theoretic generalization analysis approach.