Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing

TL;DR

This paper introduces SmoothECE, a kernel smoothing-based calibration measure and reliability diagram that addresses flaws in traditional binning methods, providing a consistent and visually interpretable calibration assessment.

Contribution

The paper proposes a novel kernel smoothing approach for calibration measures and reliability diagrams, ensuring consistency and improved interpretability over traditional binning methods.

Findings

SmoothECE is a well-behaved, consistent calibration measure.

The smoothed reliability diagram visually encodes the SmoothECE.

Provided Python package simplifies calibration measurement and visualization.

Abstract

Calibration measures and reliability diagrams are two fundamental tools for measuring and interpreting the calibration of probabilistic predictors. Calibration measures quantify the degree of miscalibration, and reliability diagrams visualize the structure of this miscalibration. However, the most common constructions of reliability diagrams and calibration measures -- binning and ECE -- both suffer from well-known flaws (e.g. discontinuity). We show that a simple modification fixes both constructions: first smooth the observations using an RBF kernel, then compute the Expected Calibration Error (ECE) of this smoothed function. We prove that with a careful choice of bandwidth, this method yields a calibration measure that is well-behaved in the sense of (B{\l}asiok, Gopalan, Hu, and Nakkiran 2023a) -- a consistent calibration measure. We call this measure the SmoothECE. Moreover, the reliability diagram obtained from this smoothed function visually encodes the SmoothECE, just as binned reliability diagrams encode the BinnedECE. We also provide a Python package with simple, hyperparameter-free methods for measuring and plotting calibration: `pip install relplot\`.