T-Cal: An optimal test for the calibration of predictive models

TL;DR

This paper introduces T-Cal, a minimax optimal statistical test for assessing the calibration of probabilistic models, especially effective when class probabilities are smooth functions, with broad practical applicability.

Contribution

The paper develops T-Cal, a new hypothesis testing method for calibration that is minimax optimal and adaptive to unknown smoothness, filling a gap in reliable calibration assessment.

Findings

T-Cal is minimax optimal for calibration testing.

It performs well with deep neural networks and standard calibration methods.

The method is practical and broadly applicable.

Abstract

The prediction accuracy of machine learning methods is steadily increasing, but the calibration of their uncertainty predictions poses a significant challenge. Numerous works focus on obtaining well-calibrated predictive models, but less is known about reliably assessing model calibration. This limits our ability to know when algorithms for improving calibration have a real effect, and when their improvements are merely artifacts due to random noise in finite datasets. In this work, we consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. The null hypothesis is that the predictive model is calibrated, while the alternative hypothesis is that the deviation from calibration is sufficiently large. We find that detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. When the conditional class probabilities are H\"older continuous, we propose T-Cal, a minimax optimal test for calibration based on a debiased plug-in estimator of the $\ell_2$-Expected Calibration Error (ECE). We further propose Adaptive T-Cal, a version that is adaptive to unknown smoothness. We verify our theoretical findings with a broad range of experiments, including with several popular deep neural net architectures and several standard post-hoc calibration methods. T-Cal is a practical general-purpose tool, which -- combined with classical tests for discrete-valued predictors -- can be used to test the calibration of virtually any probabilistic classification method.