Optimizing Estimators of Squared Calibration Errors in Classification

TL;DR

This paper introduces a new mean-squared error risk for comparing and optimizing calibration error estimators in classification, utilizing a regression reformulation to improve estimator selection and tuning.

Contribution

It reformulates calibration estimation as a regression problem, providing a systematic way to compare and optimize calibration estimators using a new risk measure.

Findings

The proposed pipeline improves calibration estimator performance.

Kernel ridge regression-based estimators outperform existing methods.

The approach enhances calibration accuracy on standard image classification tasks.

Abstract

In this work, we propose a mean-squared error-based risk that enables the comparison and optimization of estimators of squared calibration errors in practical settings. Improving the calibration of classifiers is crucial for enhancing the trustworthiness and interpretability of machine learning models, especially in sensitive decision-making scenarios. Although various calibration (error) estimators exist in the current literature, there is a lack of guidance on selecting the appropriate estimator and tuning its hyperparameters. By leveraging the bilinear structure of squared calibration errors, we reformulate calibration estimation as a regression problem with independent and identically distributed (i.i.d.) input pairs. This reformulation allows us to quantify the performance of different estimators even for the most challenging calibration criterion, known as canonical calibration. Our approach advocates for a training-validation-testing pipeline when estimating a calibration error on an evaluation dataset. We demonstrate the effectiveness of our pipeline by optimizing existing calibration estimators and comparing them with novel kernel ridge regression-based estimators on standard image classification tasks.