Rethinking Early Stopping: Refine, Then Calibrate

TL;DR

This paper introduces a new perspective on calibration and refinement in probabilistic classifiers, proposing a two-stage training process that improves prediction quality by separately optimizing these components.

Contribution

It presents a variational formulation of calibration-refinement decomposition and a novel training method that separately minimizes refinement and calibration errors.

Findings

The proposed method improves calibration and refinement in classifiers.

Calibration and refinement errors are not minimized simultaneously during training.

Separately optimizing refinement and calibration yields better probabilistic predictions.

Abstract

Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with proper losses, such as cross-entropy, which decompose into two components: calibration error assesses general under/overconfidence, while refinement error measures the ability to distinguish different classes. In this paper, we present a novel variational formulation of the calibration-refinement decomposition that sheds new light on post-hoc calibration, and enables rapid estimation of the different terms. Equipped with this new perspective, we provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training. Selecting the best epoch based on validation loss thus leads to a compromise point that is suboptimal for both terms. To address this, we propose minimizing refinement error only during training (Refine,...), before minimizing calibration error post hoc, using standard techniques (...then Calibrate). Our method integrates seamlessly with any classifier and consistently improves performance across diverse classification tasks.