Optimizing Calibration by Gaining Aware of Prediction Correctness

TL;DR

This paper introduces a novel post-hoc calibration method that improves confidence alignment with prediction correctness by leveraging transformed sample versions, addressing limitations of traditional CE loss-based calibration.

Contribution

It proposes a new calibration objective that better aligns confidence with correctness and utilizes transformed samples during training for enhanced calibration performance.

Findings

Achieves competitive calibration on in-distribution and out-of-distribution data.

Addresses limitations of CE loss in calibration tasks.

Provides analysis distinguishing the new method from traditional objectives.

Abstract

Model calibration aims to align confidence with prediction correctness. The Cross-Entropy (CE) loss is widely used for calibrator training, which enforces the model to increase confidence on the ground truth class. However, we find the CE loss has intrinsic limitations. For example, for a narrow misclassification (e.g., a test sample is wrongly classified and its softmax score on the ground truth class is 0.4), a calibrator trained by the CE loss often produces high confidence on the wrongly predicted class, which is undesirable. In this paper, we propose a new post-hoc calibration objective derived from the aim of calibration. Intuitively, the proposed objective function asks that the calibrator decrease model confidence on wrongly predicted samples and increase confidence on correctly predicted samples. Because a sample itself has insufficient ability to indicate correctness, we use its transformed versions (e.g., rotated, greyscaled, and color-jittered) during calibrator training. Trained on an in-distribution validation set and tested with isolated, individual test samples, our method achieves competitive calibration performance on both in-distribution and out-of-distribution test sets compared with the state of the art. Further, our analysis points out the difference between our method and commonly used objectives such as CE loss and Mean Square Error (MSE) loss, where the latters sometimes deviates from the calibration aim.