Calibrated Top-1 Uncertainty estimates for classification by score based models

TL;DR

This paper improves the calibration of Top-1 uncertainty estimates in classification models by focusing on Top-1 error probability, significantly reducing calibration errors and enhancing score performance.

Contribution

The paper introduces a calibration approach specifically for Top-1 error probability estimates, achieving less than one percent calibration error and better benchmark scores.

Findings

Calibration error reduced to below 1%

Significant improvement in benchmark scores

Enhanced reliability of Top-1 uncertainty estimates

Abstract

While the accuracy of modern deep learning models has significantly improved in recent years, the ability of these models to generate uncertainty estimates has not progressed to the same degree. Uncertainty methods are designed to provide an estimate of class probabilities when predicting class assignment. While there are a number of proposed methods for estimating uncertainty, they all suffer from a lack of calibration: predicted probabilities can be off from empirical ones by a few percent or more. By restricting the scope of our predictions to only the probability of Top-1 error, we can decrease the calibration error of existing methods to less than one percent. As a result, the scores of the methods also improve significantly over benchmarks.