Confidence Calibration of Classifiers with Many Classes

TL;DR

This paper introduces a new approach to calibrate confidence scores in neural network classifiers with many classes by transforming the problem into calibrating a surrogate binary classifier, improving calibration performance.

Contribution

The paper proposes a novel method that simplifies multiclass calibration by reducing it to binary calibration, enabling more effective use of existing calibration techniques.

Findings

Significant improvement in calibration accuracy across multiple neural network models.

Effective calibration method for problems with many classes.

Enhanced reliability of confidence scores in image and text classification.

Abstract

For classification models based on neural networks, the maximum predicted class probability is often used as a confidence score. This score rarely predicts well the probability of making a correct prediction and requires a post-processing calibration step. However, many confidence calibration methods fail for problems with many classes. To address this issue, we transform the problem of calibrating a multiclass classifier into calibrating a single surrogate binary classifier. This approach allows for more efficient use of standard calibration methods. We evaluate our approach on numerous neural networks used for image or text classification and show that it significantly enhances existing calibration methods.