Class-wise and reduced calibration methods

TL;DR

This paper introduces class-wise and reduced calibration techniques that improve the reliability of probabilistic classifiers, especially in multi-class and imbalanced data scenarios, by transforming calibration problems and recalibrating per class.

Contribution

It proposes two novel calibration methods: a reduced calibration approach transforming complex problems into simpler ones, and class-wise calibration based on neural collapse, enhancing calibration accuracy.

Findings

Reduced calibration minimizes miscalibration in the full problem.

Class-wise calibration outperforms non class-wise methods on imbalanced datasets.

Combined methods significantly reduce prediction and calibration errors.

Abstract

For many applications of probabilistic classifiers it is important that the predicted confidence vectors reflect true probabilities (one says that the classifier is calibrated). It has been shown that common models fail to satisfy this property, making reliable methods for measuring and improving calibration important tools. Unfortunately, obtaining these is far from trivial for problems with many classes. We propose two techniques that can be used in tandem. First, a reduced calibration method transforms the original problem into a simpler one. We prove for several notions of calibration that solving the reduced problem minimizes the corresponding notion of miscalibration in the full problem, allowing the use of non-parametric recalibration methods that fail in higher dimensions. Second, we propose class-wise calibration methods, based on intuition building on a phenomenon called neural collapse and the observation that most of the accurate classifiers found in practice can be thought of as a union of K different functions which can be recalibrated separately, one for each class. These typically out-perform their non class-wise counterparts, especially for classifiers trained on imbalanced data sets. Applying the two methods together results in class-wise reduced calibration algorithms, which are powerful tools for reducing the prediction and per-class calibration errors. We demonstrate our methods on real and synthetic datasets and release all code as open source at https://github.com/appliedAI-Initiative