Spline-Based Probability Calibration

TL;DR

SplineCalib is a robust, non-parametric calibration method using smoothing splines that improves probability estimates in classification tasks, especially for deep learning models, leading to better accuracy and log-loss.

Contribution

The paper introduces SplineCalib, a novel spline-based calibration technique that enhances probability calibration and extends to multi-class problems, with a cross-validated, data-conserving approach.

Findings

Significant improvements in log-loss and accuracy across multiple datasets.

Effective calibration in deep learning models with overconfident scores.

Open-source implementation available in the ml-insights Python package.

Abstract

In many classification problems it is desirable to output well-calibrated probabilities on the different classes. We propose a robust, non-parametric method of calibrating probabilities called SplineCalib that utilizes smoothing splines to determine a calibration function. We demonstrate how applying certain transformations as part of the calibration process can improve performance on problems in deep learning and other domains where the scores tend to be "overconfident". We adapt the approach to multi-class problems and find that better calibration can improve accuracy as well as log-loss by better resolving uncertain cases. Finally, we present a cross-validated approach to calibration which conserves data. Significant improvements to log-loss and accuracy are shown on several different problems. We also introduce the ml-insights python package which contains an implementation of the SplineCalib algorithm.