Structured Matrix Scaling for Multi-Class Calibration

TL;DR

This paper introduces structured matrix scaling methods for multi-class calibration that improve probability estimates of classifiers by managing bias-variance tradeoffs with regularization and optimization, outperforming existing techniques.

Contribution

It proposes a novel structured matrix scaling approach for multi-class calibration, addressing overfitting issues and providing efficient, open-source implementations.

Findings

Structured regularization improves calibration accuracy.

Enhanced calibration methods outperform existing temperature and matrix scaling.

Effective bias-variance tradeoff management leads to substantial gains.

Abstract

Post-hoc recalibration methods are widely used to ensure that classifiers provide faithful probability estimates. We argue that parametric recalibration functions based on logistic regression can be motivated from a simple theoretical setting for both binary and multiclass classification. This insight motivates the use of more expressive calibration methods beyond standard temperature scaling. For multi-class calibration however, a key challenge lies in the increasing number of parameters introduced by more complex models, often coupled with limited calibration data, which can lead to overfitting. Through extensive experiments, we demonstrate that the resulting bias-variance tradeoff can be effectively managed by structured regularization, robust preprocessing and efficient optimization. The resulting methods lead to substantial gains over existing logistic-based calibration techniques. We provide efficient and easy-to-use open-source implementations of our methods, making them an attractive alternative to common temperature, vector, and matrix scaling implementations.