Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?

TL;DR

This paper investigates how bin-wise variance scaling can enhance the calibration of prediction uncertainties in regression models, focusing on both consistency and adaptivity through various binning strategies and loss functions.

Contribution

It introduces adaptations of Binwise Variance Scaling, including input-feature based binning, to improve uncertainty calibration in regression tasks.

Findings

BVS improves calibration over uniform scaling.

Input-feature based binning enhances adaptivity.

Compared to isotonic regression, BVS variants show competitive performance.

Abstract

Binwise Variance Scaling (BVS) has recently been proposed as a post hoc recalibration method for prediction uncertainties of machine learning regression problems that is able of more efficient corrections than uniform variance (or temperature) scaling. The original version of BVS uses uncertainty-based binning, which is aimed to improve calibration conditionally on uncertainty, i.e. consistency. I explore here several adaptations of BVS, in particular with alternative loss functions and a binning scheme based on an input-feature (X) in order to improve adaptivity, i.e. calibration conditional on X. The performances of BVS and its proposed variants are tested on a benchmark dataset for the prediction of atomization energies and compared to the results of isotonic regression.