Calibrated Prediction Intervals for Neural Network Regressors

TL;DR

This paper introduces two novel methods, empirical calibration and temperature scaling, to produce well-calibrated prediction intervals for neural network regressors, enhancing the reliability of uncertainty estimates across various domains.

Contribution

The paper presents the first methods specifically designed for calibrated prediction intervals in neural network regression, addressing a gap in existing calibration techniques.

Findings

Both methods produce calibrated prediction intervals at any confidence level.

The methods are effective across different datasets and neural network architectures.

Practical recommendations improve the accuracy of the calibration.

Abstract

Ongoing developments in neural network models are continually advancing the state of the art in terms of system accuracy. However, the predicted labels should not be regarded as the only core output; also important is a well-calibrated estimate of the prediction uncertainty. Such estimates and their calibration are critical in many practical applications. Despite their obvious aforementioned advantage in relation to accuracy, contemporary neural networks can, generally, be regarded as poorly calibrated and as such do not produce reliable output probability estimates. Further, while post-processing calibration solutions can be found in the relevant literature, these tend to be for systems performing classification. In this regard, we herein present two novel methods for acquiring calibrated predictions intervals for neural network regressors: empirical calibration and temperature scaling. In experiments using different regression tasks from the audio and computer vision domains, we find that both our proposed methods are indeed capable of producing calibrated prediction intervals for neural network regressors with any desired confidence level, a finding that is consistent across all datasets and neural network architectures we experimented with. In addition, we derive an additional practical recommendation for producing more accurate calibrated prediction intervals. We release the source code implementing our proposed methods for computing calibrated predicted intervals. The code for computing calibrated predicted intervals is publicly available.