Learn-By-Calibrating: Using Calibration as a Training Objective

TL;DR

This paper introduces a novel training method that uses calibration error as an objective to improve deep neural network predictions, leading to better calibration and interpretability without explicit uncertainty measurement.

Contribution

It proposes a new algorithm for training deep regression models by leveraging calibration intervals, outperforming existing regularization strategies.

Findings

Our method improves calibration accuracy in deep regression models.

It enhances the quality of mean predictions compared to traditional regularization.

Augmenting partial dependence plots with prediction intervals reveals new data-target relationships.

Abstract

Calibration error is commonly adopted for evaluating the quality of uncertainty estimators in deep neural networks. In this paper, we argue that such a metric is highly beneficial for training predictive models, even when we do not explicitly measure the uncertainties. This is conceptually similar to heteroscedastic neural networks that produce variance estimates for each prediction, with the key difference that we do not place a Gaussian prior on the predictions. We propose a novel algorithm that performs simultaneous interval estimation for different calibration levels and effectively leverages the intervals to refine the mean estimates. Our results show that, our approach is consistently superior to existing regularization strategies in deep regression models. Finally, we propose to augment partial dependence plots, a model-agnostic interpretability tool, with expected prediction intervals to reveal interesting dependencies between data and the target.