Improving model calibration with accuracy versus uncertainty optimization

TL;DR

This paper introduces a novel optimization approach using an accuracy versus uncertainty loss to improve the calibration of uncertainty estimates in deep neural networks, especially under distributional shifts.

Contribution

It proposes the AvUC loss function for joint accuracy and uncertainty calibration, applicable during training and post-hoc, demonstrating superior calibration on large-scale image classification tasks.

Findings

Better calibration than existing methods under distributional shift

Effective both during training and post-hoc calibration

Improved uncertainty quantification in safety-critical applications

Abstract

Obtaining reliable and accurate quantification of uncertainty estimates from deep neural networks is important in safety-critical applications. A well-calibrated model should be accurate when it is certain about its prediction and indicate high uncertainty when it is likely to be inaccurate. Uncertainty calibration is a challenging problem as there is no ground truth available for uncertainty estimates. We propose an optimization method that leverages the relationship between accuracy and uncertainty as an anchor for uncertainty calibration. We introduce a differentiable accuracy versus uncertainty calibration (AvUC) loss function that allows a model to learn to provide well-calibrated uncertainties, in addition to improved accuracy. We also demonstrate the same methodology can be extended to post-hoc uncertainty calibration on pretrained models. We illustrate our approach with mean-field stochastic variational inference and compare with state-of-the-art methods. Extensive experiments demonstrate our approach yields better model calibration than existing methods on large-scale image classification tasks under distributional shift.