Learning for Single-Shot Confidence Calibration in Deep Neural Networks through Stochastic Inferences

TL;DR

This paper introduces a stochastic inference-based framework for confidence calibration in deep neural networks, improving accuracy and reducing overconfidence by leveraging variance-weighted loss functions.

Contribution

It presents a novel variance-weighted loss function that enables single-inference confidence calibration, enhancing model reliability and accuracy.

Findings

High correlation between prediction variance and accuracy.

Improved confidence calibration across multiple models and datasets.

Significant reduction in overconfidence in deep neural networks.

Abstract

We propose a generic framework to calibrate accuracy and confidence of a prediction in deep neural networks through stochastic inferences. We interpret stochastic regularization using a Bayesian model, and analyze the relation between predictive uncertainty of networks and variance of the prediction scores obtained by stochastic inferences for a single example. Our empirical study shows that the accuracy and the score of a prediction are highly correlated with the variance of multiple stochastic inferences given by stochastic depth or dropout. Motivated by this observation, we design a novel variance-weighted confidence-integrated loss function that is composed of two cross-entropy loss terms with respect to ground-truth and uniform distribution, which are balanced by variance of stochastic prediction scores. The proposed loss function enables us to learn deep neural networks that predict confidence calibrated scores using a single inference. Our algorithm presents outstanding confidence calibration performance and improves classification accuracy when combined with two popular stochastic regularization techniques---stochastic depth and dropout---in multiple models and datasets; it alleviates overconfidence issue in deep neural networks significantly by training networks to achieve prediction accuracy proportional to confidence of prediction.