The Bayesian Confidence (BACON) Estimator for Deep Neural Networks

TL;DR

The paper proposes BACON, a Bayesian confidence estimator for deep neural networks that improves probability calibration over Softmax, especially on imbalanced datasets, by using a geometric model and Bayesian inference.

Contribution

It introduces BACON, a novel Bayesian confidence estimator that enhances probability calibration in deep neural networks beyond traditional Softmax outputs.

Findings

BACON outperforms Softmax in ECE and ACE calibration errors on CIFAR-10.

BACON provides better probability estimates for imbalanced test sets.

Improved calibration is observed at various network accuracies, except for very high accuracy edge cases.

Abstract

This paper introduces the Bayesian Confidence Estimator (BACON) for deep neural networks. Current practice of interpreting Softmax values in the output layer as probabilities of outcomes is prone to extreme predictions of class probability. In this work we extend Waagen's method of representing the terminal layers with a geometric model, where the probability associated with an output vector is estimated with Bayes' Rule using validation data to provide likelihood and normalization values. This estimator provides superior ECE and ACE calibration error compared to Softmax for ResNet-18 at 85% network accuracy, and EfficientNet-B0 at 95% network accuracy, on the CIFAR-10 dataset with an imbalanced test set, except for very high accuracy edge cases. In addition, when using the ACE metric, BACON demonstrated improved calibration error when estimating probabilities for the imbalanced test set when using actual class distribution fractions.