Performance Measurement for Deep Bayesian Neural Network

TL;DR

This paper introduces new metrics for evaluating deep Bayesian neural networks, focusing on calibration, data rejection, and uncertainty divergence to provide more detailed performance insights.

Contribution

It proposes specific performance measurement criteria and metrics for Bayesian neural networks, addressing a gap in existing evaluation methods.

Findings

Proposed calibration measurement for Bayesian models

Introduced data rejection ability as a performance metric

Analyzed uncertainty divergence within and across distributions

Abstract

Deep Bayesian neural network has aroused a great attention in recent years since it combines the benefits of deep neural network and probability theory. Because of this, the network can make predictions and quantify the uncertainty of the predictions at the same time, which is important in many life-threatening areas. However, most of the recent researches are mainly focusing on making the Bayesian neural network easier to train, and proposing methods to estimate the uncertainty. I notice there are very few works that properly discuss the ways to measure the performance of the Bayesian neural network. Although accuracy and average uncertainty are commonly used for now, they are too general to provide any insight information about the model. In this paper, we would like to introduce more specific criteria and propose several metrics to measure the model performance from different perspectives, which include model calibration measurement, data rejection ability and uncertainty divergence for samples from the same and different distributions.