Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

TL;DR

This paper reveals that batch normalization in deep networks can be viewed as approximate Bayesian inference, enabling uncertainty estimation without altering standard training procedures, validated through extensive empirical experiments.

Contribution

It establishes a theoretical link between batch normalization and Bayesian inference, allowing uncertainty estimation in conventional deep networks.

Findings

Outperforms baseline methods with statistical significance

Provides meaningful uncertainty estimates in various tasks

Achieves competitive performance with recent Bayesian approaches

Abstract

We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models. We further demonstrate that this finding allows us to make meaningful estimates of the model uncertainty using conventional architectures, without modifications to the network or the training procedure. Our approach is thoroughly validated by measuring the quality of uncertainty in a series of empirical experiments on different tasks. It outperforms baselines with strong statistical significance, and displays competitive performance with recent Bayesian approaches.