The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning

TL;DR

This paper investigates how the placement and number of Bayesian layers in neural networks affect the ability to model uncertainty, aiming to optimize active learning performance with fewer Bayesian components.

Contribution

It demonstrates that placing a few Bayesian layers near the output can effectively capture uncertainty, reducing computational costs compared to fully Bayesian networks.

Findings

Few Bayesian layers near output suffice for uncertainty estimation

Optimal layer placement improves active learning efficiency

Partial Bayesian networks outperform deterministic models in uncertainty modeling

Abstract

One of the main challenges of deep learning tools is their inability to capture model uncertainty. While Bayesian deep learning can be used to tackle the problem, Bayesian neural networks often require more time and computational power to train than deterministic networks. Our work explores whether fully Bayesian networks are needed to successfully capture model uncertainty. We vary the number and position of Bayesian layers in a network and compare their performance on active learning with the MNIST dataset. We found that we can fully capture the model uncertainty by using only a few Bayesian layers near the output of the network, combining the advantages of deterministic and Bayesian networks.