A Bayesian Neural Network based on Dropout Regulation

TL;DR

This paper introduces Dropout Regulation, a novel method for automatically adjusting dropout rates in Bayesian Neural Networks during training, enhancing uncertainty estimation while maintaining simplicity.

Contribution

The paper proposes Dropout Regulation, a new automated approach to optimize dropout rates in BNNs, improving uncertainty estimation accuracy.

Findings

Comparable uncertainty estimation to state-of-the-art methods

Simple to implement and integrate into existing models

Effective automatic adjustment of dropout rates during training

Abstract

Bayesian Neural Networks (BNN) have recently emerged in the Deep Learning world for dealing with uncertainty estimation in classification tasks, and are used in many application domains such as astrophysics, autonomous driving...BNN assume a prior over the weights of a neural network instead of point estimates, enabling in this way the estimation of both aleatoric and epistemic uncertainty of the model prediction.Moreover, a particular type of BNN, namely MC Dropout, assumes a Bernoulli distribution on the weights by using Dropout.Several attempts to optimize the dropout rate exist, e.g. using a variational approach.In this paper, we present a new method called "Dropout Regulation" (DR), which consists of automatically adjusting the dropout rate during training using a controller as used in automation.DR allows for a precise estimation of the uncertainty which is comparable to the state-of-the-art while remaining simple to implement.