Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning

TL;DR

This paper introduces a modified ensemble method for neural networks that performs Bayesian inference, converges to Gaussian Processes, and improves exploration stability in reinforcement learning.

Contribution

It proposes a minor modification to neural network ensembles that enables Bayesian inference and convergence to Gaussian Processes, with practical reinforcement learning applications.

Findings

Ensembles with the modification perform Bayesian inference.

Anchored ensembles lead to more stable exploration in reinforcement learning.

The method converges to Gaussian Processes as ensemble size increases.

Abstract

The use of ensembles of neural networks (NNs) for the quantification of predictive uncertainty is widespread. However, the current justification is intuitive rather than analytical. This work proposes one minor modification to the normal ensembling methodology, which we prove allows the ensemble to perform Bayesian inference, hence converging to the corresponding Gaussian Process as both the total number of NNs, and the size of each, tend to infinity. This working paper provides early-stage results in a reinforcement learning setting, analysing the practicality of the technique for an ensemble of small, finite number. Using the uncertainty estimates produced by anchored ensembles to govern the exploration-exploitation process results in steadier, more stable learning.