Bag of Policies for Distributional Deep Exploration

TL;DR

This paper introduces Bag of Policies (BoP), a novel ensemble approach for distributional reinforcement learning that enhances exploration, robustness, and learning speed by maintaining multiple independent policy heads and leveraging posterior uncertainty.

Contribution

The paper proposes BoP, a general ensemble method for distributional RL that improves exploration and robustness by maintaining multiple independent heads and integrating posterior uncertainty analysis.

Findings

BoP improves learning speed on Atari games.

BoP enhances robustness and exploration efficiency.

Ensemble of distributional critics provides better uncertainty estimation.

Abstract

Efficient exploration in complex environments remains a major challenge for reinforcement learning (RL). Compared to previous Thompson sampling-inspired mechanisms that enable temporally extended exploration, i.e., deep exploration, we focus on deep exploration in distributional RL. We develop here a general purpose approach, Bag of Policies (BoP), that can be built on top of any return distribution estimator by maintaining a population of its copies. BoP consists of an ensemble of multiple heads that are updated independently. During training, each episode is controlled by only one of the heads and the collected state-action pairs are used to update all heads off-policy, leading to distinct learning signals for each head which diversify learning and behaviour. To test whether optimistic ensemble method can improve on distributional RL as did on scalar RL, by e.g. Bootstrapped DQN, we implement the BoP approach with a population of distributional actor-critics using Bayesian Distributional Policy Gradients (BDPG). The population thus approximates a posterior distribution of return distributions along with a posterior distribution of policies. Another benefit of building upon BDPG is that it allows to analyze global posterior uncertainty along with local curiosity bonus simultaneously for exploration. As BDPG is already an optimistic method, this pairing helps to investigate if optimism is accumulatable in distributional RL. Overall BoP results in greater robustness and speed during learning as demonstrated by our experimental results on ALE Atari games.