VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

TL;DR

VariBAD introduces a meta-learning approach for approximate Bayesian inference in deep reinforcement learning, enabling better exploration and higher returns in complex environments by incorporating task uncertainty during decision-making.

Contribution

It presents variBAD, a novel method that combines variational Bayes with meta-learning to perform approximate inference for Bayes-adaptive deep RL, improving exploration and performance.

Findings

Outperforms existing methods in MuJoCo meta-RL tasks

Enables structured online exploration based on task uncertainty

Achieves higher online returns in benchmark environments

Abstract

Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We further evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher online return than existing methods.