On Efficient Bayesian Exploration in Model-Based Reinforcement Learning

TL;DR

This paper introduces a theoretically grounded, information-theoretic approach to data-efficient exploration in model-based reinforcement learning, utilizing epistemic uncertainty bonuses and practical approximations to improve sample efficiency.

Contribution

It provides formal guarantees for epistemic uncertainty-based exploration bonuses and proposes a new framework, PTS-BE, combining planning with information gain for improved exploration.

Findings

PTS-BE outperforms baselines in sparse reward environments

Theoretical guarantees for IG-based exploration bonuses are established

Practical approximations enable scalable implementation

Abstract

In this work, we address the challenge of data-efficient exploration in reinforcement learning by examining existing principled, information-theoretic approaches to intrinsic motivation. Specifically, we focus on a class of exploration bonuses that targets epistemic uncertainty rather than the aleatoric noise inherent in the environment. We prove that these bonuses naturally signal epistemic information gains and converge to zero once the agent becomes sufficiently certain about the environment's dynamics and rewards, thereby aligning exploration with genuine knowledge gaps. Our analysis provides formal guarantees for IG-based approaches, which previously lacked theoretical grounding. To enable practical use, we also discuss tractable approximations via sparse variational Gaussian Processes, Deep Kernels and Deep Ensemble models. We then outline a general framework - Predictive Trajectory Sampling with Bayesian Exploration (PTS-BE) - which integrates model-based planning with information-theoretic bonuses to achieve sample-efficient deep exploration. We empirically demonstrate that PTS-BE substantially outperforms other baselines across a variety of environments characterized by sparse rewards and/or purely exploratory tasks.