Model-based Lifelong Reinforcement Learning with Bayesian Exploration

TL;DR

This paper introduces a hierarchical Bayesian model-based lifelong reinforcement learning method that improves sample efficiency and transfer learning across related tasks, combining Bayesian exploration with deep RL techniques.

Contribution

It presents a novel Bayesian lifelong RL framework with theoretical analysis and scalable algorithms for continuous domains, demonstrating superior transfer performance.

Findings

Enhanced sample efficiency in lifelong RL tasks.

Effective backward and forward transfer in complex domains.

Scalable Bayesian RL algorithm with deep learning integration.

Abstract

We propose a model-based lifelong reinforcement-learning approach that estimates a hierarchical Bayesian posterior distilling the common structure shared across different tasks. The learned posterior combined with a sample-based Bayesian exploration procedure increases the sample efficiency of learning across a family of related tasks. We first derive an analysis of the relationship between the sample complexity and the initialization quality of the posterior in the finite MDP setting. We next scale the approach to continuous-state domains by introducing a Variational Bayesian Lifelong Reinforcement Learning algorithm that can be combined with recent model-based deep RL methods, and that exhibits backward transfer. Experimental results on several challenging domains show that our algorithms achieve both better forward and backward transfer performance than state-of-the-art lifelong RL methods.