Benchmarking for Bayesian Reinforcement Learning

TL;DR

This paper introduces a comprehensive benchmarking methodology and open-source library for Bayesian Reinforcement Learning algorithms, enabling rigorous comparison across diverse MDPs and prior distributions.

Contribution

It provides a new benchmarking framework and open-source tools for evaluating BRL algorithms, addressing the lack of extensive comparative studies in the field.

Findings

Comparison of seven state-of-the-art RL algorithms on benchmark problems.

Analysis of computation time requirements for non-anytime algorithms.

Discussion of algorithm performance across different prior distributions.

Abstract

In the Bayesian Reinforcement Learning (BRL) setting, agents try to maximise the collected rewards while interacting with their environment while using some prior knowledge that is accessed beforehand. Many BRL algorithms have already been proposed, but even though a few toy examples exist in the literature, there are still no extensive or rigorous benchmarks to compare them. The paper addresses this problem, and provides a new BRL comparison methodology along with the corresponding open source library. In this methodology, a comparison criterion that measures the performance of algorithms on large sets of Markov Decision Processes (MDPs) drawn from some probability distributions is defined. In order to enable the comparison of non-anytime algorithms, our methodology also includes a detailed analysis of the computation time requirement of each algorithm. Our library is released with all source code and documentation: it includes three test problems, each of which has two different prior distributions, and seven state-of-the-art RL algorithms. Finally, our library is illustrated by comparing all the available algorithms and the results are discussed.