Near-Optimal BRL using Optimistic Local Transitions

TL;DR

This paper introduces BOLT, a heuristic Bayesian Reinforcement Learning algorithm that is optimistic about transitions, with theoretical near-optimality guarantees and promising experimental performance.

Contribution

The paper presents BOLT, a simple, deterministic heuristic for BRL with analyzed sample complexity and near-optimal Bayesian performance under certain conditions.

Findings

BOLT is near-optimal in Bayesian sense with high probability.

Experimental results show BOLT's effectiveness compared to previous methods.

BOLT offers a scalable alternative to complex BRL algorithms.

Abstract

Model-based Bayesian Reinforcement Learning (BRL) allows a found formalization of the problem of acting optimally while facing an unknown environment, i.e., avoiding the exploration-exploitation dilemma. However, algorithms explicitly addressing BRL suffer from such a combinatorial explosion that a large body of work relies on heuristic algorithms. This paper introduces BOLT, a simple and (almost) deterministic heuristic algorithm for BRL which is optimistic about the transition function. We analyze BOLT's sample complexity, and show that under certain parameters, the algorithm is near-optimal in the Bayesian sense with high probability. Then, experimental results highlight the key differences of this method compared to previous work.