Planning to the Information Horizon of BAMDPs via Epistemic State Abstraction

TL;DR

This paper introduces a complexity measure for BAMDP planning that captures the difficulty of information gathering, and proposes an abstraction-based approach to enable more efficient approximate planning in Bayesian reinforcement learning.

Contribution

It defines a new complexity measure for BAMDPs, and develops an abstraction method that reduces complexity for more tractable approximate planning.

Findings

The complexity measure highlights worst-case information acquisition difficulty.

An intractable exact planning algorithm demonstrates the measure's significance.

A state abstraction reduces complexity, enabling approximate planning.

Abstract

The Bayes-Adaptive Markov Decision Process (BAMDP) formalism pursues the Bayes-optimal solution to the exploration-exploitation trade-off in reinforcement learning. As the computation of exact solutions to Bayesian reinforcement-learning problems is intractable, much of the literature has focused on developing suitable approximation algorithms. In this work, before diving into algorithm design, we first define, under mild structural assumptions, a complexity measure for BAMDP planning. As efficient exploration in BAMDPs hinges upon the judicious acquisition of information, our complexity measure highlights the worst-case difficulty of gathering information and exhausting epistemic uncertainty. To illustrate its significance, we establish a computationally-intractable, exact planning algorithm that takes advantage of this measure to show more efficient planning. We then conclude by introducing a specific form of state abstraction with the potential to reduce BAMDP complexity and gives rise to a computationally-tractable, approximate planning algorithm.