BOTS: Batch Bayesian Optimization of Extended Thompson Sampling for Severely Episode-Limited RL Settings

TL;DR

This paper introduces BOTS, a batch Bayesian optimization method that extends Thompson sampling for reinforcement learning in severely episode-limited settings, improving policy performance with fewer episodes.

Contribution

It develops an extended Thompson sampling approach with learned action bias terms via batch Bayesian optimization, enabling better policies in limited-episode RL scenarios.

Findings

Outperforms standard Thompson sampling in total return

Requires fewer episodes than value function and policy gradient methods

Effective in behavioral dynamics simulation environment

Abstract

In settings where the application of reinforcement learning (RL) requires running real-world trials, including the optimization of adaptive health interventions, the number of episodes available for learning can be severely limited due to cost or time constraints. In this setting, the bias-variance trade-off of contextual bandit methods can be significantly better than that of more complex full RL methods. However, Thompson sampling bandits are limited to selecting actions based on distributions of immediate rewards. In this paper, we extend the linear Thompson sampling bandit to select actions based on a state-action utility function consisting of the Thompson sampler's estimate of the expected immediate reward combined with an action bias term. We use batch Bayesian optimization over episodes to learn the action bias terms with the goal of maximizing the expected return of the extended Thompson sampler. The proposed approach is able to learn optimal policies for a strictly broader class of Markov decision processes (MDPs) than standard Thompson sampling. Using an adaptive intervention simulation environment that captures key aspects of behavioral dynamics, we show that the proposed method can significantly out-perform standard Thompson sampling in terms of total return, while requiring significantly fewer episodes than standard value function and policy gradient methods.