TSEB: More Efficient Thompson Sampling for Policy Learning

TL;DR

This paper introduces TSEB, an improved Thompson Sampling algorithm with adaptive exploration for more efficient policy learning, offering tighter PAC guarantees and empirical validation in simulated environments.

Contribution

The paper proposes TSEB, a novel Thompson Sampling algorithm with adaptive exploration bonus, enhancing PAC bounds and balancing exploration and regret in model-based learning.

Findings

TSEB achieves tighter PAC guarantees compared to existing methods.

The adaptive exploration bonus effectively encourages necessary exploration.

Empirical results demonstrate improved performance in simulated domains.

Abstract

In model-based solution approaches to the problem of learning in an unknown environment, exploring to learn the model parameters takes a toll on the regret. The optimal performance with respect to regret or PAC bounds is achievable, if the algorithm exploits with respect to reward or explores with respect to the model parameters, respectively. In this paper, we propose TSEB, a Thompson Sampling based algorithm with adaptive exploration bonus that aims to solve the problem with tighter PAC guarantees, while being cautious on the regret as well. The proposed approach maintains distributions over the model parameters which are successively refined with more experience. At any given time, the agent solves a model sampled from this distribution, and the sampled reward distribution is skewed by an exploration bonus in order to generate more informative exploration. The policy by solving is then used for generating more experience that helps in updating the posterior over the model parameters. We provide a detailed analysis of the PAC guarantees, and convergence of the proposed approach. We show that our adaptive exploration bonus encourages the additional exploration required for better PAC bounds on the algorithm. We provide empirical analysis on two different simulated domains.