No Regrets for Learning the Prior in Bandits

TL;DR

This paper introduces ${\tt AdaTS}$, a Bayesian Thompson sampling algorithm that adaptively learns the prior distribution in bandit problems, leading to improved regret bounds and superior empirical performance.

Contribution

The paper presents a fully Bayesian, adaptive Thompson sampling algorithm for bandits that learns the prior distribution online, with theoretical regret bounds and demonstrated empirical advantages.

Findings

${\tt AdaTS}$ outperforms existing algorithms in experiments.

Theoretical bounds show small regret loss due to unknown prior.

Effective in real-world challenging bandit problems.

Abstract

We propose ${\tt AdaTS}$, a Thompson sampling algorithm that adapts sequentially to bandit tasks that it interacts with. The key idea in ${\tt AdaTS}$ is to adapt to an unknown task prior distribution by maintaining a distribution over its parameters. When solving a bandit task, that uncertainty is marginalized out and properly accounted for. ${\tt AdaTS}$ is a fully-Bayesian algorithm that can be implemented efficiently in several classes of bandit problems. We derive upper bounds on its Bayes regret that quantify the loss due to not knowing the task prior, and show that it is small. Our theory is supported by experiments, where ${\tt AdaTS}$ outperforms prior algorithms and works well even in challenging real-world problems.