Bayesian Online Model Selection

TL;DR

This paper introduces a Bayesian algorithm for online model selection in stochastic bandits, providing theoretical regret guarantees and empirical validation across various settings, while exploring data sharing among learners.

Contribution

The paper proposes a novel Bayesian algorithm for online model selection in bandits with theoretical guarantees and empirical validation, addressing prior mis-specification issues.

Findings

Achieves $O(d^* M \sqrt{T} + \sqrt{(MT)})$ Bayesian regret bound.

Demonstrates competitive performance across multiple bandit settings.

Shows that sharing data among learners can mitigate prior mis-specification.

Abstract

Online model selection in Bayesian bandits raises a fundamental exploration challenge: When an environment instance is sampled from a prior distribution, how can we design an adaptive strategy that explores multiple bandit learners and competes with the best one in hindsight? We address this problem by introducing a new Bayesian algorithm for online model selection in stochastic bandits. We prove an oracle-style guarantee of $O\left( d^* M \sqrt{T} + \sqrt{(MT)} \right)$ on the Bayesian regret, where $M$ is the number of base learners, $d^*$ is the regret coefficient of the optimal base learner, and $T$ is the time horizon. We also validate our method empirically across a range of stochastic bandit settings, demonstrating performance that is competitive with the best base learner. Additionally, we study the effect of sharing data among base learners and its role in mitigating prior mis-specification.