On Explore-Then-Commit Strategies

TL;DR

This paper demonstrates that explore-then-commit strategies are inherently suboptimal for two-armed Gaussian bandit problems and proposes fully sequential strategies that achieve asymptotic and minimax optimality with finite-time guarantees.

Contribution

It shows the suboptimality of explore-then-commit strategies and introduces fully sequential strategies with finite-time regret bounds that are asymptotically and minimax optimal.

Findings

Explore-then-commit strategies are necessarily suboptimal.

Proposed fully sequential strategies achieve asymptotic optimality.

Refined deviation inequalities enable finite-time regret guarantees.

Abstract

We study the problem of minimising regret in two-armed bandit problems with Gaussian rewards. Our objective is to use this simple setting to illustrate that strategies based on an exploration phase (up to a stopping time) followed by exploitation are necessarily suboptimal. The results hold regardless of whether or not the difference in means between the two arms is known. Besides the main message, we also refine existing deviation inequalities, which allow us to design fully sequential strategies with finite-time regret guarantees that are (a) asymptotically optimal as the horizon grows and (b) order-optimal in the minimax sense. Furthermore we provide empirical evidence that the theory also holds in practice and discuss extensions to non-gaussian and multiple-armed case.