Suboptimal Performance of the Bayes Optimal Algorithm in Frequentist Best Arm Identification

TL;DR

This paper shows that the Bayes optimal algorithm does not always achieve exponential decay in simple regret for best arm identification with normal rewards, challenging prior assumptions about Bayesian and frequentist equivalence.

Contribution

It reveals the suboptimal performance of the Bayes optimal algorithm in certain settings and introduces the concept of expected Bellman improvement for future research.

Findings

Bayes optimal algorithm does not guarantee exponential simple regret decay.

Contrasts with previous beliefs about Bayesian and frequentist asymptotic equivalence.

Introduces the concept of expected Bellman improvement for analyzing Bayesian algorithms.

Abstract

We consider the fixed-budget best arm identification problem with rewards following normal distributions. In this problem, the forecaster is given $K$ arms (or treatments) and $T$ time steps. The forecaster attempts to find the arm with the largest mean, via an adaptive experiment conducted using an algorithm. The algorithm's performance is evaluated by simple regret, reflecting the quality of the estimated best arm. While frequentist simple regret can decrease exponentially with respect to $T$, Bayesian simple regret decreases polynomially. This paper demonstrates that the Bayes optimal algorithm, which minimizes the Bayesian simple regret, does not yield an exponential decrease in simple regret under certain parameter settings. This contrasts with the numerous findings that suggest the asymptotic equivalence of Bayesian and frequentist approaches in fixed sampling regimes. Although the Bayes optimal algorithm is formulated as a recursive equation that is virtually impossible to compute exactly, we lay the groundwork for future research by introducing a novel concept termed the expected Bellman improvement.