Approximate Top-$m$ Arm Identification with Heterogeneous Reward Variances

TL;DR

This paper investigates the sample complexity of identifying top-$m$ arms in a multi-armed bandit setting with heterogenous reward variances, providing tight bounds that incorporate variance heterogeneity and entropy measures.

Contribution

It introduces a new complexity characterization for top-$m$ arm identification considering reward variance heterogeneity, with matching upper and lower bounds.

Findings

Derived the worst-case sample complexity involving variance heterogeneity and entropy.

Proposed a divide-and-conquer algorithm achieving the upper bound.

Established a matching lower bound through dual formulation analysis.

Abstract

We study the effect of reward variance heterogeneity in the approximate top-$m$ arm identification setting. In this setting, the reward for the $i$-th arm follows a $\sigma^2_i$-sub-Gaussian distribution, and the agent needs to incorporate this knowledge to minimize the expected number of arm pulls to identify $m$ arms with the largest means within error $\epsilon$ out of the $n$ arms, with probability at least $1-\delta$. We show that the worst-case sample complexity of this problem is $$\Theta\left( \sum_{i =1}^n \frac{\sigma_i^2}{\epsilon^2} \ln\frac{1}{\delta} + \sum_{i \in G^{m}} \frac{\sigma_i^2}{\epsilon^2} \ln(m) + \sum_{j \in G^{l}} \frac{\sigma_j^2}{\epsilon^2} \text{Ent}(\sigma^2_{G^{r}}) \right),$$ where $G^{m}, G^{l}, G^{r}$ are certain specific subsets of the overall arm set $\{1, 2, \ldots, n\}$, and $\text{Ent}(\cdot)$ is an entropy-like function which measures the heterogeneity of the variance proxies. The upper bound of the complexity is obtained using a divide-and-conquer style algorithm, while the matching lower bound relies on the study of a dual formulation.