UCB for Large-Scale Pure Exploration: Beyond Sub-Gaussianity

TL;DR

This paper extends the theoretical understanding of UCB algorithms for large-scale pure exploration problems beyond sub-Gaussian distributions, demonstrating their optimality in heavy-tailed and non-sub-Gaussian settings.

Contribution

It introduces a meta-UCB algorithm for non-sub-Gaussian pure exploration and proves its sample optimality under broad distributional assumptions.

Findings

Meta-UCB achieves sample optimality in non-sub-Gaussian settings.

UCB algorithms are effective for large-scale pure exploration beyond sub-Gaussian assumptions.

Numerical experiments validate theoretical results and compare behaviors.

Abstract

Selecting the best alternative from a finite set represents a broad class of pure exploration problems. Traditional approaches to pure exploration have predominantly relied on Gaussian or sub-Gaussian assumptions on the performance distributions of all alternatives, which limit their applicability to non-sub-Gaussian especially heavy-tailed problems. The need to move beyond sub-Gaussianity may become even more critical in large-scale problems, which tend to be especially sensitive to distributional specifications. In this paper, motivated by the widespread use of upper confidence bound (UCB) algorithms in pure exploration and beyond, we investigate their performance in the large-scale, non-sub-Gaussian settings. We consider the simplest category of UCB algorithms, where the UCB value for each alternative is defined as the sample mean plus an exploration bonus that depends only on its own sample size. We abstract this into a meta-UCB algorithm and propose letting it select the alternative with the largest sample size as the best upon stopping. For this meta-UCB algorithm, we first derive a distribution-free lower bound on the probability of correct selection. Building on this bound, we analyze two general non-sub-Gaussian scenarios: (1) all alternatives follow a common location-scale structure and have bounded variance; and (2) when such a structure does not hold, each alternative has a bounded absolute moment of order $q > 3$. In both settings, we show that the meta-UCB algorithm and therefore a broad class of UCB algorithms can achieve the sample optimality. These results demonstrate the applicability of UCB algorithms for solving large-scale pure exploration problems with non-sub-Gaussian distributions. Numerical experiments support our results and provide additional insights into the comparative behaviors of UCB algorithms within and beyond our meta-UCB framework.