Thompson Sampling for Real-Valued Combinatorial Pure Exploration of Multi-Armed Bandit

TL;DR

This paper introduces a new algorithm, GenTS-Explore, for the real-valued combinatorial pure exploration in multi-armed bandits, capable of handling exponentially large action sets and achieving near-optimal sample complexity.

Contribution

The paper proposes the first algorithm that efficiently handles exponentially large action sets in R-CPE-MAB and establishes a matching problem-dependent lower bound.

Findings

GenTS-Explore works with exponentially large action sets.

The algorithm achieves near-optimal sample complexity.

A new lower bound for R-CPE-MAB was derived.

Abstract

We study the real-valued combinatorial pure exploration of the multi-armed bandit (R-CPE-MAB) problem. In R-CPE-MAB, a player is given $d$ stochastic arms, and the reward of each arm $s\in\{1, \ldots, d\}$ follows an unknown distribution with mean $\mu_s$. In each time step, a player pulls a single arm and observes its reward. The player's goal is to identify the optimal \emph{action} $\boldsymbol{\pi}^{*} = \argmax_{\boldsymbol{\pi} \in \mathcal{A}} \boldsymbol{\mu}^{\top}\boldsymbol{\pi}$ from a finite-sized real-valued \emph{action set} $\mathcal{A}\subset \mathbb{R}^{d}$ with as few arm pulls as possible. Previous methods in the R-CPE-MAB assume that the size of the action set $\mathcal{A}$ is polynomial in $d$. We introduce an algorithm named the Generalized Thompson Sampling Explore (GenTS-Explore) algorithm, which is the first algorithm that can work even when the size of the action set is exponentially large in $d$. We also introduce a novel problem-dependent sample complexity lower bound of the R-CPE-MAB problem, and show that the GenTS-Explore algorithm achieves the optimal sample complexity up to a problem-dependent constant factor.