Optimal Regret Analysis of Thompson Sampling in Stochastic Multi-armed Bandit Problem with Multiple Plays

TL;DR

This paper introduces the multiple-play Thompson sampling (MP-TS) algorithm for stochastic multi-armed bandit problems with multiple selections, proving its optimal regret bounds and demonstrating superior empirical performance over existing algorithms.

Contribution

It extends Thompson sampling to multiple-play scenarios, providing the first computationally efficient algorithm with proven optimal regret bounds.

Findings

MP-TS achieves the optimal regret upper bound matching the lower bound.

Simulation results show MP-TS outperforms state-of-the-art algorithms.

A modified MP-TS improves empirical performance further.

Abstract

We discuss a multiple-play multi-armed bandit (MAB) problem in which several arms are selected at each round. Recently, Thompson sampling (TS), a randomized algorithm with a Bayesian spirit, has attracted much attention for its empirically excellent performance, and it is revealed to have an optimal regret bound in the standard single-play MAB problem. In this paper, we propose the multiple-play Thompson sampling (MP-TS) algorithm, an extension of TS to the multiple-play MAB problem, and discuss its regret analysis. We prove that MP-TS for binary rewards has the optimal regret upper bound that matches the regret lower bound provided by Anantharam et al. (1987). Therefore, MP-TS is the first computationally efficient algorithm with optimal regret. A set of computer simulations was also conducted, which compared MP-TS with state-of-the-art algorithms. We also propose a modification of MP-TS, which is shown to have better empirical performance.