Thompson Sampling for Combinatorial Semi-Bandits

TL;DR

This paper applies Thompson sampling to combinatorial multi-armed bandits, providing improved regret bounds, analyzing the matroid setting, and demonstrating through experiments that TS outperforms existing algorithms.

Contribution

The paper introduces a refined analysis of Thompson sampling for CMAB, achieving tighter regret bounds, extends results to matroid bandits without independence assumptions, and highlights limitations of using approximation oracles.

Findings

Thompson sampling achieves better regret bounds than prior UCB-based methods.

In the matroid bandit setting, regret bounds match the theoretical lower bounds.

Experiments show Thompson sampling outperforms existing algorithms in practice.

Abstract

In this paper, we study the application of the Thompson sampling (TS) methodology to the stochastic combinatorial multi-armed bandit (CMAB) framework. We first analyze the standard TS algorithm for the general CMAB model when the outcome distributions of all the base arms are independent, and obtain a distribution-dependent regret bound of $O(m\log K_{\max}\log T / \Delta_{\min})$, where $m$ is the number of base arms, $K_{\max}$ is the size of the largest super arm, $T$ is the time horizon, and $\Delta_{\min}$ is the minimum gap between the expected reward of the optimal solution and any non-optimal solution. This regret upper bound is better than the $O(m(\log K_{\max})^2\log T / \Delta_{\min})$ bound in prior works. Moreover, our novel analysis techniques can help to tighten the regret bounds of other existing UCB-based policies (e.g., ESCB), as we improve the method of counting the cumulative regret. Then we consider the matroid bandit setting (a special class of CMAB model), where we could remove the independence assumption across arms and achieve a regret upper bound that matches the lower bound. Except for the regret upper bounds, we also point out that one cannot directly replace the exact offline oracle (which takes the parameters of an offline problem instance as input and outputs the exact best action under this instance) with an approximation oracle in TS algorithm for even the classical MAB problem. Finally, we use some experiments to show the comparison between regrets of TS and other existing algorithms, the experimental results show that TS outperforms existing baselines.