Combinatorial Bandits without Total Order for Arms

TL;DR

This paper introduces a new combinatorial bandit model with set-dependent rewards and no total order among arms, proposing an UCB algorithm with near-optimal regret bounds and broad empirical validation.

Contribution

The paper formulates a novel reward model for combinatorial bandits without total order and develops a near-optimal UCB algorithm with rigorous regret analysis.

Findings

Achieves $O(k^2 n rac{ ext{log} T}{ ext{epsilon}})$ regret bound

Achieves $O(k^2 ext{sqrt}{n T ext{log} T})$ regret bound

Empirical results demonstrate broad applicability

Abstract

We consider the combinatorial bandits problem, where at each time step, the online learner selects a size-$k$ subset $s$ from the arms set $\mathcal{A}$, where $\left|\mathcal{A}\right| = n$, and observes a stochastic reward of each arm in the selected set $s$. The goal of the online learner is to minimize the regret, induced by not selecting $s^*$ which maximizes the expected total reward. Specifically, we focus on a challenging setting where 1) the reward distribution of an arm depends on the set $s$ it is part of, and crucially 2) there is \textit{no total order} for the arms in $\mathcal{A}$. In this paper, we formally present a reward model that captures set-dependent reward distribution and assumes no total order for arms. Correspondingly, we propose an Upper Confidence Bound (UCB) algorithm that maintains UCB for each individual arm and selects the arms with top-$k$ UCB. We develop a novel regret analysis and show an $O\left(\frac{k^2 n \log T}{\epsilon}\right)$ gap-dependent regret bound as well as an $O\left(k^2\sqrt{n T \log T}\right)$ gap-independent regret bound. We also provide a lower bound for the proposed reward model, which shows our proposed algorithm is near-optimal for any constant $k$. Empirical results on various reward models demonstrate the broad applicability of our algorithm.