Combinatorial Multi-Armed Bandit with General Reward Functions

TL;DR

This paper introduces a new algorithm for stochastic combinatorial multi-armed bandits with general nonlinear reward functions, achieving near-optimal regret bounds and enabling solutions for complex problems like $K$-MAX.

Contribution

The paper proposes the SDCB algorithm that estimates entire distributions for complex reward functions, extending bandit techniques beyond mean-based methods.

Findings

Achieves $O(\log T)$ distribution-dependent regret.

Achieves $ ilde{O}(\sqrt T)$ distribution-independent regret.

Provides a polynomial-time approximation scheme for $K$-MAX.

Abstract

In this paper, we study the stochastic combinatorial multi-armed bandit (CMAB) framework that allows a general nonlinear reward function, whose expected value may not depend only on the means of the input random variables but possibly on the entire distributions of these variables. Our framework enables a much larger class of reward functions such as the $\max()$ function and nonlinear utility functions. Existing techniques relying on accurate estimations of the means of random variables, such as the upper confidence bound (UCB) technique, do not work directly on these functions. We propose a new algorithm called stochastically dominant confidence bound (SDCB), which estimates the distributions of underlying random variables and their stochastically dominant confidence bounds. We prove that SDCB can achieve $O(\log{T})$ distribution-dependent regret and $\tilde{O}(\sqrt{T})$ distribution-independent regret, where $T$ is the time horizon. We apply our results to the $K$-MAX problem and expected utility maximization problems. In particular, for $K$-MAX, we provide the first polynomial-time approximation scheme (PTAS) for its offline problem, and give the first $\tilde{O}(\sqrt T)$ bound on the $(1-\epsilon)$-approximation regret of its online problem, for any $\epsilon>0$.