Stochastic Multi-Armed Bandits with Control Variates

TL;DR

This paper introduces a new multi-armed bandit algorithm that uses control variates to reduce variance in reward estimates, improving decision-making in applications with exogenous variables.

Contribution

It develops the UCB-CV algorithm that incorporates control variates for variance reduction and extends it to various distributions with resampling techniques.

Findings

UCB-CV achieves tighter confidence bounds and lower regret.

Theoretical regret bounds depend on reward-control variate correlation.

Experimental results confirm improved performance over traditional methods.

Abstract

This paper studies a new variant of the stochastic multi-armed bandits problem where auxiliary information about the arm rewards is available in the form of control variates. In many applications like queuing and wireless networks, the arm rewards are functions of some exogenous variables. The mean values of these variables are known a priori from historical data and can be used as control variates. Leveraging the theory of control variates, we obtain mean estimates with smaller variance and tighter confidence bounds. We develop an upper confidence bound based algorithm named UCB-CV and characterize the regret bounds in terms of the correlation between rewards and control variates when they follow a multivariate normal distribution. We also extend UCB-CV to other distributions using resampling methods like Jackknifing and Splitting. Experiments on synthetic problem instances validate performance guarantees of the proposed algorithms.