Stochastic Multi-Armed Bandits with Limited Control Variates

TL;DR

This paper introduces UCB-LCV, a new bandit algorithm that leverages limited control variates to improve decision-making, outperforming existing methods in scenarios with partial auxiliary information.

Contribution

The paper proposes UCB-LCV, a novel algorithm that effectively combines reward estimates with limited control variates, extending bandit solutions to more realistic settings.

Findings

UCB-LCV outperforms existing algorithms in experiments.

UCB-NORMAL improves standard MAB with normal rewards.

Variants of UCB-LCV work with general distributions.

Abstract

Motivated by wireless networks where interference or channel state estimates provide partial insight into throughput, we study a variant of the classical stochastic multi-armed bandit problem in which the learner has limited access to auxiliary information. Recent work has shown that such auxiliary information, when available as control variates, can be used to get tighter confidence bounds, leading to lower regret. However, existing works assume that control variates are available in every round, which may not be realistic in several real-life scenarios. To address this, we propose UCB-LCV, an upper confidence bound (UCB) based algorithm that effectively combines the estimators obtained from rewards and control variates. When there is no control variate, UCB-LCV leads to a novel algorithm that we call UCB-NORMAL, outperforming its existing algorithms for the standard MAB setting with normally distributed rewards. Finally, we discuss variants of the proposed UCB-LCV that apply to general distributions and experimentally demonstrate that UCB-LCV outperforms existing bandit algorithms.