Optimal Exploitation of Clustering and History Information in Multi-Armed Bandit

TL;DR

This paper introduces algorithms for multi-armed bandit problems that leverage historical data and pre-clustered arms, improving exploration efficiency and regret bounds, with a robust META algorithm balancing different information sources.

Contribution

The paper proposes the META algorithm that adaptively combines clustering and historical data to optimize exploration in multi-armed bandits, validated by theoretical analysis and experiments.

Findings

META outperforms individual algorithms when clustering quality is high.

The algorithms achieve improved regret bounds with pre-clustering and historical data.

Experimental results confirm robustness and practical effectiveness on real datasets.

Abstract

We consider the stochastic multi-armed bandit problem and the contextual bandit problem with historical observations and pre-clustered arms. The historical observations can contain any number of instances for each arm, and the pre-clustering information is a fixed clustering of arms provided as part of the input. We develop a variety of algorithms which incorporate this offline information effectively during the online exploration phase and derive their regret bounds. In particular, we develop the META algorithm which effectively hedges between two other algorithms: one which uses both historical observations and clustering, and another which uses only the historical observations. The former outperforms the latter when the clustering quality is good, and vice-versa. Extensive experiments on synthetic and real world datasets on Warafin drug dosage and web server selection for latency minimization validate our theoretical insights and demonstrate that META is a robust strategy for optimally exploiting the pre-clustering information.

Equations

$$E[R(T)]\stackrel{\text{def}}{=}E\left[\sum _{t=1}^T\left({\theta }^{\ast }-{\theta }_{k(t)}\right)\right]=\sum _k{\Delta }_{k}E[n_k(T)]$$

$${\mathbf{H}}_k(t)={\mathbf{H}}_k(t-1)+{\vec{x}}_t{\vec{x}}_t^{\top },$$

$$b_k^h(t)=b_k^h(t-1)+r_k^h(t){\vec{x}}_t,$$

$${\hat{\theta }}_k^h(t)=\frac{{\sum }_{t^{\prime }=1}^{H_k}{r}_k^h(t^{\prime })+{\sum }_{t^{\prime }=1}^t{r}_k(t^{\prime })}{H_k+n_k(t)}$$

$${\mathrm{HUCBC}}_k(t)={\hat{\theta }}_k^h(t)+\sqrt{\frac{2\log (t+H_k)}{n_k(t)+H_k}}$$

$${\hat{\theta }}_i^{hc}(t)=\frac{{\sum }_{t^{\prime }=1}^{H_i^c}{r}_i^{hc}(t^{\prime })+{\sum }_{t^{\prime }=1}^t{r}_i^c(t^{\prime })}{H_i^c+n_i^c(t)}$$

$${\mathrm{HUCBC}}_i^c(t)={\hat{\theta }}_i^{hc}(t)+\sqrt{\frac{2\log (t+H_i^c)}{n_i^c(t)+H_i^c}}$$

$$\sum _{i\ne i^{\ast }}\left(1+{\ell }_i^c+\frac{{\pi }^2(1+6H_i^c)}{6(2H_i^c+1{)}^2}+\frac{{\pi }^2(1+6H_{i^{\ast }}^c)}{6(2H_{i^{\ast }}^c+1{)}^2}\right)$$

$$+\sum _{k|c(k)=i^{\ast }}\left(1+{\ell }_k+\frac{{\pi }^2(1+6H_k)}{6(2H_k+1{)}^2}+\frac{{\pi }^2(1+6H_{k^{\ast }})}{6(2H_{k^{\ast }}+1{)}^2}\right)$$

$$E[n_k(T)]\le 1+{\ell }_k+\frac{{\pi }^2(1+6H_k)}{6(2H_k+1{)}^2}+\frac{{\pi }^2(1+6H_{k^{\ast }})}{6(2H_{k^{\ast }}+1{)}^2}$$

$$z_i^c(\ell )=\{\begin{array}{ll}0& \text{if }\ell =0\\ z_i^c(\ell -1)+r_i^{hc}(\ell )-{\theta }_{j^h(\ell )}& \text{if }1\le \ell \le H_i^c\\ z_i^c(\ell -1)+r_i^c(\ell -H_i^c)-{\theta }_{j(\ell -H_i^c)}& \text{if }\ell >H_i^c\end{array}$$

$$E[z_i^c(\ell )|z_i^c(\ell -1)=z]=z+E[r_i^{hc}(\ell )-{\theta }_{j^h(\ell )}|z_i^c(\ell -1)=z]$$

$$=z+\sum _{j|c(j)=i}\mathrm{Pr}[j^h(\ell )=j|z_i^c(\ell -1)=z]$$

$$E[r_j^h(\ell )-{\theta }_j|z_i^c(\ell -1)=z\bigwedge j^h(\ell )=j]=z$$

$$\mathrm{Pr}[|z_i^c(\ell )|\ge v]=\mathrm{Pr}[|z_i^c(\ell )-z_i^c(0)|\ge v]\le 2e^{-\frac{v^2}{2\ell }}$$

$$\text{(refeqn:azumaHUCBC)}⟹\forall t:\mathrm{Pr}\left[HUCB{C}_{i^{\ast }}^c(t)\le l_{i^{\ast }}\right]$$

$$=\mathrm{Pr}\left[(n_{i^{\ast }}^c(t)+H_{i^{\ast }}^c)HUCB{C}_{i^{\ast }}^c(t)\le (n_{i^{\ast }}^c(t)+H_{i^{\ast }}^c)l_{i^{\ast }}\right]$$

$$\displaystyle\leq\Pr\Bigg{[}\sum_{\ell=1}^{H_{i^{*}}^{c}}r_{i^{*}}^{hc}(\ell)+\sum_{\ell=1}^{n_{i^{*}}^{c}(t)}r_{i^{*}}^{c}(\ell)+\sqrt{2(n_{i^{*}}^{c}(t)+H_{i^{*}}^{c})\log(t+H_{i^{*}}^{c})}$$

$$\displaystyle\leq\sum_{k|c(k)=i^{*}}(n_{k}(t)+H_{k})\theta_{j}\Bigg{]}$$

$$=\mathrm{Pr}\left[z_{i^{\ast }}^c(n_{i^{\ast }}^c(t)+H_{i^{\ast }}^c)\le -\sqrt{2(n_{i^{\ast }}^c(t)+H_{i^{\ast }}^c)\log (t+H_{i^{\ast }}^c)}\right]$$

$$\le \mathrm{Pr}\left[|z_{i^{\ast }}^c(n_{i^{\ast }}^c(t)+H_{i^{\ast }}^c)|\ge \sqrt{2(n_{i^{\ast }}^c(t)+H_{i^{\ast }}^c)\log (t+H_{i^{\ast }}^c)}\right]$$

$$\stackrel{\text{(refeqn:azumaHUCBC)}}{\le }\frac{2}{(t+H_{i^{\ast }}^c{)}^2}$$

$$\displaystyle\text{(\ref{eqn:azumaHUCBC})}\implies\Pr\left[HUCBC^{c}_{i}(t)\geq u_{i}+\delta_{i}\bigg{|}n_{i}^{c}(t)=\gamma\right]$$

$$\displaystyle=\Pr\left[(\gamma+H_{i}^{c})HUCBC^{c}_{i}(t)\geq(\gamma+H_{i}^{c})(u_{i}+\delta_{i})\bigg{|}n_{i}^{c}(t)=\gamma\right]\leq$$

$$\displaystyle\Pr\Bigg{[}\sum_{\ell=1}^{H_{i}^{c}}r_{i}^{hc}(\ell)+\sum_{\ell=1}^{\gamma}r_{i}^{c}(\ell)+\sqrt{2(\gamma+H_{i}^{c})\log(t+H_{i}^{c})}$$

$$\displaystyle\geq\sum_{k|c(k)=i}(H_{k}+n_{k}(t))\theta_{k}+(\gamma+H_{i}^{c})\delta_{i}\bigg{|}n_{i}^{c}(t)=\gamma\Bigg{]}$$

$$\displaystyle=\Pr\Bigg{[}z_{i}^{c}(\gamma+H_{i}^{c})\geq(\gamma+H_{i}^{c})\delta_{i}-\sqrt{2(\gamma+H_{i}^{c})\log(t+H_{i}^{c})}$$

$$\displaystyle\bigg{|}n_{i}^{c}(t)=\gamma\Bigg{]}\leq\Pr\Bigg{[}|z_{i}^{c}(\gamma+H_{i}^{c})|\geq\sqrt{2(\gamma+H_{i}^{c})\log(t+H_{i}^{c})}$$

$$\displaystyle\bigg{|}n_{i}^{c}(t)=\gamma\Bigg{]}\stackrel{{\scriptstyle\text{(\ref{eqn:azumaHUCBC})}}}{{\leq}}\frac{2}{(t+H_{i}^{c})^{2}}$$

$$\displaystyle\Pr\left[HUCBC^{c}_{i}(t)\geq u_{i}+\delta_{i}\bigg{|}n_{i}^{c}(t)\geq\ell_{i}^{c}\right]\leq\frac{2}{(t+H_{i}^{c})^{2}}$$

$$\mathcal{A}⟹(\mathcal{A}\wedge \mathcal{E})\vee (\mathcal{A}\wedge \overline{\mathcal{E}})⟹(\mathcal{A}\wedge \mathcal{E})\vee ((\mathcal{B}\vee \mathcal{C}\vee \mathcal{D})\wedge \overline{\mathcal{E}})$$

$$\text{Hence, }\mathrm{Pr}[\mathcal{A}]\le \mathrm{Pr}[\mathcal{B}\wedge \overline{\mathcal{E}}]+\mathrm{Pr}[\mathcal{C}\wedge \overline{\mathcal{E}}]+\mathrm{Pr}[\mathcal{A}\wedge \mathcal{E}]+\mathrm{Pr}[\mathcal{D}\wedge \overline{\mathcal{E}}]$$

$$\le \mathrm{Pr}[\mathcal{B}]+\mathrm{Pr}[\mathcal{C}]+\mathrm{Pr}[\mathcal{A}|\mathcal{E}]+\mathrm{Pr}[\mathcal{D}|\overline{\mathcal{E}}]$$

$$E[n_i^c(T)]=\sum _{t=1}^T\mathrm{Pr}[i(t)=i]\le 1+\sum _{t=1}^T\mathrm{Pr}\left[HUCB{C}_{i^{\ast }}^c(t)\le l_{i^{\ast }}\right]$$

$$\displaystyle+\sum_{t=1}^{T}\Pr\left[i(t)=i\bigg{|}n_{i}^{c}(t)<\ell_{i}^{c}\right]$$

$$\displaystyle+\sum_{t=1}^{T}\Pr\left[HUCBC^{c}_{i}(t)\geq u_{i}+\delta_{i}\bigg{|}n_{i}^{c}(t)\geq\ell_{i}^{c}\right]$$

$$\le 1+{\ell }_i^c+\sum _{t=1}^T\mathrm{Pr}\left[HUCB{C}_{i^{\ast }}^c(t)\le l_{i^{\ast }}\right]$$

$$\displaystyle+\sum_{t=1}^{T}\Pr\left[HUCBC^{c}_{i}(t)\geq u_{i}+\delta_{i}\bigg{|}n_{i}^{c}(t)\geq\ell_{i}^{c}\right]$$

$$\stackrel{\text{(refeqn:condii*) and (refeqn:uncondii)}}{\le }1+{\ell }_i^c+\sum _{t=1}^T\frac{2}{(t+H_i^c{)}^2}+\sum _{t=1}^T\frac{2}{(t+H_{i^{\ast }}^c{)}^2}$$

$$\le 1+{\ell }_i^c+\frac{{\pi }^2(1+6H_i^c)}{6(2H_i^c+1{)}^2}+\frac{{\pi }^2(1+6H_{i^{\ast }}^c)}{6(2H_{i^{\ast }}^c+1{)}^2}$$

$$\mathbb{E}[R(T)]\le \sum _{i\ne i^{\ast }}\underset{k|c(k)=i}{\max }\left(\frac{16r\log T}{({\Delta }_k/2{)}^2}+2s+\frac{\pi }{3}\right)$$

$$+\sum _{k|c(k)=i^{\ast }}\left(1+{\ell }_k+\frac{{\pi }^2(1+6H_k)}{6(2H_k+1{)}^2}+\frac{{\pi }^2(1+6H_{k^{\ast }})}{6(2H_{k^{\ast }}+1{)}^2}\right)$$

Full text

1 Introduction

Many sequential decision problems ranging from clinical trials to online ad placement can be modeled as multi-armed bandit problems (classical bandit) Li et al. (2010a); Tang et al. (2013). At each time step, the algorithm chooses one of several possible actions and observes its reward with the goal of maximizing the cumulative reward over time. A useful extension to classical bandit is the contextual multi-arm bandit problem, where before choosing an arm, the algorithm observes a context vector in each iteration Langford and Zhang (2007). In the conventional formulation of these problems, arms are assumed to be unrelated to each other and no prior knowledge about the arms exist. However, applications can often provide historical observations about arms and also similarity information between them prior to the start of the online exploration phase. In this work, we assume that similarity between the arms is given in the form of a fixed pre-clustering of arms and we design algorithms which exploit the historical information and the clustering information opportunistically. In particular, we seek algorithms which satisfy the following property: if the quality of clustering is good, the algorithm should quickly learn to use this information aggressively during online exploration; however, if the quality of clustering is poor, the algorithm should ignore this information. We note at the outset that it is possible to consider alternative formulations of the bandit problem in the presence of history and clustering information. For instance, one alternative is to use clustering metrics such as Dunn or Dunn-like indices Liu et al. (2010) to decide if the clustering information should be used during online exploration. However, in general, cluster validation metrics are tightly coupled with specific classes of clustering algorithms (e.g., distance vs. density based clustering); hence, we focus on bandit algorithms which are agnostic to specific clustering metrics and work with any given clustering of the arms. Other alternatives include the design of algorithms for optimal pre-clustering of arms prior to online exploration, as well as incrementally modifying the clustering of the arms during online exploration. These are both valuable refinements to the problem we study but are beyond the scope of the current work. In this paper, we focus on the problem of optimal exploitation of the clustering information given as part of the input. A real-world motivation for our work is the problem of dosing the drug Warfarin Sharabiani et al. (2015). Correctly dosing Warfarin is a significant challenge since it is dependent on the patient’s clinical, demographic and genetic information. In the contextual bandit setting, this information can be modeled as the context vector and the bandit arms model the various dosage levels. The dosage levels are further grouped into distinct clusters (15 arms and 3 clusters in our dataset). Historical treatment responses for various dosage levels as well as clustering information derived from medical domain knowledge is available as part this problem instance. Another motivating application is web server selection for latency minimization in content distribution networks (CDNs) Krishnamurthy, Wills, and Zhang (2001). A CDN can choose mirrored content from several distributed web servers. The latencies of these servers are correlated and vary widely due to geography and network traffic conditions. This domain can be modeled as the classical bandit problem with the bandit arms representing various web servers, and latency being the negative reward. The web servers are further grouped into clusters based on their historical latencies (700 arms and 5 clusters in our dataset). These problem domains share the characteristics of the availability of historical observations to seed the online phase and grouping of arms into clusters either through domain knowledge or through the use of clustering algorithms.

2 Related Work

Both the classical multi-armed bandit and the contextual bandit problems have been studied extensively along with their variants Lai and Robbins (1985); Auer, Cesa-Bianchi, and Fischer (2002); Auer and Cesa-Bianchi (1998); Lin et al. (2018); Maillard and Mannor (2014); Balakrishnan et al. (2019); Nguyen and Lauw (2014); Korda, Szörényi, and Li (2016); Gentile et al. (2017); Bouneffouf and Rish (2019); Pandey, Chakrabarti, and Agarwal (2007); Shivaswamy and Joachims (2012); Noothigattu et al. (2018); Allesiardo, Féraud, and Bouneffouf (2014); Bouneffouf et al. (2017). The works which are closely related to ours are  Pandey, Chakrabarti, and Agarwal (2007) and  Shivaswamy and Joachims (2012). Pandey, Chakrabarti, and Agarwal (2007); Wang, Zhou, and Shen (2018) study the classical bandit problem under the same model of arm clustering as in this work, and  Shivaswamy and Joachims (2012) studies the classical bandit problem under the same model of historical observations as in this work. In contrast to  Pandey, Chakrabarti, and Agarwal (2007); Wang, Zhou, and Shen (2018); Shivaswamy and Joachims (2012), our work provides 1) algorithms which simultaneously incorporate historical observations and clustering information in the classical bandit setting, 2) regret guarantees under tight and adversarial clustering for this setting, 3) algorithms which simultaneously incorporate historical observations and clustering information in the contextual bandit setting; we also provide regret guarantees for our classical bandit algorithm which uses history; prior to this work, we are not aware of such extensions for the contextual bandit setting, and 4) the META algorithm which effectively hedges between the strategy that uses both clustering and historical observations vs. the strategy which uses only the historical observations and not the clustering information.

3 Problem Setting

**Classical Bandit: ** The classical bandit problem is defined as exploring the response of $K$ arms within $T$ trials. Playing an arm yields an immediate, independent stochastic reward according to some fixed unknown distribution associated with the arm whose support is in $(0,1)$. The task is to find the reward-maximizing policy.

We adapt this scenario by assuming that the arms are grouped into $C$ clusters with arm $k$ assigned to cluster $c(k)$. Unlike the classical setting, we also wish to incorporate historical observations which may be available for the arms. Specifically, for $t\in\{1,2,\ldots,T\}$, let $r_{k}(t)$ denote the online reward from arm $k$ at time $t$. For notational convencience, we set it to [math] if $k$ was not played at time $t$.111This is purely for notational convenience, and does not affect the mean rewards computed per arm. Let $r^{h}_{k}(t)$ denote the historical reward for the $t^{th}$ instance of playing arm $k$ in history. $H_{k}$ is the number of historical instances available for arm $k$, and $H$ is defined as $H:=\sum^{K}_{k=1}H_{k}$. For each arm, the historical rewards are drawn independently from the same distribution as the online rewards. Let $\theta_{k}$ denote the expected reward for arm $k$. The goal is to maximize the expected total reward during $T$ iterations $\operatorname{\mathbb{E}}\left[\sum^{T}_{t=1}\theta_{k(t)}\right]$, where $k(t)$ is the arm played in step $t$, and the expectation is over the random choices of $k(t)$ made by the algorithm. An equivalent performance measure is the expected total regret, which is the amount of total reward lost by a specific algorithm compared to an oracle which plays the (unknown) optimal arm during each step. The expected total regret is defined as:

[TABLE]

with $\theta^{*}\operatorname{\overset{\text{def}}{=}}\max_{k}\theta_{k}$, $\Delta_{k}\operatorname{\overset{\text{def}}{=}}\theta^{*}-\theta_{k}$, and $n_{k}(t)$ denotes the number of times arm $k$ has been played until $t$.

**Contextual Bandit: ** The contextual bandit with history and pre-clustered arms is defined as follows: At each step $t\in\{1,...,T\}$, the player is presented with a context (feature vector) $\vec{x}_{t}\in\mathbb{R}^{d}$ before playing an arm from the set $A=\{1,\ldots,N\}$ that are grouped into $C$ known clusters. Let $r_{k}(t)$ denote the reward which the player can obtain by playing arm $k$ at time $t$ given context $\vec{x}_{t}$. As in Chu et al. (2011), we will primarily focus on bandits where the expected reward of an arm is linearly dependent on the context. Specifically, $\forall k,t:r_{k}(t)\in[0,1]$ and $\operatorname{\mathbb{E}}[r_{k}(t)|\vec{x}_{t}]=\theta_{k}^{\top}\vec{x}_{t}$ where $\theta_{k}\in\mathbb{R}^{d}$ is an unknown coefficient vector associated with the arm $k$ which needs to be learned from data. To incorporate historical information about rewards of the arms in a principled manner, we assume that each arm $k$ is associated with a matrix $\mathbf{H}_{k}\in\mathbb{R}^{d\times d}$ which records covariance information about the contexts played in history during times $t$ with $1\leq t\leq H_{k}$. $\mathbf{H}_{k}$ is computed using all observed context $\vec{x}_{t}$ so that:

[TABLE]

Here, $\mathbf{I}_{d}$ is the identity matrix of dimension $d$. The arm is also associated with a vector parameter $b^{h}_{k}$ which is the weighted sum of historical contexts, where the weights are the respective rewards:

[TABLE]

The role of the covariance matrix in HLINUCBC is analogous to its role in LINUCB which in turn is analogous to its role in least squares linear regression.

4 HUCBC for Classical Bandit

One solution for the classical bandit problem is the well known Upper Confidence Bound (UCB) algorithm Auer, Cesa-Bianchi, and Fischer (2002). This algorithm plays the current best arm at every time step, where the best arm is defined as one which maximizes the sum of observed mean rewards and an uncertainty term. We adapt UCB for our new setting such that it can incorporate both clustering and historical information. Our algorithm incorporates the historical observations by utilizing it both in the computation of the observed mean rewards and the uncertainty term. Our algorithm incorporates the clustering information by playing at two levels: first picking a cluster using a UCB-like strategy at each time step, and subsequently picking an arm within the cluster, again using a UCB-like strategy.

The pseudocode for HUCBC is presented in Algorithm 1 and the mean reward for arm $k$ is defined in (2).

[TABLE]

[TABLE]

At each time step $t$ of the exploration, HUCBC computes the quantity $\operatorname{HUCBC}_{k}(t)$ for arm $k$ using (3). $\operatorname{HUCBC}_{k}(t)$ represents an optimistic estimate of the reward attainable by playing arm $k$: this estimate incorporates the mean reward observed for the arm so far including the plays of the arm in history and an upper confidence term to account for the possibility of underestimation due to randomness in the rewards. In a similar manner, HUCBC also computes $\hat{\theta}_{i}^{hc}(t)$ and $\operatorname{HUCBC}_{i}^{c}(t)$ for each cluster $i$:

[TABLE]

[TABLE]

Note that $\hat{\theta}_{i}^{hc}(t)$ is the mean reward observed for cluster $i$ including plays of the cluster in history and $\operatorname{HUCBC}_{i}^{c}(t)$ represents an optimistic estimate of the reward attainable by playing cluster $i$. The quantities $r_{i}^{hc}(t)$, $r_{i}^{c}(t)$, $n_{i}^{c}(t)$, and $H_{i}^{c}$ in (4) and (5) are the per-cluster analogues of the corresponding quantities defined per-arm. Also note that, while the per-cluster quantities carry a superscript $c$ as part of their notation, the per-arm quantities do not. At each time step $t$, HUCBC first picks the cluster which maximizes $\operatorname{HUCBC}_{i}^{c}(t)$ and then picks the arm within this cluster which maximizes $\operatorname{HUCBC}_{k}(t)$.

**Regret Analysis: ** We upper-bound the expected number of plays of a sub optimal cluster under tight clustering of arms. Let $i^{*}$ denote the cluster containing the best arm. Arms are said to be tightly clustered if: 1) for each cluster $i$, there exists an interval $[l_{i},u_{i}]$ which contains the expected reward of every arm in cluster $i$, and 2) for $i\neq i^{*}$, the intervals $[l_{i},u_{i}]$ and $[l_{i^{*}},u_{i^{*}}]$ are disjoint. For any cluster $i\neq i^{*}$, define $\delta_{i}=l_{i^{*}}-u_{i}$.

Theorem 1** (Tight Clustering).**

The expected regret $\mathbb{E}[R(T)]$ at any time horizon $T$ under tight clustering of the arms in HUCBC is at most the following:

[TABLE]

Here, $\ell_{i}^{c}=\max\left(0,\frac{8\log(T+H_{i}^{c})}{\delta_{i}^{2}}-H_{i}^{c}\right)$, $\ell_{k}=\max\left(0,\frac{8\log(T+H_{k})}{\Delta_{k}^{2}}-H_{k}\right)$ and $k^{*}=\arg\max_{k}\theta_{k}$.

**Intuition behind Theorem 1: ** First, the cumulative regret up to time $T$ is logarithmic in $T$. Second, as the separation between clusters increases resulting in increased $\Delta_{k}$ values, the regret decreases. Third and a somewhat subtle aspect of this regret bound is that the number of terms in the summation equals the number of arms in the optimal cluster + the number of sub-optimal clusters. This number is always upper bound by the total number of arms. In fact, the difference between them can be pronounced when the clustering is well balanced – for example, when there are $\sqrt{n}$ clusters with $\sqrt{n}$ arms each, the total number of terms in the summation is $2\sqrt{n}$ while the total number of arms is $n$. When the quality of clustering is good, this is exactly the aspect of the regret bound which tilts the scales in favor of HUCBC compared to UCB or HUCB.

Proof.

The proof consists of three steps whose ideas are as follows. In the first step, we decompose the HUCBC regret into two parts: regret contributed by sub-optimal clusters (i.e., clusters without the optimal arm) and regret contributed by the sub-optimal arms within the optimal cluster (i.e, the cluster containing the optimal arm). The latter quantity can be upper-bounded using known results for HUCB. The main challenge in our analysis is to upper-bound the former, which we accomplish in Step 2. The main idea in Step 2 is to develop a concentration inequality for the reward obtained from any cluster. We accomplish this by proving that the cumulative reward deviation (i.e., the cumulative sum of actual rewards minus their expectation) is a martingale. In the analysis of UCB and HUCB, a concentration inequality of this type is derived through the use of Chernoff bounds: these are not applicable in our analysis due to the fact that rewards from a cluster are not only non-stationary but also not independent and highly correlated with rewards from previous plays of the cluster. We circumvent this difficulty through the use of martingale concentration inequalities in place of Chernoff bounds. In Step 3, we tie the results of Steps 1 and 2 together to obtain our final regret bound for HUCBC. The formal proof is as follows. Step 1: Since HUCBC uses HUCB to play arms within each cluster, we start by introducing the following regret bound for HUCB Shivaswamy and Joachims (2012).

Fact 2** **(Theorem 2, Shivaswamy and

Joachims (2012)).

The expected number of plays of any sub optimal arm $k$, within any cluster $i$, for any time horizon $T$, for any clustering of arms, satisfies:

[TABLE]

Here $k^{*}$ is the best arm within cluster $i$, and $\ell_{k}=\max\left(0,\frac{8\log(T+H_{k})}{\nabla_{k}^{2}}-H_{k}\right)$, where $\nabla_{k}=\theta_{k^{*}}-\theta_{k}$.

For any time horizon, the number of plays of a sub optimal arm in HUCBC is an upper bound on its regret. This quantity equals the sum of the number of plays of a sub optimal cluster along with the number of plays of sub optimal arms within the optimal cluster. The latter quantity is upper-bounded using Fact 2 since for any sub optimal arm $k$ in the optimal cluster $i^{*}$, $\Delta_{k}=\nabla_{k}$. The former quantity is upper bounded in Steps 2 and 3.

Step 2: Given cluster $i$, let $t_{\ell}$ be the time slot when cluster $i$ is played for the $\ell^{th}$ time in the online phase. Suppose $t_{1},t_{2},\ldots$ be fixed and given. Define:

[TABLE]

In this definition, $j^{h}(\ell)$ is the arm that was played during the $\ell^{th}$ instance of cluster $i$ in the historical phase, and $j(\ell)$ is the arm that is played during the $\ell^{th}$ instance of cluster $i$ in the online phase, $r$ (with the appropriate subscripts and superscripts) refers to the actual reward obtained for that instance, and $\theta$ (with the appropriate subscripts and superscripts) refers to the mean reward of the arm for that instance. We claim that the random sequence $\{z_{i}^{c}(\ell)|\ell=1,2,\ldots\}$ is a martingale. Indeed, for $1\leq\ell\leq H_{i}^{c}$, we have:

[TABLE]

A similar argument holds when $\ell>H_{i}$ which implies $\{z_{i}^{c}(\ell)|\ell=1,2,\ldots\}$ is a martingale. It is easy to verify that $\forall\ell,~{}|z_{i}^{c}(\ell)-z_{i}^{c}(\ell-1)|<1$; hence, by Azuma-Hoeffding inequality Alon and Spencer (2016), we have:

[TABLE]

Consider $i=i^{*}$. We note that since $t_{1},t_{2},\ldots$ are fixed, $\forall t$, $n_{i}^{c}(t)$ is a fixed number. We have:

[TABLE]

We note that since (9) holds conditionally for any fixed sequence $t_{1},t_{2},\ldots$, it holds unconditionally as well. Now consider $i\neq i^{*}$, and any fixed $t$ such that $n_{i}^{c}(t)=\gamma$, where $\gamma\geq\ell_{i}^{c}~{}\operatorname{\overset{\text{def}}{=}}~{}\max\left\{0,\frac{8\log(t+H_{i}^{c})}{\delta_{i}^{2}}-H_{i}^{c}\right\}$.

[TABLE]

We note that since (10) holds conditionally for any fixed sequence $t_{1},t_{2},\ldots$, we have:

[TABLE]

Step 3: Suppose we have an event $\mathcal{A}$ such that $\mathcal{A}\implies\mathcal{B}\vee\mathcal{C}\vee\mathcal{D}$. Then, for any event $\mathcal{E}$, we have:

[TABLE]

Consider a suboptimal cluster $i$. If $H_{i}^{c}=0$, then HUCB (like UCB) will initialize itself by playing cluster $i$ once at the start of the online phase. Define this as event $\mathcal{B}$. Define $\mathcal{A}=\mathbf{1}(i(t)=i)$, $\mathcal{C}=\mathbf{1}(HUCBC^{c}_{i^{*}}(t)\leq l_{i^{*}})$, $\mathcal{D}=\mathbf{1}(HUCBC^{c}_{i}(t)\geq u_{i}+\delta_{i})$, and $\mathcal{E}=\mathbf{1}\left(n_{i}(t)<\ell_{i}^{c}\right)$ above. We now have:

[TABLE]

The theorem now follows by combining (13) and (7). ∎

We now upper-bound the expected number of plays of a sub optimal cluster where an adversary is free to cluster the arms in order to elicit the worst case behavior from HUCBC. For ease of analysis, we will analyze a variant of HUCBC which we denote as $HUCBC^{\prime}$, which plays UCB at the inter-cluster level and plays HUCB at the intra-cluster level.

Theorem 3** (Adversarial Clustering).**

The expected regret at any time horizon $T$ under any clustering of the arms in $HUCBC^{\prime}$ satisfies the following:

[TABLE]

Here, $r,s$ are constants and $\ell_{k}$ and $k^{*}$ are defined as in Theorem 1.

Proof.

(7) implies that if we have a cluster which contains $\alpha$ arms, then the expected number of plays of sub optimal arms within that cluster as incurred by the HUCB strategy is $O(\alpha\log T)$ for any time horizon $T$. This fact in turn implies that the expected per-step regret converges in to [math], and hence the expected reward of this cluster converges to the reward of the best arm within this cluster:

[TABLE]

Further, set $v=\frac{4\log(t)}{\ell}$ in Eqn (8). Then, for any time horizon $t>1$ it follows that total reward of a cluster after $\ell$ pulls is within the interval of width $2v$ centered around its expectation with probability at least $1-\frac{1}{t^{4}}$. This high concentration condition along with Eqn (15) satisfy the drift conditions for non-stationary arms in Kocsis and Szepesvári (2006).

Suppose the UCB algorithm is applied to a non-stationary collection of ‘drifting’ arms where the drift conditions are as above. Then, from Kocsis and Szepesvári (2006), we know that the expected number of plays of any sub optimal ‘drifting’ arm $i$, for any time horizon $T$ satisfies:

[TABLE]

where $r$ and $s$ are constants, and $\theta_{i}^{c}$ is the steady state reward for the ‘drifting’ arm $i$. In $HUCBC^{\prime}$, the role of the drifting arms is played by the clusters. The regret of $HUCBC^{\prime}$ can be decomposed into the regret from the sub-optimal arms in the optimal cluster + the regret due to the sub-optimal clusters. The former is upper-bounded by (2) and the latter is upper-bounded by (16). Combining them yields the theorem. ∎

**Comparing Theorems 1 and 3: ** Both these theorems share important similarities. First, the regret is $O(\log{T})$. Second, as the distance between clusters increase, the regret decreases. Third, the number of terms in the summation equals the number of arms in the optimal cluster + the number of sub-optimal clusters. However, there are significant differences. First, the distance between two clusters in Theorem 3 is measured differently: it is now the difference between the mean rewards of the best arms in the clusters. Second and more importantly, the constants involved in the bound of Theorem 3 arise from the results of Kocsis and Szepesvári (2006) and are significantly bigger than those involved in the bound of Theorem 1. We emphasize that Theorem 3 establishes only an upper bound on the regret: the actual regret for typical instances that arise in practice can be a lot smaller than this upper bound. The proof of this theorem leverages the martingale concentration inequality derived as part of Theorem 1. This is a key part of the drift condition which needs to be satisfied in order for us to use the result of Kocsis and Szepesvári (2006) for bounding the performance of UCB in non-stationary environments.

5 HLINUCBC for Contextual Bandit

A well known solution for the contextual bandit with linear payoffs is LINUCB Li et al. (2010b) where the key idea is to apply online ridge regression to the training data to estimate the coefficients $\theta_{k}$. We propose HLINUCBC (Algorithm 2) which extends this idea with both historical and clustering information.

**Clustering Information: ** HLINUCBC deals with arms that are clustered; it applies online ridge regression at the per-cluster level to obtain an estimate of the coefficients $\hat{\theta}_{i}^{c}$ for each cluster (Line 7), plays the best cluster and then applies online ridge regression to the arms within the chosen cluster to obtain an estimate of the coefficients $\hat{\theta}_{k}$ for each arm (Line 6). To find the best cluster, at each trial $t$, HLINUCBC computes the quantities:

[TABLE]

[TABLE]

For each cluster $i$ and selects the cluster with the highest value of $p^{c}_{t,i}$ (Line 4). This quantity encapsulates the estimated reward from this cluster, along with an uncertainty term. Then, HLINUCBC finds the best arm within this cluster by computing for each arm $k$ in this cluster, the quantity (18) and selects the arm with the highest value $p_{t,k}$ (Line 5).

**Historical Information: ** HLINUCBC applies online ridge regression to the historical data at both per-cluster and per-arm levels. The aim is to collect enough information about how the context vectors and rewards relate to each other for each cluster and arm using the historical data, so that it can jump-start the algorithm by achieving a low number of errors at the early stages of the online exploration. At the initialization step, HLINUCBC is seeded with $\mathbf{H}^{c}_{i}$ and $\mathbf{H}_{k}$ which are respectively the history matrices for the clusters and arms. This is in contrast to LINUCB, where the initialization is done using an identity matrix. It is also seeded with the vectors $b^{hc}_{i}$ and $b^{h}_{k}$, which record the weighted sum of historical contexts, with the weight being the rewards. In contrast, in LINUCB, this quantity is initialized to $\mathbf{0}$.

Theorem 4**.**

With probability $1-\delta$, where $0<\delta<1$, the upper bound on the R(T) for the HLINUCB in the contextual bandit problem, $K$ arms and $d$ features (context size) is as follows:

[TABLE]

with $||x_{t}||_{2}\leq L$ and $\phi\in R$

Proof.

We need the following lemma from Abbasi-Yadkori, Pál, and Szepesvári (2011),

Lemma 5**.**

Assuming that, the measurement noise $n_{t}$ is independent of everything and is $\sigma$-sub-Gaussian for some $\sigma>0$, i.e., $E[e^{\phi\,n_{t}}]\leq exp(\frac{\phi^{2}\sigma^{2}}{2})$ for all $\phi\in R$ and $r_{t}=\langle x_{t},\theta^{*}\rangle+n_{t}$ . With probability $1-\delta$, where $0<\delta<1$ and $\theta^{*}$ lies in the confidence ellipsoid.

[TABLE]

The inner product is denoted by $\langle\cdot,\cdot\rangle$. The lemma is directly adopted from theorem 2 in Abbasi-Yadkori, Pál, and Szepesvári (2011), we follow the same step of proof, the main difference is that they have $\mathbf{A}_{t}=\lambda I+\sum_{s=1}^{t}x_{s}x_{s}^{\top}$ and we have,

[TABLE]

Following the same step as the proof of theorem 2 in Abbasi-Yadkori, Pál, and Szepesvári (2011) We also have the following,

$R(t)\leq 2c_{t}||x_{t}||_{\mathbf{A}_{t}^{-1}}$, Since $x^{\top}\theta_{t}^{*}\in[-1,1]$ for all $x\in X_{t}$ then we have $R(t)\leq 2$. Therefore,

[TABLE]

[TABLE]

we have,

[TABLE]

with $c_{t}$ monotonically increasing. Since $x\leq 2\,log(1+x)$ for $x\in[0,1]$,

[TABLE]

here we also use the fact that have $\mathbf{A}_{t}=\mathbf{H}+\sum_{s=1}^{t}x_{s}x_{s}^{\top}$ to get the last inequality.

[TABLE]

by upper-bounding $c_{T}$ using lemma 5 we get our result. ∎

Theorem 4 shows that the HLINUCB upper bound has $\log(\frac{det(\mathbf{A}_{t})}{det(\mathbf{H})})$ under the square root whereas LINUCB has $\log(det(\mathbf{A}_{t})$. Recall from Abbasi-Yadkori, Pál, and Szepesvári (2011) that LINUCB has a regret guarantee which is almost the same as the one in Theorem 4 except for the $det(\mathbf{H})$ term. We now demonstrate that $\frac{det(\mathbf{A}_{t})}{det(\mathbf{H})}\leq det(\mathbf{A}_{t})$ and thereby show that HLINUCB has a provably better guarantee than LINUCB. The matrix $H$ can be written as $I+D_{h}$ where $I$ is the identity matrix and $D_{h}$ is the design matrix constructed using the contexts in history. Both $I$ and $D_{h}$ are real symmetric and hence Hermitian matrices. Further, $D_{h}$ is positive semi-definite since $D_{h}=\sum_{i}x_{i}x_{i}^{T}$, where the $x_{i}$ are the historical contexts; to see this, note that $\forall y,~{}y^{T}D_{h}y=\sum_{i}y^{T}x_{i}x_{i}^{T}y=\sum_{i}(x_{i}^{T}y)^{2}\geq 0$. Since all the eigenvalues of $I$ equal $1$ and since all the eigenvalues of $D_{h}$ are non-negative, by Weyl’s inequality in matrix theory for perturbation of Hermitian matrices Thompson and Freede (1971), the eigenvalues of $H$ are lower-bounded by $1$. Hence $\mathbf{det(H)}$ which is the product of the eigenvalues of $H$ is lower-bounded by $1$ which proves our claim.

6 META Algorithm

Intuitively, HUCBC can be expected to outperform HUCB when the quality of clustering is good. However, when the clustering quality is poor, this is far less likely to be the case. In order to choose between these two strategies correctly, we employ the META algorithm. Specifically, for the classical bandit problem, META uses UCB to decide between HUCBC and HUCB at each iteration. For the contextual bandit problem, META uses UCB to decide between HLINUCBC and HLINUCB at each iteration. The basic idea is that if the clustering quality is poor, the fraction of the time when META plays the strategy with both clustering and history will become vanishingly small, leading to the desired behavior.

Theorem 6**.**

The META algorithm is asymptotically optimal for the classical bandit problem under the assumption that the drift conditions of Kocsis and Szepesvári (2006) hold for HUCBC.

Proof.

The META algorithm essentially has two arms to play in each iteration, namely HUCB and HUCBC. We know from Fact 2 that HUCB has a sublinear regret and is asymptotically optimal. Further, in the proof of Theorem 3, we also establish that HUCB satisfies the drift conditions of  Kocsis and Szepesvári (2006). Hence, under the assumption that HUCBC also satisfies the drift conditions, Kocsis and Szepesvári (2006) guarantees that META will also have sub-linear regret and is asymptotically optimal. ∎

7 Experimental Evaluation

Experiments with Synthetic Classical Bandit: We compare our proposed strategies HUCBC (Section 4) and META (Section 6) to the state-of-the-art competitors UCB Auer, Cesa-Bianchi, and Fischer (2002), HUCB Shivaswamy and Joachims (2012), and UCBC Pandey, Chakrabarti, and Agarwal (2007). HUCB is the degenerate version of HUCBC where all arms belong to the same cluster, i.e., no use of clustering information is made only historical data is employed. In contrast, UCBC is the degenerate version which uses the clustering information but no historical data.

Our synthetically generated data assumes 10 clusters and 100 arms. A random permutation assigns each arm $k$ to a cluster $c(k)$, where the centroid of cluster $i$ is defined as $\lambda(i)=\frac{1}{2}\left(u_{i}+\frac{1}{i}\right)$ and $u_{i}\sim\mathcal{U}(0,1)$. The reward obtained when arm $k$ is played is sampled from $\mathcal{U}(0,2\alpha_{k}\lambda(c(k)))$, where $\alpha_{k}$ is a arm dependent constant. Thus, the expected reward for cluster $i$ is $\lambda(i)$. The historical data is generated as follows: for 25% we generate data playing them $X$ times where $X$ is sampled from a Poisson distribution with parameter 10. We report the results of 20 trials in in Figure 1(a). Each trial consisted of $10^{4}$ rounds and we plot the the number of rounds per-round-reward (cumulative reward until that round / number of rounds). The error bars correspond to the interval of $\pm 1$ standard deviation around the mean. Clearly, the per-round-reward of HUCBC is the fastest to converge to that of the best arm. Interestingly, while historical data improves the performance of HUCB over UCB only to a small degree (possibly due to asymmetry in history), combining this information with the clustering information improves the performance of HUCBC over UCBC to a significantly larger degree. This is an effect of the fact that even though historical data is sparse and asymmetric at the arm level, combining this information at the cluster level is more impactful. META converges rather quickly to the correct choice (HUCBC) in this setting.

Experiments with Synthetic Contextual Bandit: We will now compare our proposed contextual bandits HLINUCBC (Section 5) and the META algorithm selection to LINUCB Li et al. (2010b), HLINUCB and LINUCBC. Similarly to HUCB and UCBC, HLINUCB and LINUCBC are degenerated versions of HLINUCBC using either only the historical data or the clustering information.

The synthetic data was created by assuming 10 clusters and 100 arms. Again, a random permutation assigns each arm $k$ to a cluster $c(k)$. The centroid $\theta^{c}_{i}$ of cluster $i$ is sampled from $\mathcal{N}(\mathbf{0},I_{5})$. The coefficient $\theta_{k}$ of arm $k$ is defined as $\theta_{k}=\theta^{c}_{i}+\epsilon\nu_{k}$, where $\nu_{k}$ is sampled independently at random from $\mathcal{N}(\mathbf{0},I_{5})$. The reward for playing arm $k$ in context $x$ is sampled from $\mathcal{U}(0,2\theta_{k}^{\top}x)$. Thus, our synthetically generated problem is linear in the input and the expected distance between clusters is $\sqrt{5}\epsilon$. By varying $\epsilon$, we control the tightness of the clusters and can observe the impact of clustering information with varying quality of clusters. Contexts in each round of a trial are drawn i.i.d from the multivariate normal distribution $\mathcal{N}(\mathbf{0},I_{5})$. We created asymmetric historical data for arms by playing an arm $X$ times with $X$ distinct contexts, where $X\sim Poisson(10)$.

We report the results for $\epsilon=0.1$ in Figure 1(b). As in the case of classical bandits, we repeat the experiment 20 times, where each trial consists of 10,000 rounds. The error bars correspond to the interval of $\pm 1$ standard deviation around the mean. HLINUCBC is clearly outperforming its competitors and META is able to identify HLINUCBC as the stronger method over HLINUCB. In this set up, historical data and clustering are equally important such that HLINUCB and LINUCBC provide similar results. LINUCB clearly provides the worst results. We also compare the performance of HLINUCBC vs. LINUCBC under different values of $\epsilon\in\{0.1,0.8,3.2\}$. We normalize the rewards in these three distinct settings to enable a meaningful comparison. We present the results in Figure 1(c). Since HLINUCBC utilizes historical data in addition to the clustering information, we can see that it improves upon the performance of LINUCBC for every setting of $\epsilon$.

**Web Server Selection: Krishnamurthy, Wills, and Zhang (2001) ** Essentially one needs to decide from what source one should pull content available at multiple sources while minimizing the cumulative latency for successive retrievals. Similar to Vermorel and Mohri (2005), we assume that only a single resource can be picked at a time. The university web page data set222https://sourceforge.net/projects/bandit/ features more than $700$ sources with about $1300$ response times each. In order to facilitate our clustering approach in this setting, we perform two types of clustering: 1) split the resources based on average latencies into five categories ranging from ‘fast’ to ‘slow’, 2) based on domain names into 17 clusters. Historical data was generated by choosing 200 observations in total at random. In Figures 1(d) and 1(e) we show the mean cumulative reward for the UCB, UCB with history (HUCB), UCB with clustering (UCBC), UCB with both (HUCBC), and META over 10 repetitions. Before each experiment, the data has been shuffled. We can observe a clear impact of employing clustering and history and the combination of both, and the type of clustering. The first type of clustering achieves an effective grouping of arms while the second one does not. Confirming our observations on the synthetic data, only clustering information provides better results than only historic data when clustering is of good quality. The two kinds of clustering also highlight the usefulness of our META approach that in both cases converges to the correct approach.

**Warfarin Drug Dosage: ** The Warfarin problem data set is concerned with determining the correct initial dosage of the drug Warfarin for a given patient. Correct dosage is variable due to patient’s clinical, demographic and genetic context. We select this benchmark because it enables us to compare the different algorithms since this problem allows us to create a hierarchical classification data set. Originally, the data set was converted to a classification data set with three classes using the nominal feedback of the patient. We create a hierarchy as follows. We divide each of the original three classes into five more granular classes. Bandit methods which do not use hierarchy will simply tackle this problem as a classification task with 15 classes. Others will first assign instances to one of the three classes and finally decide to choose one of the final five options. The order of the patients is shuffled, 1500 of them are chosen as historical data. We report the mean and standard deviation of 10 runs in Figure 1(f). HLINUCBC is outperforming all competitor methods and also META is able to successfully detect that HLINUCBC is the dominating method. For this problem HLINUCB provides better results than LINUCBC, indicating that historic data is worth more than clustering information.

8 Conclusions

We introduced a variety of algorithms for classical and contextual bandits which incorporate historical observations and pre-clustering information between arms in a principled manner. We demonstrated their effectiveness and robustness both through rigorous regret analysis as well as extensive experiments on synthetic and real world datasets. Two interesting open problems emerge from this work: 1) Are there instance independent regret bounds for HUCBC which depend only on the total number of clusters and the number of arms in the optimal cluster? 2) What are the upper and lower bounds on the regret of HLINUCBC?