Exploiting Relevance for Online Decision-Making in High-Dimensions

TL;DR

This paper introduces a new online learning algorithm, CMAB-RL, for high-dimensional decision-making tasks, leveraging sparsity to improve learning efficiency and applying it to personalized diabetes management.

Contribution

The paper proposes the CMAB-RL algorithm that exploits relevance and sparsity in high-dimensional data for improved regret bounds in contextual bandit problems.

Findings

CMAB-RL achieves asymptotically zero regret under smooth reward conditions.

The algorithm outperforms classical CMAB algorithms in high-dimensional settings.

Application to diabetes control demonstrates practical effectiveness.

Abstract

Many sequential decision-making tasks require choosing at each decision step the right action out of the vast set of possibilities by extracting actionable intelligence from high-dimensional data streams. Most of the times, the high-dimensionality of actions and data makes learning of the optimal actions by traditional learning methods impracticable. In this work, we investigate how to discover and leverage sparsity in actions and data to enable fast learning. As our learning model, we consider a structured contextual multi-armed bandit (CMAB) with high-dimensional arm (action) and context (data) sets, where the rewards depend only on a few relevant dimensions of the joint context-arm set, possibly in a non-linear way. We depart from the prior work by assuming a high-dimensional, continuum set of arms, and allow relevant context dimensions to vary for each arm. We propose a new online learning algorithm called {\em CMAB with Relevance Learning} (CMAB-RL) and prove that its time-averaged regret asymptotically goes to zero when the expected reward varies smoothly in contexts and arms. CMAB-RL enjoys a substantially improved regret bound compared to classical CMAB algorithms whose regrets depend on dimensions $d_x$ and $d_a$ of the context and arm sets. Importantly, we show that when the learner has prior knowledge on sparsity, given in terms of upper bounds $\overline{d}_x$ and $\overline{d}_a$ on the number of relevant dimensions, then CMAB-RL achieves $\tilde{O}(T^{1-1/(2+2\overline{d}_x +\overline{d}_a)})$ regret. Finally, we illustrate how CMAB algorithms can be used for optimal personalized blood glucose control in type 1 diabetes mellitus patients, and show that CMAB-RL outperforms other contextual MAB algorithms in this task.

Tables

Table 1. TABLE I: Comparison of our work with the related works.

MAB algorithm	Regret bound	Relevance learning	High-dimensional arm set
Contextual Zooming [14]	$\tilde{O}\left(T^{1-1/\left(2+d_z\right)}\right)$	No	Yes
Query-Ad-Clustering [18]	$\tilde{O}\left(T^{1-1/\left(2+d_c\right)}\right)$	No	Yes
RELEAF [19]	$\tilde{O}\left(T^{g\left({\overline{d}}_x\right)}\right)$	Yes	No
CMAB-RL (our work)	$\tilde{O}\left(T^{1-1/\left(2+2{\overline{d}}_x+{\overline{d}}_a\right)}\right)$	Yes	Yes

Table 2. TABLE II: Percentages of samples for all approaches and patients

		Patient 559	Patient 563	Patient 570	Patient 575	Patient 588	Patient 591	Overall
<80 mg/dL	CMAB-RL	00.25	00.04	00.13	00.47	00.24	00.71	00.27
	C-HOO	00.30	00.05	00.14	00.58	00.32	00.89	00.34
	IUP	00.35	00.05	00.18	00.69	00.38	00.96	00.38
	Dataset	01.97	00.25	01.59	04.50	00.00	03.41	01.75
80-180 mg/dL	CMAB-RL	66.92	70.49	66.64	88.65	69.06	79.31	72.78
	C-HOO	53.57	62.54	54.98	81.54	56.73	67.27	62.30
	IUP	50.10	56.08	51.76	77.11	52.39	64.25	57.99
	Dataset	36.84	56.78	37.93	62.00	39.81	57.95	49.06
>180 mg/dL	CMAB-RL	32.83	29.47	33.23	10.88	30.70	19.98	26.95
	C-HOO	46.12	37.41	44.88	17.87	42.95	31.84	37.36
	IUP	49.56	43.87	48.06	22.20	47.23	34.79	41.63
	Dataset	61.18	42.96	60.48	33.50	60.19	38.64	49.19

Table 3. TABLE III: Notation related to problem formulation

Notation	Description
$\left|𝒮\right|$	Cardinality of a set $𝒮$
$\wp \left(𝒮\right)$	The power set of set $𝒮$
$𝒳$	Context set
$d_x$	Number of context dimensions
${𝒟}_x$	Set of context dimensions: ${𝒟}_x≔\{1,\dots ,d_x\}$
${𝒱}_x^l$	$\{𝒗\in \wp \left({𝒟}_x\right):\left|𝒗\right|=l\}$
${𝒱}_x^l\left(𝒗\right)$	$\{𝒘\in {𝒱}_x^l:𝒗\subseteq 𝒘\}$ for $𝒗\subseteq {𝒟}_x$
${𝒳}_{𝒛}$	Subset of $𝒳$ that contains the values of context dimensions in $𝒛\subseteq {𝒟}_x$
$x_{𝒛}$	$\left|𝒛\right|$-tuple subcontext whose elements are elements of $x$ that correspond to the context dimensions in $𝒛$
$x\left(t\right)$	Context in round $t$
$𝒜$	Arm set
$d_a$	Number of arm dimensions
${𝒟}_a$	Set of arm dimensions: ${𝒟}_a≔\{1,\dots ,d_a\}$
${𝒱}_a^l$	$\{𝒗\in \wp \left({𝒟}_a\right):\left|𝒗\right|=l\}$
${𝒜}_{𝒛}$	Subset of $𝒜$ that contains the values of arm dimensions in $𝒛\subseteq {𝒟}_a$
$a_{𝒛}$	$\left|𝒛\right|$-tuple subarm whose elements are elements of $a$ that correspond to the arm dimensions in $𝒛$
$a^{\ast }\left(x\right)$	Optimal arm for context $x$
$a\left(t\right)$	Arm chosen by the learner in round $t$
${\mu }_a\left(x\right)$	Expected reward of arm $a$ for context $x$
$r\left(t\right)$	Reward obtained in round $t$
$\kappa \left(t\right)$	Noise in the reward in round $t$
$𝒄$	Subset of relevant arm dimensions: $𝒄\subseteq {𝒟}_a$
${\underset{¯}{d}}_a$	Number of relevant arm dimensions
${\overline{d}}_a$	Known upper bound on the number of relevant arm dimensions
${𝒄}_a$	Subset of relevant context dimensions for arm $a$: ${𝒄}_a\subseteq {𝒟}_x$
${\underset{¯}{d}}_x$	Maximum number of relevant context dimensions
${\overline{d}}_x$	Known upper bound on the maximum number of relevant context dimensions
$\text{Reg}\left(t\right)$	Cumulative regret until round $t$

Table 4. TABLE IV: Notation related to regret analysis

Notation	Description
UC	Unconfident event
${𝒱}_x(𝒗,d^{\prime })$	Set of $d^{\prime }$-tuples of context dimensions whose elements are from the set ${𝒟}_x\setminus 𝒗$ for any $𝒗\in {𝒱}_x^{{\overline{d}}_x}$ and $d^{\prime }\le d_x-{\overline{d}}_x$, $d^{\prime }\in {\mathbb{Z}}^{+}$
${\tau }_{p_{𝒘}}\left(t\right)$	Round in which a context arrives to $p_{𝒘}$ for the $t$th time
${\tau }_{y,p_{𝒘}}\left(t\right)$	Round in which a context arrives to $p_{𝒘}$, arm $y$ is selected and $𝒘={𝒘}_y\left(t\right)$ for the $t$th time
${\tilde{x}}_{p_{𝒘}}\left(t\right)$	$x\left({\tau }_{p_{𝒘}}\left(t\right)\right)$
${\tilde{N}}_{y,p_{𝒘}}\left(t\right)$	$N_{y,p_{𝒘}}\left({\tau }_{p_{𝒘}}\left(t\right)\right)$
${\tilde{\mu }}_{y,p_{𝒘}}\left(t\right)$	${\hat{\mu }}_{y,p_{𝒘}}\left({\tau }_{p_{𝒘}}\left(t\right)\right)$
${\tilde{u}}_{y,p_{𝒘}}\left(t\right)$	$u_{y,p_{𝒘}}\left({\tau }_{p_{𝒘}}\left(t\right)\right)$
${\tilde{y}}_{p_{𝒘}}\left(t\right)$	Arm chosen by the learner in round ${\tau }_{y,p_{𝒘}}\left(t\right)$
$y^{\ast }\left(t\right)$	Optimal arm in set $𝒴$ for the context in round $t$

Table 5. TABLE V: Notation related to CMAB-RL

Notation	Description
$m$	Partition number
${𝒫}_i$	Uniform partition of the subcontext in dimension $i$: ${𝒫}_i:=\{[0,\frac{1}{m}],(\frac{1}{m},\frac{2}{m}],\dots ,(\frac{m-1}{m},1]\}$
${𝒫}_{𝒘}$	${\prod }_{i\in 𝒘}{𝒫}_i$
${ℐ}_i$	Uniform partition of the subarm in dimension $i$: ${ℐ}_i:=\{[0,\frac{1}{m}],(\frac{1}{m},\frac{2}{m}],\dots ,(\frac{m-1}{m},1]\}$
${ℐ}_{𝒗}$	${ℐ}_{𝒗}:={\prod }_{i\in 𝒗}{ℐ}_i$
$𝒞\left(𝒜\right)$	${\cup }_{𝒗\in {𝒱}_a^{{\overline{d}}_a}}{ℐ}_{𝒗}$
$y$	Index of the geometric centers of the elements of $𝒞\left(𝒜\right)$
$𝒴$	Set of arms that correspond to geometric centers of the elements of $𝒞\left(𝒜\right)$
$p_{𝒘}\left(t\right)$	Element of ${𝒫}_{𝒘}$ that $x_{𝒘}\left(t\right)$ belongs to for $𝒘\in {𝒱}_x^{2{\overline{d}}_x}$
$N_{y,𝒘}\left(t\right)$	Counter that counts the number of times context was in $p_{𝒘}$ and arm $y$ was selected before round $t$
${\hat{\mu }}_{y,𝒘}\left(t\right)$	Sample mean of the rewards that is obtained from rounds prior to round $t$ in which context was in $p_{𝒘}$ and arm $y$ was selected
$u_{y,𝒘}\left(t\right)$	Uncertainty term for arm $y\in 𝒴$ and $𝒘\in {𝒱}_x^{2{\overline{d}}_x}$ in round $t$
${ℛ}_y\left(t\right)$	Set of candidate relevant tuples of context dimensions for $y\in 𝒴$ in round $t$
${\hat{\sigma }}_{y,𝒗}^2$	Variance estimate of $𝒗\in {ℛ}_y\left(t\right)$ and $y\in 𝒴$ in round $t$
${\widehat{𝒄}}_y$	Tuple of estimated relevant context dimensions for arm $y\in 𝒴$
${\hat{\mu }}_y^{𝒗}\left(t\right)$	Sample mean reward of arm $y\in 𝒴$ for the tuple of contex dimensions $𝒗\in {𝒱}_x^{{\overline{d}}_x}$ in round $t$

Equations

$$E[e^{\lambda \kappa (t)}\text{nonscript}|\text{nonscript}{a}_{1:t},x_{1:t},{\kappa }_{1:t-1}]\le \text{exp}({\lambda }^2/2)$$

$$|{\mu }_a(x)-{\mu }_{a^{\prime }}(x^{\prime })|\le L(\Vert x_{{\mathbf{c}}_a}-x_{{\mathbf{c}}_a}^{\prime }\Vert +\Vert a_{\mathbf{c}}-a_{\mathbf{c}}^{\prime }\Vert )$$

$$|{\mu }_a(x)-{\mu }_{a^{\prime }}(x^{\prime })|\le L(\Vert x_{{\mathbf{c}}_{a^{\prime }}}-x_{{\mathbf{c}}_{a^{\prime }}}^{\prime }\Vert +\Vert a_{\mathbf{c}}-a_{\mathbf{c}}^{\prime }\Vert ).$$

$$\text{Reg}(T):=\sum _{t=1}^T{\mu }_{a^{\ast }(x(t))}(x(t))-\sum _{t=1}^T{\mu }_{a(t)}(x(t)).$$

$${\hat{\mu }}_y^{\mathbf{v}}(t):=\frac{\sum _{\mathbf{w}\in {\mathcal{V}}_x^{2{\overline{d}}_x}(\mathbf{v})}{\hat{\mu }}_{y,\mathbf{w}}(t)N_{y,\mathbf{w}}(t)}{\sum _{\mathbf{w}\in {\mathcal{V}}_x^{2{\overline{d}}_x}(\mathbf{v})}{N}_{y,\mathbf{w}}(t)}.$$

$${\mathcal{R}}_y(t):=\{\mathbf{v}\in {\mathcal{V}}_x^{{\overline{d}}_x}:|{\hat{\mu }}_{y,\mathbf{w}}(t)-{\hat{\mu }}_{y,{\mathbf{w}}^{\prime }}(t)|$$

$$\le 2L\sqrt{{\overline{d}}_x}/m+u_{y,\mathbf{w}}(t)+u_{y,{\mathbf{w}}^{\prime }}(t),\forall \mathbf{w},{\mathbf{w}}^{\prime }\in {\mathcal{V}}_x^{2{\overline{d}}_x}(\mathbf{v})\}.$$

$${\hat{\sigma }}_{y,\mathbf{v}}^2(t):=\underset{\mathbf{w},{\mathbf{w}}^{\prime }\in {\mathcal{V}}_x^{2{\overline{d}}_x}(\mathbf{v})}{\max }|{\hat{\mu }}_{y,\mathbf{w}}(t)-{\hat{\mu }}_{y,{\mathbf{w}}^{\prime }}(t)|.$$

$${\text{UCB}}_y(t):={\hat{\mu }}_y^{{\hat{\mathbf{c}}}_y(t)}(t)+5u_{y,{\mathbf{w}}_y(t)}(t).$$

$${\hat{\mu }}_{y(t),\mathbf{w}}(t+1)$$

$$N_{y(t),\mathbf{w}}(t+1)$$

$$\text{Reg}(T)\le C_{\max }|{\mathcal{V}}_x^{2{\overline{d}}_x}|\left(\genfrac{}{}{0ex}{}{d_a}{{\overline{d}}_a}\right){\tilde{T}}^{\frac{2{\overline{d}}_x+{\overline{d}}_a}{2+2{\overline{d}}_x+{\overline{d}}_a}}$$

$$+(L(10\sqrt{{\overline{d}}_x}+\sqrt{{\overline{d}}_a})+2\sqrt{|{\mathcal{V}}_{2{\overline{d}}_x}|\left(\genfrac{}{}{0ex}{}{d_a}{{\overline{d}}_a}\right)}{B}_{m,T}){\tilde{T}}^{\frac{1+2{\overline{d}}_x+{\overline{d}}_a}{2+2{\overline{d}}_x+{\overline{d}}_a}}$$

$${\text{UC}}_{y,p_{(\mathbf{v},\mathbf{j})}}:=\bigcup _{t=1}^{N_{p_{(\mathbf{v},\mathbf{j})}}(T)}\{{\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))$$

$$[L_{y,p_{(\mathbf{v},\mathbf{j})}}(t)$$

$$\text{UC}:=\bigcup _{\mathbf{v}\in {\mathcal{V}}_x^{{\overline{d}}_x}({\mathbf{c}}_y),\mathbf{j}\in {\mathcal{V}}_x(\mathbf{v},{\overline{d}}_x)}{\text{UC}}_{(\mathbf{v},\mathbf{j})}.$$

$${\overline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}$$

$$\mathrm{Pr}(\text{UC})\le \frac{1}{T}.$$

$${\tilde{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)=\frac{{\sum }_{l=1}^{t-1}{\tilde{R}}_{y,p_{(\mathbf{v},\mathbf{j})}}(l)\text{I}({\tilde{y}}_{p_{(\mathbf{v},\mathbf{j})}}(l)=y)}{{\tilde{N}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)}$$

$${\tilde{R}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)={\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))+{\tilde{\kappa }}_{p_{(\mathbf{v},\mathbf{j})}}(t)$$

$${\overline{R}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)={\overline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}+{\tilde{\kappa }}_{p_{(\mathbf{v},\mathbf{j})}}(t)$$

$${\underline{R}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)={\underline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}+{\tilde{\kappa }}_{p_{(\mathbf{v},\mathbf{j})}}(t).$$

$${\overline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)$$

$${\underline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)$$

$${\underline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)\le {\tilde{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)\le {\overline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)$$

$${\overline{L}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)$$

$${\overline{U}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)$$

$${\underline{L}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)$$

$${\underline{U}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)$$

$$\{{\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))\notin [L_{y,p_{(\mathbf{v},\mathbf{j})}}(t)-ϵ,U_{y,p_{(\mathbf{v},\mathbf{j})}}(t)+ϵ]\}$$

$$\subset \{{\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))\notin [{\overline{L}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)-ϵ,{\overline{U}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)+ϵ]\}$$

$$\cup \{{\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))\notin [{\underline{L}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)-ϵ,{\underline{U}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)+ϵ]\}.$$

$${\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))\le {\overline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}\le {\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))+ϵ$$

$${\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))-ϵ\le {\underline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}\le {\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t)).$$

$$\{{\mu }_y({\tilde{x}}_{p_{(\mathbf{v},\mathbf{j})}}(t))$$

$$\subset \{{\overline{\mu }}_{y,p_{(\mathbf{v},\mathbf{j})}}\notin [{\overline{L}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t),{\overline{U}}_{y,p_{(\mathbf{v},\mathbf{j})}}(t)]\},$$

Full text

I Introduction

AI-enabled technologies are becoming ubiquitous for many applications that involve repeated decision-making under uncertainty. Delivering personalized medicine for treatment of complex diseases [1], discovering and recommending interesting articles for a particular user from huge corpora of documents [2, 3] and optimizing hyper-parameters of deep learning architectures given a particular dataset [4] all require context-driven learning of optimal decisions over huge action sets. As the dimensionality of the contexts and the actions grow, learning the optimal decision for each context becomes a formidable task since what has been learned in the past cannot be used to accurately estimate the action rewards for the current context. Nevertheless, in many high-dimensional settings, only a subset of context and action dimensions affect the reward. For instance, in controlling the blood glucose of type 1 diabetes mellitus (T1DM) patients, data analysis highlights that future blood glucose of a patient only depends on blood glucose before the treatment, dose of the treatment and carbohydrate intake, whilst the affect of other physiological and environmental variables on blood glucose are found to be negligible [5, 6]. Similarly, when training deep neural networks, it is observed that in general not only a small subset of hyperparameters can be considered relevant, but also the content of relevant subset of hyperparameters differs from one task to another [7].

In this paper, we model online decision-making in high-dimensions as a multi-armed bandit (MAB) [8, 9]. MABs have successfully modeled a wide set of applications that involve sequential decision-making under uncertainty ranging from dynamic spectrum sharing [10, 11, 12] to medical diagnosis [13]. Specifically, we formalize the problem as a contextual MAB (CMAB) [14], where the learner observes a $d_{x}$-dimensional context from a context set ${\cal X}$ at the beginning of each round before selecting a $d_{a}$-dimensional action (arm) from an arm set ${\cal A}$.111In general, ${\cal X}$ and ${\cal A}$ have uncountably many elements. This generalizes the MAB model and allows the arms’ reward distributions depend on the context. The goal of the learner in this setting is to compete with an oracle that selects at each round the arm with the highest expected reward for the current context. The cumulative loss of the learner with respect to this oracle is called the regret, thereby minimizing the regret is equivalent to maximizing the cumulative expected reward. The learner’s time-averaged expected reward will approach to that of the oracle as long as it can keep its regret sublinearly growing over time. Being able to capture intricacies of data-driven decision-making, CMAB algorithms have been successfully used in recommender systems [15], personalized medicine [16] and cognitive communications [17].

Since the cardinalities of ${\cal X}$ and ${\cal A}$ are very large, further assumptions on the problem structure are required to obtain sublinear in time regret. In this paper, we consider a variant of CMAB with similarity information [14], where the reward from a context-arm pair comes from a fixed distribution, expected rewards vary smoothly in contexts and arms, and no stochastic assumptions are made on how contexts arrive over time.222Analysis holds for any fixed sequence of contexts. In this setting, dimensionality of the context and arm sets play a key role on the performance of learning algorithms [18]. In the worst-case, the regret has exponential dependence on $d_{x}$ and $d_{a}$, and thus, grows almost linearly in time in high-dimensional problems.

This motivates us to develop a new CMAB model and algorithm that address the learning challenges arising from high-dimensional context and arm sets. As discussed in the preceding paragraphs, in many applications of the CMAB, although the contexts and arms are high-dimensional, the most relevant information is embedded into a small number of relevant dimensions. Therefore, we consider a CMAB problem with similarity information where the expected reward only depends on relevant subcomponents of the arms and contexts. While the relevant subcomponent of the arms is fixed, the relevant subcomponent of the contexts can be different for each arm. For instance, in personalized treatment assignment, each arm can represent a drug cocktail and each component of an arm may correspond to the dose of a particular drug. Then, the relevance information tells that the outcome of the treatment only depends on a subset of relevant drugs in the cocktail and a subset of contexts of the patient (e.g., physiological data, genomic data) that are relevant to the drug cocktail. Minimizing the regret in this problem is extremely challenging since the learner knows neither the reward distributions nor what is relevant beforehand. All of these need to be learned online by only using the observed contexts, the selected arms and the random rewards observed from the selected arms in the past.

In this paper, we solve the problem described above by only assuming that the learner knows upper bounds $\overline{d}_{x}$ and $\overline{d}_{a}$ on the number of relevant context and arm dimensions. Essentially, we propose a new algorithm called CMAB with Relevance Learning (CMAB-RL) that learns the relevant context and arm dimensions to achieve $\tilde{O}(T^{1-1/(2+2\overline{d}_{x}+\overline{d}_{a})})$ regret, while on the other hand, CMAB algorithms that do not learn the relevance achieve $\tilde{O}(T^{1-1/(2+{d}_{x}+{d}_{a})})$ regret in the worst-case [18]. This implies that CMAB-RL has a better regret bound than these algorithms in terms of its dependence on time as long as $2\overline{d}_{x}<d_{x}$ is satisfied, and significantly improves over the prior work for sparse MAB problems, where $\overline{d}_{x}<<d_{x}$ and/or $\overline{d}_{a}<<d_{a}$.

The most closely related work to ours is [19], which considers a CMAB problem with finite number of arms, where the relevant context dimensions may vary from arm to arm. Provided with the same upper bound on the number of relevant context dimensions, the algorithm RELEAF in [19] is shown to achieve $\tilde{O}(T^{g(\bar{d}_{x})})$ regret, where $g(\bar{d}_{x})=(2+2\bar{d}_{x}+\sqrt{4{\bar{d}_{x}}^{2}+16\bar{d}_{x}+12})/(4+2\bar{d}_{x}+\sqrt{4{\bar{d}_{x}}^{2}+16\bar{d}_{x}+12})$. However, the setting in [19] is quite different from ours, since the authors assume that reward feedback is costly, and thus, needs to be acquired only when there is a need to explore. Therefore, their algorithm achieves a worse regret bound than CMAB-RL (the regret of CMAB-RL for this setting is $\tilde{O}(T^{(1+2\bar{d}_{x})/(2+2\bar{d}_{x})})$), because it needs to rely on control functions to either perform exploration or exploitation in each round, while CMAB-RL does not explicitly separate these two. Moreover, our formulation allows us to deal with high-dimensional and continuum sets of arms, which can be used in representing action sets for drug dosage, online auctions [20], routing [21], web-based recommendations [22] and web page content optimization [23].

In the core of CMAB-RL reside two new methods to identify and exploit relevance. The first one generates a collection of partitions of the context and arm sets formed by low-dimensional subsets of context and arm dimensions. This allows CMAB-RL to estimate rewards of context-arm pairs for only certain subsets of context and arm dimensions, thereby mitigating estimation errors caused by sparsity of similar samples that emerge from high-dimensionality. The second one identifies for each arm the candidate relevant tuples of context dimensions by comparing the variation of the sample mean rewards with confidence intervals constructed using selection statistics of related context-arm pairs. After identifying the candidate relevant tuples, CMAB-RL chooses the tuple with the minimum variation for each arm. Then, it uses the selected tuples to form reward estimates, and uses the principle of optimism in the face of uncertainty to minimize its regret.

Apart from the regret bounds, we also show the superiority of CMAB-RL as compared to other learning methods via extensive simulations on synthetic and real-world datasets. We model optimal personalized blood glucose control problem in T1DM patients for the first time (to the best of our knowledge) as a CMAB problem, where the contexts represent multimodal physiological data streams obtained from sensor readings and the arms represent bolus insulin doses that are appropriate for injection, and show that blood glucose control can be significantly improved by using our method.

In a nutshell, our main contribution is to design an online learning algorithm that can maximize the cumulative expected reward (minimize the regret) in sequential decision-making problems that involve high-dimensional and large context and arm sets with a sparse structure, where the expected reward is a (possibly) non-linear function of contexts and arms. While doing so, we do not make any assumptions on how contexts arrive over time as stochastic models may fail to accurately capture real-world phenomena that generate the contexts. Nevertheless, we show that time-averaged regret can be made arbitrarily small by utilizing the prior knowledge which states that similar contexts and actions should yield similar expected rewards.

The rest of the paper is organized as follows. Related work is given in Section II. CMAB and the regret are described in Section III. CMAB-RL is introduced in Section IV and its regret is analyzed in Section V. The effectiveness of learning the relevant dimensions is shown via simulations over (i) a high-dimensional synthetic dataset and (ii) a model created from real-world data collected from T1DM patients in Section VI. Concluding remarks are provided in Section VII and appendices, including tables of notation and auxiliary results, are given in the supplemental document.

II Related Work

Research relevant to our work can be categorized along two dimensions: related work in CMAB and related work in relevance learning and dimension reduction.

II-A Related Work in CMAB

CMAB has been studied under various assumptions on the relation between context-arm pairs and rewards. In the context of our work, prior art in CMAB can be categorized into three groups.

Problems in the first category (including our model) usually assume that there is an unknown but fixed reward distribution for every context-arm pair and the expected reward is a Lipschitz continuous function of the distance between context-arm pairs. Generally, for this category, no stochastic assumptions are made on the context arrivals. Under these assumptions, [18] proposes an algorithm that achieves $O(T^{1-1/(2+d_{c})+\epsilon})$ regret for any $\epsilon>0$ where $d_{c}$ is the covering dimension of the similarity space, i.e., the space of feasible context-arm pairs. The proposed algorithm partitions the similarity space and uses the past history in each set of the partition to form reward estimates of context-arm pairs within that particular set. It is also shown that a lower bound of order $\Omega(T^{1-1/(2+d_{p})-\epsilon})$ exists where $d_{p}$ is the packing dimension of the similarity space. Another related work [14] proposes an algorithm that adaptively divides the similarity space with the help of a covering oracle, essentially by “zooming” into regions where the context arrivals concentrate and arms provide high rewards, in order to perform high-precision exploration in these areas. It is shown that this algorithm achieves $\tilde{O}(T^{1-1/(2+d_{z})})$ regret where $d_{z}$ is the zooming dimension, which is linked to the covering dimension of the set of near-optimal context-arm pairs. The same problem is considered in [24] with a Gaussian process prior on the reward, and a CMAB algorithm that constructs a tree of partitions inspired by the HOO strategy in [25] is shown to achieve an optimal regret bound. To the best of our knowledge, the only other paper that considers relevance learning in this category is [19]. As noted in the introduction section, different from [19], we consider a high-dimensional arm set and provide improved regret bounds by constructing a novel method to test the relevance.

The second category works under the linearly realizability assumption. Here, contexts represent arm features and the expected reward of an arm is a linear function of its context. [15] proposes LinUCB algorithm for personalized news article recommendation, and [26] proves that a variant of LinUCB achieves $\tilde{O}(\sqrt{Td})$ regret, where $d$ is the dimension of the context. [27] extends these algorithms by introducing kernel functions, and shows that the proposed algorithm achieves $\tilde{O}(\sqrt{T\tilde{d}})$ regret, where $\tilde{d}$ represents the effective dimension of the kernel feature space. Notably, [28] provides an improved regret analysis for this problem by constructing more refined confidence sets. Sparsity in the context of linear CMAB is considered in [29] and [30]. In these works, sparsity corresponds to having arm weight vectors with many zero elements, as dimensions with zero weights have no effect on the expected reward. Similar to our setting, these works also assume prior knowledge on sparsity in terms of an upper bound on the number of relevant dimensions. Unlike sparse linear CMAB, we consider sparsity in a much more general environment, where the reward is allowed to be a non-linear function of arms and contexts. We only impose a mild Lipschitz continuity assumption (Assumption 1) on the expected reward, which allows our framework to be applicable to a much broader set of problems. We would also like to note that any linear bandit also satisfies the Lipschitz continuity assumption. Therefore, it can be said that [29] and [30] assume a much stronger prior knowledge on the form of the expected reward than our work.

The third category assumes that at each round the context and the arm rewards in that round are jointly drawn from a time-invariant distribution and the goal is to compete with the best policy in a given policy class. Among many works that fall into this category, [31] proposes the Epoch-Greedy algorithm that achieves $O(T^{2/3})$ regret. Follow-up works such as [32] and [33] propose improved algorithms with $\tilde{O}(T^{1/2})$ regret.

Apart from these, [34] considers that each element of the context comes from a binary distribution and proposes the Bandit Forest algorithm. This algorithm chooses relevant contexts and eliminates the irrelevant ones by using conditional probabilities. However, it considers only finitely many arms and contexts. Learning the optimal policy from a logged dataset with bandit feedback is considered in [35]. There, the authors identify the relevant context dimensions from logged data by constructing a relevance test that uses the importance sampling method. However, their method can only detect whether a context dimension is individually relevant or not.

In addition to these, [36] and [37] investigate non-contextual MAB with high-dimensional arms. Like our work, [36] assumes that only a subset of the arm dimensions are relevant and proposes a smart discretization of the arm set to achieve regret whose time order only depends on the number of relevant arm dimensions. On the other hand, [37] assumes that the expected reward is low-dimensional and smooth, and proposes an explore-then-exploit strategy that performs subspace identification followed by Bayesian optimization to minimize the regret. Methods in these works cannot be directly applied in our setting since we also need to take into account exogenously arriving contexts.

Table I lists the assumptions and regret bounds of the works that are most closely related to ours.

II-B Related Work in Relevance Learning and Dimension Reduction

Related work in relevance learning (or feature selection) mainly consists of offline methods. Similar to the related work in CMAB, offline feature selection can be categorized into three: Filter, wrapper and embedded approaches. In the embedded approach, feature selection is a part of the training procedure of a classifier. Wrapper methods select features based on the classifier’s feedback. In contrast, filter methods do not take classifier feedback into account, and select features based on intrinsic and statistical properties of the features such as correlations and marginal distributions. A plethora of papers exist for each approach. For the embedded approach, decision trees [38] and lasso based methods [39] are commonly used. As an example of the wrapper methods, Recursive Feature Elimination proposed in [40] iteratively trains the classifier, computes the ranking for each feature and removes the feature with smallest rank to find an optimal subset of the feature set. Examples of filter methods include feature weighting [41] and information-theoretic feature selection algorithms [42].

Online methods in feature selection can be seen as adaptations of offline methods. Due to computational efficiency, filter methods are generally preferred in the online framework [43]. For instance, [44] proposes a method called Online Streaming Feature Selection (OSFS). This algorithm divides the feature set into three disjoint sets: strongly relevant, weakly relevant and irrelevant. OSFS works in two phases. In the first phase, it learns strongly and weakly relevant features and eliminates irrelevant features. In the second phase, features that are relevant but redundant due to correlations with the other features are eliminated. While there is an abundance of literature in online feature selection (see e.g., [45] and references therein), they do not fit into the CMAB setting where the goal is to learn the relevant features in order to minimize the regret. Moreover, these works try to identify a fixed set of relevant features, while in our case the set of relevant context dimensions may differ among arms.

III Problem Formulation

The system operates in rounds indexed by $t\in\{1,2,\ldots\}$. At the beginning of each round, the learner observes a context $x(t)$ that comes from a ${d}_{x}$-dimensional context set ${\cal X}\coloneqq[0,1]^{{d}_{x}}$, and then, chooses an arm $a(t)$ from a ${d}_{a}$-dimensional arm set ${\cal A}\coloneqq[0,1]^{{d}_{a}}$. The set of feasible context-arm pairs is denoted by ${\cal F}\coloneqq{\cal X}\times{\cal A}$. The random reward obtained from playing arm $a(t)$ in round $t$ is given as $r(t)\coloneqq\mu_{a(t)}(x(t))+\kappa(t)$, where $\mu_{a}(x)$ denotes the expected reward of a context-arm pair $(x,a)\in{\cal F}$ and $\kappa(t)$ is the noise process whose marginal distribution is conditionally 1-sub-Gaussian, i.e. $\forall\lambda\in\mathbb{R}$

[TABLE]

where for $b\in\{a,x,\kappa\}$, $b_{1:t}\coloneqq(b(1),\ldots b(t))$.

Let ${\cal D}_{a}\coloneqq\{1,\ldots,d_{a}\}$ denote the set of arm dimensions. For any $\boldsymbol{z}\subseteq{\cal D}_{a}$, ${\cal A}_{\boldsymbol{z}}\coloneqq[0,1]^{|\boldsymbol{z}|}$ denotes the subset of ${\cal A}$ that contains the values of arm dimensions in $\boldsymbol{z}$ and for any $a\in{\cal A}$, $a_{\boldsymbol{z}}\in{\cal A}_{\boldsymbol{z}}$ denotes the $|\boldsymbol{z}|$-tuple subarm whose elements are elements of $a$ that correspond to the arm dimensions in $\boldsymbol{z}$. For any $\boldsymbol{z}\subseteq{\cal D}_{a}$ and $\boldsymbol{z}^{\prime}={\cal D}_{a}\setminus\boldsymbol{z}$, we write $a=\{a_{\boldsymbol{z}},a_{\boldsymbol{z}^{\prime}}\}$. Let $\boldsymbol{c}$ denote the subset of ${\cal D}_{a}$ that contains the relevant arm dimensions, i.e. $\forall\boldsymbol{z}\subseteq{\cal D}_{a}\setminus\boldsymbol{c}$, $\forall a_{\boldsymbol{z}},a^{\prime}_{\boldsymbol{z}}\in{\cal A}_{\boldsymbol{z}}$, $\forall a_{{\cal D}_{a}\setminus\boldsymbol{z}}\in{\cal A}_{{\cal D}_{a}\setminus\boldsymbol{z}}$ and $\forall x\in{\cal X}$, we have $\mu_{\{a_{\boldsymbol{z}},a_{{\cal D}_{a}\setminus\boldsymbol{z}}\}}(x)=\mu_{\{a^{\prime}_{\boldsymbol{z}},a_{{\cal D}_{a}\setminus\boldsymbol{z}}\}}(x)$.

Similarly, let ${\cal D}_{x}\coloneqq\{1,\ldots,d_{x}\}$ denote the set of context dimensions. For any $\boldsymbol{z}\subseteq{\cal D}_{x}$, ${\cal X}_{\boldsymbol{z}}\coloneqq[0,1]^{|\boldsymbol{z}|}$ denotes the subset of ${\cal X}$ that contains values of the context dimensions in $\boldsymbol{z}$ and for any $x\in{\cal X}$, $x_{\boldsymbol{z}}\in{\cal X}_{\boldsymbol{z}}$ denotes the $|\boldsymbol{z}|$-tuple subcontext whose elements are elements of $x$ that correspond to the context dimensions in $\boldsymbol{z}$. For any $\boldsymbol{z}\subseteq{\cal D}_{x}$ and $\boldsymbol{z}^{\prime}={\cal D}_{x}\setminus\boldsymbol{z}$, we write $x=\{x_{\boldsymbol{z}},x_{\boldsymbol{z}^{\prime}}\}$. Since relevant context dimensions may be different for different arms, for any $a\in{\cal A}$, let $\boldsymbol{c}_{a}$ denote the subset of ${\cal D}_{x}$ that contains the relevant context dimensions, i.e. $\forall a\in{\cal A}$, $\forall\boldsymbol{z}\subseteq{\cal D}_{x}\setminus\boldsymbol{c}_{a}$, $\forall x_{\boldsymbol{z}},x^{\prime}_{\boldsymbol{z}}\in{\cal X}_{\boldsymbol{z}}$ and $\forall x_{{\cal D}_{x}\setminus\boldsymbol{z}}\in{\cal X}_{{\cal D}_{x}\setminus\boldsymbol{z}}$, we have $\mu_{a}(\{x_{\boldsymbol{z}},x_{{\cal D}_{x}\setminus\boldsymbol{z}}\})=\mu_{a}(\{x^{\prime}_{\boldsymbol{z}},x_{{\cal D}_{x}\setminus\boldsymbol{z}}\})$.

For a given context $x$, the optimal arm is defined as $a^{*}(x)\coloneqq\operatorname*{arg\,max}_{a\in{\cal A}}\mu_{a}(x)$. Since there are infinitely many arms and contexts, it is impossible to learn the optimal arm for each context without any further assumptions on the expected rewards. To overcome this issue, the following assumption provides a similarity structure on the expected rewards with respect to the set of context-arm pairs, which is a modified version of the Lipschitz continuity assumption commonly used in the contextual MAB literature [14]. It states that the variation of the expected reward between two context-arm pairs is bounded by the distance between the context-arm pairs in the relevant dimensions.

Assumption 1.

$\exists L>0$* such that $\forall a,a^{\prime}\in{\cal A}$ and $x,x^{\prime}\in{\cal X}$, we have*

[TABLE]

where $\lVert.\rVert$ represents the Euclidean norm.

Assumption 1 also implies that

[TABLE]

We assume that the learner knows $L$ given in Assumption 1, but does not know $\mu_{a}(x)$, $a\in{\cal A}$, $x\in{\cal X}$. To evaluate the performance of the learner given an arbitrary sequence of contexts $x_{1:T}$, we adopt the commonly used (pseudo) regret notion, given as

[TABLE]

Note that $\text{Reg}(T)$ is a random variable since $a(t)$ itself depends on the learning algorithm and its observations. In essence, $\text{Reg}(T)$ compares the expected reward accumulated by the learner with that of the oracle. Our goal is to design a learning algorithm to minimize the regret. Algorithms that do not take relevant dimensions into account (see, e.g. [18]) will achieve $\tilde{O}(T^{1-1/(2+d_{x}+d_{a})})$ regret in the worst-case. On the other hand, our algorithm CMAB-RL achieves $\tilde{O}(T^{1-1/(2+2\overline{d}_{x}+\overline{d}_{a})})$ regret where $\overline{d}_{x}$ and $\overline{d}_{a}$ are known upper bounds on the number of relevant context and arm dimensions: $\underline{d}_{x}:=\max_{a\in{\cal A}}|\boldsymbol{c}_{a}|\leq\overline{d}_{x}$ and $\underline{d}_{a}=|\boldsymbol{c}|\leq\overline{d}_{a}$. This shows that when $2\overline{d}_{x}+\overline{d}_{a}<d_{x}+d_{a}$, CMAB-RL achieves better regret compared to the algorithms that do not exploit the relevance structure. Thus, in the rest of the paper, we assume that $2\overline{d}_{x}\leq d_{x}$. Note that we do not require existence of a unique low-dimensional subspace of ${\cal F}$ that captures all the relevance, since it is possible that $\cup_{a\in{\cal A}}\boldsymbol{c}_{a}={\cal D}_{x}$.

IV The Learning Algorithm

Our algorithm, called CMAB with Relevance Learning (CMAB-RL), is described in Algorithms 1 and 2. CMAB-RL is a CMAB algorithm that optimizes itself by generating supersets of the relevant context and arm dimensions with sizes $2\overline{d}_{x}$ and $\overline{d}_{a}$. The main step in learning relevance is to form a set of candidate dimensions (tuples) that contains the relevant dimensions with a high probability. Past observations that fall into these tuples are then used to estimate expected rewards of the arms, which results in highly accurate estimates when the tuples that contain the relevant dimensions are correctly identified.

For any $l\in\mathbb{Z}^{+}$, let ${\cal V}^{l}_{x}$ denote the set of all $l$-tuples of context dimensions, i.e. ${\cal V}^{l}_{x}\coloneqq\{\boldsymbol{v}\in\wp({\cal D}_{x}):|\boldsymbol{v}|=l\}$ where $\wp({{\cal D}_{x}})$ denotes the power set (set of all subsets) of ${{\cal D}_{x}}$. Similarly for any $l\in\mathbb{Z}^{+}$, let ${\cal V}^{l}_{a}$ denote the set of all $l$-tuples of arm dimensions. For $\boldsymbol{v}\subseteq{\cal D}_{x}$ and $l\in\{|\boldsymbol{v}|,|\boldsymbol{v}|+1,\ldots,\overline{d}_{x}\}$, let ${\cal V}^{l}_{x}(\boldsymbol{v})$ denote the set of all $l$-tuples of context dimensions that contain $\boldsymbol{v}$, i.e. if we have $\boldsymbol{w}\in{\cal V}^{l}_{x}(\boldsymbol{v})$, then $\boldsymbol{v}\subseteq\boldsymbol{w}$ is satisfied.

At the beginning, CMAB-RL takes as inputs the context set ${\cal X}$, the arm set ${\cal A}$, the total number of rounds $T$, $L$ given in Assumption 1, the partition number $m$ (which will be optimized later), an integer that is an upper bound on the number of relevant arm dimensions $\overline{d}_{a}\leq d_{a}$ and an integer that is an upper bound on the number of relevant context dimensions $\overline{d}_{x}\leq d_{x}/2$. CMAB-RL uses Assumption 1 to learn together for similar arms and similar contexts. This is achieved by properly discretizing the arm and context sets. In its initialization phase, CMAB-RL generates a discretized arm set ${\cal Y}\subseteq{\cal A}$ and a collection of partitions of ${\cal X}$, denoted by $C({\cal X})$ using the Generate subroutine given in Algorithm 2.

Next, we describe this initialization process in detail. CMAB-RL first generates the set ${\cal V}^{\overline{d}_{a}}_{a}$. For all $\boldsymbol{v}\in{\cal V}^{\overline{d}_{a}}_{a}$, each dimension of the arm subset ${\cal A}_{\boldsymbol{v}}$ is partitioned into $m$ intervals with equal lengths. Letting ${\cal I}_{i}:=\{[0,\frac{1}{m}],(\frac{1}{m},\frac{2}{m}],\ldots,(\frac{m-1}{m},1]\}$ denote the partition of the arm subset in dimension $i$, ${\cal I}_{\boldsymbol{v}}:=\prod_{i\in\boldsymbol{v}}{\cal I}_{i}$ forms a partition of ${\cal A}_{\boldsymbol{v}}$ into $m^{\overline{d}_{a}}$ non-overlapping sets. The collection of partitions of the $\overline{d}_{a}$-dimensional subsets of the arm set formed this way is denoted by ${\cal C}({\cal A}):=\cup_{\boldsymbol{v}\in{\cal V}^{\overline{d}_{a}}_{a}}{\cal I}_{\boldsymbol{v}}$. Note that ${\cal C}({\cal A})$ contains $\genfrac{(}{)}{0.0pt}{1}{d_{a}}{\overline{d}_{a}}m^{\overline{d}_{a}}$ sets. We index the geometric centers of these sets by $y$, and the set of arms that correspond to these centers is denoted by ${\cal Y}$. For an arm that corresponds to the geometric center of a set in ${\cal I}_{\boldsymbol{v}}$, values of the dimensions of that arm in $i\in{\cal D}_{a}\setminus\boldsymbol{v}$ are set as $0.5$.333$0.5$ is chosen for convenience. Indeed, any value in $[0,1]$ will work.

Similarly, CMAB-RL also generates the set ${\cal V}^{2\overline{d}_{x}}_{x}$. For all $\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}$ each dimension of the context subset ${\cal X}_{\boldsymbol{w}}$ is partitioned into $m$ intervals with equal lengths. Letting ${\cal P}_{i}:=\{[0,\frac{1}{m}],(\frac{1}{m},\frac{2}{m}],\ldots,(\frac{m-1}{m},1]\}$ denote the partition of the context subset in dimension $i$, ${\cal P}_{\boldsymbol{w}}:=\prod_{i\in\boldsymbol{w}}{\cal P}_{i}$ forms a partition of ${\cal X}_{\boldsymbol{w}}$ into $m^{2\overline{d}_{x}}$ non-overlapping sets. The collection of partitions of the $2\overline{d}_{x}$-dimensional subsets of the context set formed this way is denoted by ${\cal C}({\cal X}):=\cup_{\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}}{\cal P}_{\boldsymbol{w}}$. Note that ${\cal C}({\cal X})$ contains $\genfrac{(}{)}{0.0pt}{1}{d_{x}}{2\overline{d}_{x}}m^{2\overline{d}_{x}}$ sets.

For simplicity of notation, for any $x\in{\cal X}$ if $x_{\boldsymbol{w}}\in p_{\boldsymbol{w}}$ for $p_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$, then we say that $x\in p_{\boldsymbol{w}}$ for $\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}$. Also, we let $p_{\boldsymbol{w}}(t)\in{\cal P}_{\boldsymbol{w}}$ denote the set that $x_{\boldsymbol{w}}(t)$ belongs to.

For each $\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}$, $p_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$ and $y\in{\cal Y}$, CMAB-RL stores a counter $N_{y,p_{\boldsymbol{w}}}(t)$ that counts the number of times context was in $p_{\boldsymbol{w}}$ and arm $y$ was selected before round $t$, and the sample mean of the rewards $\hat{\mu}_{y,p_{\boldsymbol{w}}}(t)$ that is obtained from rounds prior to round $t$ in which context was in $p_{\boldsymbol{w}}$ and arm $y$ was selected. In order to define the arm selection rule, CMAB-RL also needs to calculate another statistic, called the uncertainty term, which is defined for all $\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}$, $p_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$, $y\in{\cal Y}$ as $u_{y,p_{\boldsymbol{w}}}(t)\coloneqq\sqrt{(2+4\log(2|{\cal Y}|\overline{C}m^{2\overline{d}_{x}}T^{3/2}))/N_{y,p_{\boldsymbol{w}}}(t)}$, where $\overline{C}\coloneqq\genfrac{(}{)}{0.0pt}{1}{d_{x}-1}{2\overline{d}_{x}-1}$. For simplicity of notation, we use $\hat{\mu}_{y,\boldsymbol{w}}(t)\coloneqq\hat{\mu}_{y,p_{\boldsymbol{w}}(t)}(t)$, $u_{y,\boldsymbol{w}}(t)\coloneqq u_{y,p_{\boldsymbol{w}}(t)}(t)$ and $N_{y,\boldsymbol{w}}(t)\coloneqq N_{y,p_{\boldsymbol{w}}(t)}(t)$, since in each round $t$ there exists only one $p_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$ such that $x_{\boldsymbol{w}}(t)\in p_{\boldsymbol{w}}$. Based on this, the sample mean reward of arm $y\in{\cal Y}$ for the tuple of context dimensions $\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}$ in round $t$ is defined as

[TABLE]

At the beginning of round $t$, CMAB-RL first observes the context $x(t)$. Then, for each $\boldsymbol{w}\in{\cal V}_{x}^{2\overline{d}_{x}}$, it identifies the set $p_{\boldsymbol{w}}(t)$ in ${\cal P}_{\boldsymbol{w}}$ that $x(t)$ belongs to. Using this information and the sample mean rewards, it generates the set of candidate relevant tuples of context dimensions for each $y\in{\cal Y}$ as follows:

[TABLE]

Here, the term $2L\sqrt{\overline{d}_{x}}/m+u_{y,\boldsymbol{w}}(t)+u_{y,\boldsymbol{w}^{\prime}}(t)$ accounts for the joint uncertainty over the sample mean rewards of arm $y$ calculated using observations in $p_{\boldsymbol{w}}(t)$ and $p_{\boldsymbol{w}^{\prime}}(t)$. If the absolute difference between the sample mean rewards is larger than the joint uncertainty term, we can say that the subset of relevant context dimensions that is in tuple $\boldsymbol{w}$ is different from the subset of relevant context dimensions that is in tuple $\boldsymbol{w}^{\prime}$ with high probability. Since $\boldsymbol{v}\subset\boldsymbol{w}$ and $\boldsymbol{v}\subset\boldsymbol{w}^{\prime}$, this implies that $\boldsymbol{v}$ does not contain all relevant context dimensions. Therefore, the tuple $\boldsymbol{v}$ is not included in the set of candidate relevant tuples of context dimensions ${\cal R}_{y}(t)$.

Let $\hat{\boldsymbol{c}}_{y}(t)$ denote the tuple of estimated relevant context dimensions for arm $y$ in round $t$. If ${\cal R}_{y}(t)$ is empty, then CMAB-RL selects $\hat{\boldsymbol{c}}_{y}(t)$ from ${\cal V}_{x}^{\overline{d}_{x}}$ randomly. Otherwise, to compute $\hat{\boldsymbol{c}}_{y}(t)$, CMAB-RL calculates the variation of the sample mean rewards for every $\boldsymbol{v}\in{\cal R}_{y}(t)$ as follows:

[TABLE]

After calculating the variation, CMAB-RL chooses $\hat{\boldsymbol{c}}_{y}(t)$ for all $y\in{\cal Y}$ as $\hat{\boldsymbol{c}}_{y}(t)=\operatorname*{arg\,min}_{\boldsymbol{v}\in{\cal R}_{y}(t)}\hat{\sigma}^{2}_{y,\boldsymbol{v}}(t)$. Then, using $\hat{\boldsymbol{c}}_{y}(t)$, CMAB-RL calculates $\mu_{y}^{\hat{\boldsymbol{c}}_{y}(t)}(t)$ for all $y\in{\cal Y}$. To select an arm from ${\cal Y}$, CMAB-RL uses the principle of optimism under the face of uncertainty. The estimated rewards of the context-arm pairs are inflated by a certain level, such that the inflated reward estimates become an upper confidence bound (UCB) for the expected reward with high probability. Denote the $2\overline{d}_{x}$-tuple of context dimensions with the highest uncertainty term for arm $y$ in round $t$ by $\boldsymbol{w}_{y}(t)\coloneqq\operatorname*{arg\,max}_{\boldsymbol{w}^{\prime}\in{\cal V}^{2\overline{d}_{x}}_{x}}u_{y,\boldsymbol{w}^{\prime}}(t)$ (where ties are broken randomly). UCB of arm $y\in{\cal Y}$ at time $t$ is calculated as

[TABLE]

Then, CMAB-RL selects the arm with the highest UCB, i.e. $y(t)=\operatorname*{arg\,max}_{y\in{\cal Y}}\text{UCB}_{y}(t)$. This forces the arms that are rarely selected by CMAB-RL to get explored (since they have high uncertainty) while balancing the trade-off between exploration and exploitation. After selecting arm $y(t)$, CMAB-RL observes the reward $r(t)$ and updates the parameters for arm $y(t)$ for all $\boldsymbol{w}\in{\cal V}_{x}^{2\overline{d}_{x}}$ as follows:

[TABLE]

In addition, for $y\in{\cal Y}\setminus y(t)$, $\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}$ and $p_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$, we have $N_{y,p_{\boldsymbol{w}}}(t+1)=N_{y,p_{\boldsymbol{w}}}(t)$, $\hat{\mu}_{y,p_{\boldsymbol{w}}}(t+1)=\hat{\mu}_{y,p_{\boldsymbol{w}}}(t)$, hence these values remain unchanged. Please refer to Appendix B in the supplemental document for the analysis of memory and computational complexities of CMAB-RL.

Remark 1.

After a simple modification, CMAB-RL can also work when it is restricted to make choices from a given finite set of arms ${\cal A}_{f}$, which is a subset of the $d_{a}$-dimensional arm set ${\cal A}$. For this, it will first identify sets in ${\cal C}(A)$ that contain at least one arm in ${\cal A}_{f}$. Let ${\cal C}_{f}(A)$ represent the collection of such sets. For each set in ${\cal C}_{f}(A)$, CMAB-RL will pick a unique arm from ${\cal C}_{f}(A)$ and include it in ${\cal Y}$. By this construction, all arms in ${\cal Y}$ will be from ${\cal A}_{f}$. After initializing the arm set ${\cal Y}$ this way, CMAB-RL will compute and update UCB indices for these arms in the same way as the original algorithm.

V Regret Analysis

We first state and discuss our main result, and then, present the technical details.

V-A The Main Result

Our main result is given in the following theorem.

Theorem 1.

Given an arbitrary fixed sequence of contexts $x_{1:T}$, when CMAB-RL is run with $m=\lceil T^{1/(2+2\overline{d}_{x}+\overline{d}_{a})}\rceil$, we have with probability at least $1-1/T$

[TABLE]

where $\tilde{T}=(T^{1/(2+2\overline{d}_{x}+\overline{d}_{a})}+1)^{2+2\overline{d}_{x}+\overline{d}_{a}}$ and $C_{\max}\coloneqq\max_{y,y^{\prime}\in{\cal Y},x\in{\cal X}}(\mu_{y^{\prime}}(x)-\mu_{y}(x))$.

Importantly, Theorem 1 says that CMAB-RL incurs $\tilde{O}(T^{1-1/(2+2\overline{d}_{x}+\overline{d}_{a})})$ regret with probability at least $1-1/T$ when it is run with $m=\lceil T^{1/(2+2\overline{d}_{x}+\overline{d}_{a})}\rceil$. A standard doubling trick argument [25] can be used to make the algorithm anytime (does not require $T$ as input) while preserving the order of the regret. As a side result, this sublinear regret bound also implies average reward optimality of CMAB-RL. On the other hand, classical CMAB algorithms that do not exploit the relevance structure achieve $\tilde{O}(T^{1-1/(2+{d}_{x}+{d}_{a})})$ regret in the worst-case [18]. Thus, when $2\overline{d}_{x}+\overline{d}_{a}<d_{x}+d_{a}$, CMAB-RL achieves a better regret order compared to the classical CMAB algorithms. As noted before, for the finite-armed version of our problem, RELEAF [19] achieves $\tilde{O}(T^{g(\bar{d}_{x})})$ regret for $g(\bar{d}_{x})=(2+2\bar{d}_{x}+\sqrt{4{\bar{d}_{x}}^{2}+16\bar{d}_{x}+12})/(4+2\bar{d}_{x}+\sqrt{4{\bar{d}_{x}}^{2}+16\bar{d}_{x}+12})$, while our regret bound for this case becomes $\tilde{O}(T^{(1+2\bar{d}_{x})/(2+2\bar{d}_{x})})$, which is strictly better than that of RELEAF. As a final remark, we would also like to note that if $\boldsymbol{c}_{a}$ is fixed for all $a\in{\cal A}$, then it is possible to construct a strategy based on Exp4 [46] that achieves $\tilde{O}(T^{1-1/(2+\overline{d}_{x}+\overline{d}_{a})})$ regret even though it requires defining an infeasible number of experts (see Appendix C in the supplemental document for details). In addition to assuming that the set of relevant context dimensions is the same for each arm, when the set of relevant context and arm dimensions are known (which is not the case in our work), an obvious lower bound on the worst-case regret would be $\Omega(T^{1-1/(2+\overline{d}_{x}+\overline{d}_{a})})$ [18]. It is therefore an interesting future research direction to close the gap between this lower bound and our upper bound.

Remark 2.

It is also possible to consider a joint upper bound $\bar{d}_{z}$ on the number of relevant context and arm dimensions. In this case, since the learner does not know how many of these dimensions correspond to contexts or arms, it needs to consider all possible ways how $\bar{d}_{z}$-dimensions can be split between context and arms. Two extreme non-trivial cases are $(\bar{d}_{x}=\bar{d}_{z}-1,\bar{d}_{a}=1)$ and $(\bar{d}_{x}=0,\bar{d}_{a}=\bar{d}_{z})$. Note that the case when $(\bar{d}_{x}=\bar{d}_{z},\bar{d}_{a}=0)$ is trivial as all arms in this case will yield the same expected reward for a given context, i.e., all arms are equally well and there is no need for learning. Thus, if only given $\bar{d}_{z}$, then the learner can set $\bar{d}_{x}=\bar{d}_{z}-1$ and $\bar{d}_{a}=\bar{d}_{z}$ in CMAB-RL. Based on Theorem 1, this will result in a regret bound of $\tilde{O}(T^{1-1/(3\bar{d}_{z})})$ when $2(\bar{d}_{z}-1)\leq d_{x}$, which is still sublinear in $T$.

We end this subsection by giving a high-level explanation of the proof Theorem 1. To prove Theorem 1, as the first step, we construct contextual variants of the tight confidence sets derived from analysis of self-normalized martingale processes [28]. We build our analysis over concentration of these sets (intervals in our case) for the tuples that contain the relevant context dimensions. Our first result (Lemma 1) indicates that the confidence intervals remain reasonably small over all rounds with a high probability. The rest of our analysis focuses on what happens under this high probability event. For instance, defining the relevance test as given in (1) ensures that all $\bar{d}_{x}$-tuples of context dimensions that include the relevant context dimensions pass the test (Lemma 2), and this further guarantees that the estimated reward of each arm concentrates around its true mean value for the current context (Lemma 3). As a result of this, the UCB index used by CMAB-RL to select its arm ensures that the suboptimality gap of the selected arm is proportional to its uncertainty term (Lemma 4). As the uncertainty of an arm for the current context decreases every time that arm is selected, as time goes on, we conclude that the suboptimality gaps of the selected arms go to zero, which when summed over all rounds, gives us the worst-case regret bound. Technical details of the proof can be found in the next subsection.

V-B Proof of Theorem 1

We start by introducing the notation. For an event ${\cal H}$, let ${\cal H}^{c}$ denote its complement. For any $\boldsymbol{w}\in{\cal V}_{x}^{2\overline{d}_{x}}$ and ${p}_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$, let $N_{{p}_{\boldsymbol{w}}}(t)$ denote the number of context arrivals to ${p}_{\boldsymbol{w}}$ by the end of round $t$, $\tau_{{p}_{\boldsymbol{w}}}(t)$ denote the round in which a context arrives to ${p}_{\boldsymbol{w}}$ for the $t$th time and $R_{y}(t)$ denote the random reward of arm $y$ in round $t$.

For any $\boldsymbol{w}\in{\cal V}_{x}^{2\overline{d}_{x}}$, ${p}_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$ and $y\in{\cal Y}$ let $\tilde{x}_{{p}_{\boldsymbol{w}}}(t):=x(\tau_{{p}_{\boldsymbol{w}}}(t))$, $\tilde{R}_{y,p_{\boldsymbol{w}}}(t):=R_{y}(\tau_{{p}_{\boldsymbol{w}}}(t))$, $\tilde{N}_{y,{p}_{\boldsymbol{w}}}(t):=N_{y,{p}_{\boldsymbol{w}}}(\tau_{{p}_{\boldsymbol{w}}}(t))$, $\tilde{\mu}_{y,{p}_{\boldsymbol{w}}}(t):=\hat{\mu}_{y,{p}_{\boldsymbol{w}}}(\tau_{{p}_{\boldsymbol{w}}}(t))$, $\tilde{u}_{y,{p}_{\boldsymbol{w}}}(t):=u_{y,{p}_{\boldsymbol{w}}}(\tau_{{p}_{\boldsymbol{w}}}(t))$. $\tilde{y}_{{p}_{\boldsymbol{w}}}(t):=y(\tau_{{p}_{\boldsymbol{w}}}(t))$ and $\tilde{\kappa}_{{p}_{\boldsymbol{w}}}(t)=\kappa(\tau_{{p}_{\boldsymbol{w}}}(t))$.

For any $\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}$ and $d^{\prime}\leq d_{x}-\overline{d}_{x}$, $d^{\prime}\in\mathbb{Z}^{+}$, let ${\cal V}_{x}(\boldsymbol{v},d^{\prime})$ be the set of $d^{\prime}$-tuples of context dimensions whose elements are from the set ${\cal D}_{x}\setminus\boldsymbol{v}$. Hence, for any $\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}$ and $\boldsymbol{j}\in{\cal V}_{x}(\boldsymbol{v},d^{\prime})$, $(\boldsymbol{v},\boldsymbol{j})$ denotes a ($\overline{d}_{x}+d^{\prime}$)-tuple of context dimensions.

For any $y\in{\cal Y}$, $\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}(\boldsymbol{c}_{y})$, $\boldsymbol{j}\in{\cal V}_{x}(\boldsymbol{v},\overline{d}_{x})$ and $p_{(\boldsymbol{v},\boldsymbol{j})}\in{\cal P}_{(\boldsymbol{v},\boldsymbol{j})}$ we define the following lower and upper bounds: $L_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t):=\tilde{\mu}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)-\tilde{u}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)$ and $U_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t):=\tilde{\mu}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)+\tilde{u}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)$.

For $\epsilon=L\left(\sqrt{\overline{d}_{x}}/m\right)$, $y\in{\cal Y}$, $\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}(\boldsymbol{c}_{y})$, $\boldsymbol{j}\in{\cal V}_{x}(\boldsymbol{v},\overline{d}_{x})$ and $p_{(\boldsymbol{v},\boldsymbol{j})}\in{\cal P}_{(\boldsymbol{v},\boldsymbol{j})}$, let

[TABLE]

denote the event that the learner is not confident about its reward estimate for at least once in time steps in which the contexts is in ${p}_{(\boldsymbol{v},\boldsymbol{j})}$ by round $T$. Also, let $\text{UC}_{y,(\boldsymbol{v},\boldsymbol{j})}\coloneqq\cup_{{p}_{(\boldsymbol{v},\boldsymbol{j})}\in{{\cal P}}_{(\boldsymbol{v},\boldsymbol{j})}}\text{UC}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}},\text{UC}_{(\boldsymbol{v},\boldsymbol{j})}\coloneqq\cup_{y\in{\cal Y}}\text{UC}_{y,(\boldsymbol{v},\boldsymbol{j})}$ and

[TABLE]

Similarly for any $y\in{\cal Y}$, $\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}(\boldsymbol{c}_{y}),\boldsymbol{j}\in{\cal V}_{x}(\boldsymbol{v},\overline{d}_{x})$ and $p_{(\boldsymbol{v},\boldsymbol{j})}\in{\cal P}_{(\boldsymbol{v},\boldsymbol{j})}$, let

[TABLE]

The following lemma states that UC occurs with a small probability.

Lemma 1.

[TABLE]

Proof.

Let $\{\tilde{R}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)\}_{t=1}^{N_{{p}_{(\boldsymbol{v},\boldsymbol{j})}}(T)}$ denote the sequence of rewards observed from arm $y$ in time steps when the context is in ${p}_{(\boldsymbol{v},\boldsymbol{j})}$. We can express the sample mean reward of $y$ as

[TABLE]

for $\tilde{N}_{y,p_{(\boldsymbol{v},\boldsymbol{j})}}(t)>0$, where $\textbf{I}(\cdot)$ is the indicator function. When $\tilde{N}_{y,p_{(\boldsymbol{v},\boldsymbol{j})}}(t)=0$ we have $\tilde{\mu}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)=0$. We also have

[TABLE]

where $\{\tilde{\kappa}_{{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)\}_{t=1}^{N_{{p}_{(\boldsymbol{v},\boldsymbol{j})}}(T)}$ is a sequence of zero mean $1$-sub-Gaussian random variables. We define two new sequences of random variables, whose sample mean values will lower and upper bound $\tilde{\mu}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)$. The best sequence is defined as $\{\bar{R}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)\}_{t=1}^{N_{{p}_{(\boldsymbol{v},\boldsymbol{j})}}(T)}$ where

[TABLE]

and the worst sequence is defined as $\{\underline{R}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)\}_{t=1}^{N_{{p}_{(\boldsymbol{v},\boldsymbol{j})}}(T)}$ where

[TABLE]

Let

[TABLE]

for $\tilde{N}_{y,p_{(\boldsymbol{v},\boldsymbol{j})}}(t)>0$. When $\tilde{N}_{y,p_{(\boldsymbol{v},\boldsymbol{j})}}(t)=0$ we have $\overline{\mu}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)=\underline{\mu}_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(t)=0$. Since $\boldsymbol{v}\in{\cal V}^{\overline{d}_{x}}_{x}(\boldsymbol{c}_{y})$, we have $\forall t\in\{1,\ldots,N_{{p}_{(\boldsymbol{v},\boldsymbol{j})}}(T)\}$

[TABLE]

almost surely. Let

[TABLE]

Then, we have

[TABLE]

The following inequalities are obtained using Assumption 1 since $\boldsymbol{v}\in{\cal V}^{\overline{d}_{x}}_{x}(\boldsymbol{c}_{y})$:

[TABLE]

Using (4) and (5) it can be shown that

[TABLE]

Plugging this to (3), we get

[TABLE]

Using the equation above and the union bound we obtain

[TABLE]

Both terms on the right-hand side of the inequality above can be bounded using the concentration inequality in Appendix D in the supplemental document by setting $\delta=1/(2|{\cal Y}|\overline{C}m^{2\overline{d}_{x}}T)$:

[TABLE]

since $1+N_{y,{p}_{(\boldsymbol{v},\boldsymbol{j})}}(T)\leq T$. Finally, the union bound gives us $\Pr(\text{UC})\leq 1/T$. ∎

The next lemma states that ${\cal R}_{y}(t)\neq\emptyset$ for all $y\in{\cal Y}$ on event $\text{UC}^{c}$.

Lemma 2.

On event $\text{UC}^{c}$, $\forall y\in{\cal Y}$, $\forall\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}(\boldsymbol{c}_{y})$ and $\forall t\in\{1,\ldots,T\}$, we have $\boldsymbol{v}\in{\cal R}_{y}(t)$.

Proof.

$\forall y\in{\cal Y}$, $\forall\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}(\boldsymbol{c}_{y})$ and $\forall\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}(\boldsymbol{v})$, we have $\boldsymbol{w}\supset\boldsymbol{c}_{y}$, since $\boldsymbol{w}\supset\boldsymbol{v}$. By definition of UC, on event $\text{UC}^{c}$, $\forall t\in\{1,\ldots,T\}$, we have $|\hat{\mu}_{y,\boldsymbol{w}}(t)-\mu_{y}(x(t))|\leq\epsilon+u_{y,\boldsymbol{w}}(t)$. Thus, $\forall\boldsymbol{w},\boldsymbol{w}^{\prime}\in{\cal V}^{2\overline{d}_{x}}_{x}(\boldsymbol{v})$, we obtain $|\hat{\mu}_{y,\boldsymbol{w}}(t)-\hat{\mu}_{y,\boldsymbol{w}^{\prime}}(t)|\leq 2\epsilon+u_{y,\boldsymbol{w}}(t)+u_{y,\boldsymbol{w}^{\prime}}(t)$ and consequently, we have $\boldsymbol{v}\in{\cal R}_{y}(t)$ by definition of ${\cal R}_{y}(t)$. ∎

The next lemma shows that the difference between estimated and expected rewards of an arm is small on event $\text{UC}^{c}$.

Lemma 3.

On event $\text{UC}^{c}$, for all $y\in{\cal Y}$ and $t\in\{1,\ldots,T\}$ we have

[TABLE]

Proof.

Fix $\boldsymbol{v}\in{\cal V}_{x}^{\overline{d}_{x}}(\boldsymbol{c}_{y})$. Since $\boldsymbol{c}_{y}\subseteq\boldsymbol{v}$, we have on event $\text{UC}^{c}$

[TABLE]

Similarly, we also have

[TABLE]

Combining these two yields

[TABLE]

Next, consider $\hat{\boldsymbol{c}}_{y}(t)$, which is chosen from ${\cal R}_{y}(t)$ as the $\overline{d}_{x}$-tuple of context dimensions with the minimum variation. We have for all $\boldsymbol{j},\boldsymbol{k}\in{\cal V}_{x}({\hat{\boldsymbol{c}}_{y}(t)},\overline{d}_{x})$

[TABLE]

Also, on event $\text{UC}^{c}$, we have for all $\boldsymbol{l}\in{\cal V}_{x}(\boldsymbol{v},\overline{d}_{x})$

[TABLE]

Thus, on event $\text{UC}^{c}$, we obtain for all $\boldsymbol{l},\boldsymbol{n}\in{\cal V}_{x}(\boldsymbol{v},\overline{d}_{x})$

[TABLE]

Let $\boldsymbol{g}(\boldsymbol{v},\hat{\boldsymbol{c}}_{y}(t))$ be a $2\overline{d}_{x}$-tuple of context dimensions that includes all entries of $\boldsymbol{v}$ and $\hat{\boldsymbol{c}}_{y}(t)$, i.e., for all $i\in\boldsymbol{v}$ and $j\in\hat{\boldsymbol{c}}_{y}(t)$, we have $i,j\in\boldsymbol{g}(\boldsymbol{v},\hat{\boldsymbol{c}}_{y}(t))$. The existence of at least one such $2\overline{d}_{x}$-tuple of context dimensions is guaranteed since $\boldsymbol{v}$ and $\hat{\boldsymbol{c}}_{y}(t)$ are both $\overline{d}_{x}$-tuples of context dimensions. Combining what we have obtained thus far, we get

[TABLE]

Finally, combining the result above with (6), we obtain

[TABLE]

∎

To prove the next lemma, we introduce new notation. For $y\in{\cal Y}$, $\boldsymbol{w}\in{\cal V}_{x}^{2\overline{d}_{x}}$ and $p_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$, let

[TABLE]

and $\tau_{y,\boldsymbol{w},{p}_{\boldsymbol{w}}}(t)$ denote the round in which a context arrives to ${p}_{\boldsymbol{w}}$, arm $y$ is chosen and $\boldsymbol{w}_{y}(t)=\boldsymbol{w}$ for the $t$th time. For simplicity, with an abuse of notation we let ${\cal T}_{y,{p}_{\boldsymbol{w}}}\coloneqq{\cal T}_{y,\boldsymbol{w},{p}_{\boldsymbol{w}}}$ and $\tau_{y,{p}_{\boldsymbol{w}}}(t)\coloneqq\tau_{y,\boldsymbol{w},{p}_{\boldsymbol{w}}}(t)$.

Lemma 4.

On event $\text{UC}^{c}$, for all $y\in{\cal Y}$, $\boldsymbol{w}\in{\cal V}_{x}^{2\overline{d}_{x}}$, $p_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$ and for all $t\in\{1,\ldots,|{\cal T}_{y,{p}_{\boldsymbol{w}}}|\}$, we have

[TABLE]

where $y^{*}(t)\in\operatorname*{arg\,max}_{y^{\prime}\in{\cal Y}}\mu_{y^{\prime}}(x(t))$.

Proof.

Since CMAB-RL chooses arm $y$ in round $\tau_{y,{p}_{\boldsymbol{w}}}(t)$, we have $y\in\operatorname*{arg\,max}_{y^{\prime}\in{\cal Y}}\{\hat{\mu}^{\hat{\boldsymbol{c}}_{y^{\prime}}(\tau_{y,{p}_{\boldsymbol{w}}}(t))}_{y^{\prime}}(\tau_{y,{p}_{\boldsymbol{w}}}(t))+5u_{y^{\prime},\boldsymbol{w}_{y^{\prime}}(\tau_{y,{p}_{\boldsymbol{w}}}(t))}(\tau_{y,{p}_{\boldsymbol{w}}}(t))\}$. By Lemma 3, we have

[TABLE]

For all $y^{\prime}\in{\cal Y}$, let

[TABLE]

Note that by the selection rule of CMAB-RL, $U^{\prime}_{y}(\tau_{y,{p}_{\boldsymbol{w}}}(t))\geq U^{\prime}_{y^{*}(\tau_{y,{p}_{\boldsymbol{w}}}(t))}(\tau_{y,{p}_{\boldsymbol{w}}}(t))$. Combining this with the result of Lemma 3 we obtain $U^{\prime}_{y}(\tau_{y,{p}_{\boldsymbol{w}}}(t))\geq U^{\prime}_{y^{*}(\tau_{y,{p}_{\boldsymbol{w}}}(t))}(\tau_{y,{p}_{\boldsymbol{w}}}(t))\geq\mu_{y^{*}(\tau_{y,{p}_{\boldsymbol{w}}}(t))}(x(\tau_{y,{p}_{\boldsymbol{w}}}(t)))\geq\mu_{y}(x(\tau_{y,{p}_{\boldsymbol{w}}}(t)))\geq L^{\prime}_{y}(\tau_{y,{p}_{\boldsymbol{w}}}(t))$. Therefore, we get $\mu_{y^{*}(\tau_{y,{p}_{\boldsymbol{w}}}(t))}(x(\tau_{y,{p}_{\boldsymbol{w}}}(t)))-\mu_{y}(x(\tau_{y,{p}_{\boldsymbol{w}}}(t)))\leq U^{\prime}_{y}(\tau_{y,{p}_{\boldsymbol{w}}}(t))-L^{\prime}_{y}(\tau_{y,{p}_{\boldsymbol{w}}}(t))=10\epsilon+10u_{y,\boldsymbol{w}_{y}(\tau_{y,{p}_{\boldsymbol{w}}}(t))}(\tau_{y,{p}_{\boldsymbol{w}}}(t))$. Finally, note that in round $\tau_{y,{p}_{\boldsymbol{w}}}(t)$ it holds that $\boldsymbol{w}_{y}(\tau_{y,{p}_{\boldsymbol{w}}}(t))=\boldsymbol{w}$, hence we also have $u_{y,\boldsymbol{w}_{y}(\tau_{y,{p}_{\boldsymbol{w}}}(t))}(\tau_{y,{p}_{\boldsymbol{w}}}(t))=u_{y,\boldsymbol{w}}(\tau_{y,{p}_{\boldsymbol{w}}}(t))$. Using this information we get the inequality stated in the lemma. ∎

For each $y\in{\cal Y}$, there are $|{\cal V}_{x}^{2\overline{d}_{x}}|=\genfrac{(}{)}{0.0pt}{1}{d_{x}}{2\overline{d}_{x}}$ different $2\overline{d}_{x}$-tuples of context dimensions and for each $2\overline{d}_{x}$-tuple of context dimensions $\boldsymbol{w}\in{\cal V}_{x}^{2\overline{d}_{x}}$, $|{{\cal P}}_{\boldsymbol{w}}|=m^{2\overline{d}_{x}}$. Thus, we have

[TABLE]

where $B_{m,T}\coloneqq 10\sqrt{2A_{m,T}}$ and $A_{m,T}\coloneqq(1+2\log(2|{\cal Y}|\bar{C}m^{2\overline{d}_{x}}T^{3/2}))$.

In order to bound the regret, next, we evaluate the error due to discretization of the arm set. Recall that instead of choosing arms from ${\cal A}$, CMAB-RL chooses arms from ${\cal Y}$ such that $|{\cal Y}|=m^{\overline{d}_{a}}\genfrac{(}{)}{0.0pt}{1}{d_{a}}{\overline{d}_{a}}$. The regret due to this discretization can be bounded as

[TABLE]

Combining this with the regret bound obtained above and recalling that $\epsilon=L\left(\sqrt{\overline{d}_{x}}/m\right)$, we get

[TABLE]

with probability $1-1/T$. Finally, after choosing $m=\lceil T^{1/(2+2\overline{d}_{x}+\overline{d}_{a})}\rceil$ the regret bound becomes

[TABLE]

which proves Theorem 1.

VI Illustrative Results

In this section, we numerically evaluate the performance of CMAB-RL in two experiments. In the first experiment, we generate a synthetic simulation environment with multi-dimensional context and arm sets, where in each set only a single dimension is relevant. In the second experiment, we apply CMAB-RL to the dynamic drug dosage regulation problem (bolus insulin administration) by utilizing OhioT1DM dataset [47].

VI-A Competitor Learning Algorithms

VI-A1 Instance-based Uniform Partitioning (IUP) [13]

This is a contextual MAB algorithm that learns the optimal arm for each context by uniformly partitioning the set of feasible context-arm pairs ${\cal F}$ into $m^{d_{x}+d_{a}}$ hypercubes, where the choice $m=\lceil T^{1/(2+d_{x}+d_{a})}\rceil$ is shown minimize the regret. In each round, IUP first identifies the set of hypercubes that contain the current context, and then, plays an arm within the hypercube with the highest UCB among all hypercubes in that set. IUP does not take the relevance information into account.

VI-A2 Contextual Hierarchical Optimistic Optimization (C-HOO)

This is the contextual version of hierarchical optimistic optimization (HOO) strategy proposed in [25].444Another related work [24] also proposes a contextual version of HOO for the Bayesian version of the MAB problem with Gaussian process prior. Originally, HOO adaptively partitions the arm set ${\cal A}$, by the help of a binary tree structure it stores. Each node of the tree corresponds to a subset of ${\cal A}$, and as the depth level of a node increases, the subset it represents gets smaller. Subsets that correspond to nodes that have the same depth level form a partition on ${\cal A}$. The tree of partitions is constructed in a way such that the union of the regions covered by the children of a node $n$ is equal to the region that node $n$ covers. In each round, HOO constructs a path starting from the root node, which corresponds to ${\cal A}$. The path is constructed such that at every level of the tree, the child node with the highest UCB is added to the path. When a node with at most one child is reached, if the node has one child, the second child is created. Otherwise, a random child is created. The arm to be played is selected from the region that the newly created child represents. As HOO gathers information about the environment, it “zooms” into regions with potentially high expected rewards, thereby performing more careful exploration in these regions.

We create C-HOO based on HOO as follows. First of all, we construct a tree of partitions over ${\cal F}$ instead of ${\cal A}$. In each round, C-HOO first observes the context, and then, constructs its path similar to HOO. The difference is that when constructing the path, at every level of the tree, first the availability (whether a node contains the context) of the children are checked, and among the children that contain the current context, the one with the highest UCB is added to the path. It is also important to note that since the computational complexity of HOO increases quadratically with the number of rounds, we construct C-HOO based on the truncated version of HOO [25], which is more efficient and enjoys the same regret bound as HOO except an additive factor of $4\sqrt{T}$.

VI-A3 Uniform Random

This benchmark randomly selects an arm in each round without taking the current context or past information into account.

VI-B Parameters Used in the Experiments

We assume that the Lipschitz constants in both experiments are unknown to the learner, thus simply set $L=1$ in the learning algorithms. Moreover, the set of all feasible context-arm pairs ${\cal F}$, time horizon $T$, dimensionality of context and arm sets, i.e., $d_{x}$ and $d_{a}$, are given as inputs to all learning algorithms. In addition, we set $\overline{d}_{x}=\underline{d}_{x}$ and $\overline{d}_{a}=\underline{d}_{a}$ for CMAB-RL, and $v_{1}=2\sqrt{d_{x}+d_{a}}$ and $\rho=2^{(-1/(d_{x}+d_{a}))}$ for C-HOO (consistent with Assumption A1 in [25]). For IUP, no additional parameters are required. The confidence terms of all learning algorithms are scaled (multiplied) with a constant that is chosen from the set $\{0.001,0.005,0.01,0.05,0.1,0.25,0.5,1\}$ which pushes algorithms to exploit more. The rationale behind this choice is that during our experiments we observed that the confidence terms start large and vanish slowly forcing learning algorithms to explore too much, and scaling helps learning algorithms achieve higher cumulative rewards. For each learning algorithm, the optimal multiplier for the confidence term is found by grid search. For all experiments, in order to reduce the effect of randomness due to context arrivals, arm selections and reward generation on the performance measurements, the reported results correspond to the average of 20 independent repetitions.

VI-C Experiments on a Synthetic Simulation Environment

We consider a setting with $d_{x}=5$, $d_{a}=5$, $\underline{d}_{x}=1$ and $\underline{d}_{a}=1$, and assume that the relevant context dimension is the same for all arms. We let the relevant arm and context dimensions to be the first arm and context dimensions respectively, i.e., $\boldsymbol{c}=\{1\}$ and $\boldsymbol{c}_{a}=\{1\}$, $\forall a\in{\cal A}$. Since the expected reward function does not depend on the irrelevant context dimensions, we have $\underline{d}_{x}+\underline{d}_{a}=2$. The expected reward function is defined by using a multivariate Gaussian mixture model, where the expected reward for context-arm pair $(x,a)\in{\cal F}$ is given as

[TABLE]

for $\sum_{i=1}^{K}\rho_{i}=1$ and $\rho_{i}>0$, for $1\leq i\leq K$. Here, $s$ denotes the scaling factor, $K$ denotes the number of components, $f$ denotes the probability density function of a multivariate Gaussian distribution and $\rho_{i},\theta_{i}\text{ and }\Sigma_{i}$ stand for the component weight, mean vector and covariance matrix of the $i$th component, respectively. The parameters of the Gaussian mixture are set as follows: $s=0.25$, $K=2$, $\rho_{1}=\rho_{2}=0.5$, $\theta_{1}=[0.25,0.75]^{T}$, $\theta_{2}=[0.5,0.5]^{T}$ and

[TABLE]

Variation of the expected reward function over the relevant context and arm dimensions can be seen in Fig. 1. The reward that the learner receives in round $t$ is sampled from a Bernoulli distribution with parameter $\mu_{a(t)}(x(t))$ independently from the other rounds.

Learning algorithms are run for a time horizon of $T=10^{5}$ rounds. In each round, a context arrives uniformly at random. The optimal multipliers for the confidence terms are found to be $0.001$ for CMAB-RL, $0.01$ for IUP and $0.05$ for C-HOO. Reported results correspond to this choice of multipliers. Cumulative rewards of the algorithms over time are given in Fig. 2. As we can see, CMAB-RL achieves more than $29\%$ and $100\%$ improvement over the cumulative rewards of C-HOO and IUP respectively. Although C-HOO does not utilize relevancy information, it significantly outperforms IUP as a result of employing adaptive exploration using a tree of partitions. On the other hand, IUP performs poorly due to the curse of dimensionality. As a result, its cumulative reward is only slightly higher than that of Uniform Random.

Results on the regret are given in Fig. 3. The increase in the regret of CMAB-RL significantly drops down after $15000$ rounds, while the increase in the regrets of C-HOO and IUP does not drop significantly in the given time horizon. Since $T$ is an input to the learning algorithms, we provide additional results on the regret when the algorithms are run with input time horizons ranging from $T=5000$ to $T=10^{5}$. Fig. 4 shows that CMAB-RL achieves the smallest regret for all time horizons.

VI-D Experiments on the OhioT1DM dataset

For our second experiment, we use the OhioT1DM dataset that consists of several physiological measurements for 6 T1DM patients who are on continuous glucose monitoring and insulin pump therapy over a time period of 8 weeks (see [47] for the details). While the original dataset is split into training and test sets for each patient in advance, we merge them into a single set to perform online learning.

Our aim in this experiment is to learn the optimal bolus insulin dose for a patient such that their mean blood glucose levels remain within the desired range of $80$ to $180$ mg/dL (see, e.g., [48]) by making use of contextual information such as the state of the patient and the ongoing basal insulin treatment before a bolus injection. As the state of the patient, we consider means of (i) continuous glucose measurements (CGMs), (ii) heart rate, (iii) skin temperature, (iv) air temperature and (v) galvanic skin response measurement, and sums of (i) carbohydrate intake from meals, (ii) exercise scores (multiplication of the duration and the intensity of an exercise session) and (iii) number of steps taken for the last $30$ minutes before a bolus injection. As the ongoing basal insulin treatment, we consider the mean of the basal insulin dosages for the last $30$ minutes. This corresponds to the setting where $d_{x}=9$. As the arms, we only consider the bolus insulin dosages, thus $d_{a}=\underline{d}_{a}=1$. Since bolus insulin doses are administered by an insulin pump that provides doses of insulin with a fine granularity, the set of bolus insulin doses can be approximated well by a continuum of values. Note that data is scaled such that it resides in range $[0,1]$ for all context and arm dimensions.

The rewards are based on the mean of the CGMs of the patients for the next 30 minutes to 2 hours after a bolus injection. Thus, for the sake of simplicity, in the rest of this section, we call CGM values that we use as contexts as past CGMs and CGM values that we use for reward generation as resulting CGMs.

We impute the missing values as follows. If no data is available to generate the contexts, then we set the contexts for carbohydrate intake, exercise and number of steps as zero, since lack of data suggests no activity. For heart rate, skin temperature, air temperature and galvanic skin response, we take the mean value of the whole dataset. Data is always available for bolus injections as we first locate the bolus events and extract other variables near the bolus events. If however, no data is available for past or resulting CGMs of a bolus event, then we ignore that bolus event.

In order to setup the simulation, for each patient we fit a multivariate Gaussian distribution to all context dimensions, using only the said patient’s data. Moreover, we learn a prior distribution over the patients by considering how frequently they appear in the dataset. We also need to model every possible combination of contexts, arms and rewards, which means that we need to learn a mapping from the context-arm space to the reward space. To achieve this, we use a Gradient Boosting regression model with Huber loss, which has $100$ decision trees as weak estimators where each tree is constrained to have a maximum depth of $5$. The inputs to the regression model are contexts and arms, whereas the outputs are the resulting CGMs. We use oversampling so that all patients have equal amount of data prior to the training of Gradient Boosting. The oversampling is done by sampling with replacement. During the experiment, in each round $t$, we select a patient randomly using the prior distribution, then we sample the context vector $x(t)$ from the selected patient’s Gaussian distribution. If the generated context is not in range $[0,1]^{(dx+da)}$, we repeat the sampling process until a valid context is generated. Then, we feed the generated context to the CMAB algorithm. When the CMAB algorithm returns the arm $a(t)$, we query the regression model for the reward $r(t)$, inputting $x(t)$ and $a(t)$. Upon receiving the query, the environment generates a resulting CGM value, and translates it into $r(t)$ using the following mapping:

[TABLE]

Note that we add zero-mean Gaussian noise with standard deviation of 5 to the resulting CGMs to introduce randomness to the rewards.

After training the Gradient Boosting regression model, we examine average impurity decrease for each input across all trees which are then normalized so that the sum of the average impurities for all inputs add up to 1. This examination shows that only the past CGM values before a bolus event yields a score higher than 0.5, while all the other variables yield scores lower than 0.1. This result is consistent with other works that study this dataset in the setting of forecasting, including [5] and [6]. Therefore, it can be argued that past CGM values are the most relevant in the set of available features. In light of this information, we set $\underline{d}_{x}=1$ during the experiment and fix the horizon to be $T=10^{5}$. The confidence term multipliers in this experiment are $0.001$ for CMAB-RL, $0.05$ for IUP and $0.1$ for C-HOO.

The histograms of resulting CGMs of all learning algorithms and the original dataset are given in Fig. 5. These are normalized such that the area under individual histograms sum up to $1$, so that the difference between the glucose control in the original dataset and that of the learning algorithms can be observed better. It is observed that in general all learning algorithms provide better glucose management than the one in the original dataset. In addition, Table II, represents the percentage of samples for which the resulting CGMs represent hypoglycemia or hyperglycemia, or are in the desired range. It is seen that for each patient, CMAB-RL has the highest percentage of samples between the desired range of $80$ to $180$ mg/dL. Moreover, CMAB-RL also has the lowest density in the regions that correspond to hypoglycemia and hyperglycemia, except for patient 588, for which the original dataset has no hypoglycemic CGMs.

VII Conclusion

In this work, we considered a CMAB problem with high-dimensional context and arm sets, and motivated by real-world applications, assumed that the reward only depends on a few relevant dimensions of the context and the arm sets. For this problem, we proposed an online learning algorithm, called CMAB-RL, which learns the relevant context and arm dimensions simultaneously, thereby achieving a regret bound that only depends on the maximum number of relevant dimensions given that this number is known by the learner. Our regret analysis does not require any stochastic assumptions on the context arrivals, and CMAB-RL is shown to beat other contextual MAB algorithms that do not exploit the relevance in both synthetic and real-world datasets.

Appendix A Tables of Notation

Appendix B Complexity of CMAB-RL

CMAB-RL stores sample mean reward estimates and counters for all $y\in{\cal Y}$, $\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}$ and $p_{\boldsymbol{w}}\in{\cal P}_{\boldsymbol{w}}$. Thus, the memory requirement is $O\left(\genfrac{(}{)}{0.0pt}{1}{d_{a}}{\overline{d}_{a}}\genfrac{(}{)}{0.0pt}{1}{d_{x}}{2\overline{d}_{x}}m^{2\overline{d}_{x}+\overline{d}_{a}}\right)$. For $m=\lceil T^{1/(2+2\overline{d}_{x}+\overline{d}_{a})}\rceil$ as given in Theorem 1, the memory complexity in time becomes $O(T^{(2\overline{d}_{x}+\overline{d}_{a})/(2+2\overline{d}_{x}+\overline{d}_{a})})$, which is sublinear. Note that CMAB-RL can be implemented in a more efficient way by only creating and storing sample mean reward estimates and counters for sets in the partition to which a context had arrived in the past.

Next, we investigate the computational complexity of CMAB-RL during run-time. In round $t$, finding $p_{\boldsymbol{w}}(t)\in{\cal P}_{\boldsymbol{w}}$ for all $\boldsymbol{w}\in{\cal V}^{2\overline{d}_{x}}_{x}$, requires $O\left(d_{x}+\genfrac{(}{)}{0.0pt}{1}{d_{x}}{2\overline{d}_{x}}\right)$ computations. Construction of set ${\cal R}_{y}(t)$ for all $y\in{\cal Y}$, calculation of $\hat{\sigma}^{2}_{y,\boldsymbol{v}}(t)$ for all $y\in{\cal Y}$, $\boldsymbol{v}\in{\cal R}_{y}(t)$ and determination of $\hat{\boldsymbol{c}}_{y}(t)$ for all $y\in{\cal Y}$ all together require $O\left(\genfrac{(}{)}{0.0pt}{1}{d_{a}}{\overline{d}_{a}}m^{\overline{d}_{a}}\genfrac{(}{)}{0.0pt}{1}{d_{x}}{\overline{d}_{x}}{\genfrac{(}{)}{0.0pt}{1}{d_{x}-\overline{d}_{x}}{\overline{d}_{x}}}^{2}\right)$ operations. In order to estimate $\hat{\mu}^{\hat{\boldsymbol{c}}_{y}(t)}_{y}(t)$ for all $y\in{\cal Y}$, we need $O\left(\genfrac{(}{)}{0.0pt}{1}{d_{a}}{\overline{d}_{a}}m^{\overline{d}_{a}}{\genfrac{(}{)}{0.0pt}{1}{d_{x}-\overline{d}_{x}}{\overline{d}_{x}}}\right)$ operations. Determination of $\boldsymbol{w}_{y}(t)$ for all $y\in{\cal Y}$ requires $O\left(\genfrac{(}{)}{0.0pt}{1}{d_{a}}{\overline{d}_{a}}m^{\overline{d}_{a}}\genfrac{(}{)}{0.0pt}{1}{d_{x}}{2\overline{d}_{x}}\right)$ comparisons. Finally, determining $y(t)$ requires $O\left(\genfrac{(}{)}{0.0pt}{1}{d_{a}}{\overline{d}_{a}}m^{\overline{d}_{a}}\right)$ comparisons. Hence, the overall computational complexity of CMAB-RL in round $t$ is

[TABLE]

For $m=\lceil T^{1/(2+2\overline{d}_{x}+\overline{d}_{a})}\rceil$ as given in Theorem 1, per-round computational complexity in time becomes $O(T^{\overline{d}_{a}/(2+2\overline{d}_{x}+\overline{d}_{a})})$, which is sublinear.

Appendix C Regret Analysis of Exp4 for a Special Case

In this section, we assume that the set of relevant context dimensions is the same for each arm and derive a regret bound for Exp4 [46] under this assumption. We define the experts of Exp4 in the following way. Recall the Generate procedure from Section IV. For each $\boldsymbol{v}\in{\cal V}^{\overline{d}_{x}}_{x}$, we have $|{\cal Y}|=\genfrac{(}{)}{0.0pt}{1}{d_{a}}{\overline{d}_{a}}m^{\overline{d}_{a}}$ arms. Expert $(\boldsymbol{v},i)$ assumes that the tuple of relevant context dimensions is $\boldsymbol{v}$. It partitions ${\cal X}_{\boldsymbol{v}}$ into $m^{\overline{d}_{x}}$ sets denoted by ${\cal P}_{\boldsymbol{v}}$. Then, it assigns one action in ${\cal Y}$ to each set in ${\cal P}_{\boldsymbol{v}}$. The number of different experts that can be defined for $\boldsymbol{v}$ is

[TABLE]

Thus, the total number of experts is

[TABLE]

The regret of the best expert in the pool of experts defined above with respect to the optimal arm is proportional to $TL(\sqrt{\overline{d}_{a}}/m+\sqrt{\overline{d}_{x}}/m)$ due to discretization. If we use Exp4, then its regret with respect to the best expert in the pool of experts defined above is

[TABLE]

To balance the regret due to discretization and the regret due to Exp4 we set $m=\lceil T^{1/(\overline{d}_{x}+\overline{d}_{a}+2)}\rceil$, which results in total regret $\tilde{O}(T^{1-1/(\overline{d}_{x}+\overline{d}_{a}+2)})$.

Appendix D Concentration Inequality [28]

Consider an arm $y$, a tuple $\boldsymbol{w}$, and a set $p_{\boldsymbol{w}}$ in the partition ${\cal P}_{\boldsymbol{w}}$ for which the rewards are generated by a process $\{{R}_{y,{p}_{\boldsymbol{w}}}(t)\}_{t=1}^{N_{y,{p}_{\boldsymbol{w}}}(T)}$ with ${\mu}_{y,{p}_{\boldsymbol{w}}}=\mathrm{E}[{R}_{y,{p}_{\boldsymbol{w}}}(t)]$, where the noise ${R}_{y,{p}_{\boldsymbol{w}}}(t)-{\mu}_{y,{p}_{\boldsymbol{w}}}$ is conditionally 1-sub-Gaussian. Let $N_{y,{p}_{\boldsymbol{w}}}(T)\geq 1$ denote the number of times $y$ is selected by the end of time $T$. Let $\hat{\mu}_{y,{p}_{\boldsymbol{w}}}(T)=\sum_{t=1}^{T}\mathrm{I}(y(t)=y){R}_{y,{p}_{\boldsymbol{w}}}(t)/N_{y,{p}_{\boldsymbol{w}}}(T)$. For any $\delta>0$ with probability at least $1-\delta$ we have

[TABLE]