Adaptive Sequential Experiments with Unknown Information Arrival Processes

TL;DR

This paper extends the multi-armed bandit framework to incorporate unknown information arrival processes and auxiliary data, providing theoretical bounds and practical algorithms for improved sequential decision-making.

Contribution

It introduces a generalized MAB model with unknown information arrival, characterizes minimax complexities, and proposes a robust policy leveraging auxiliary data for better performance.

Findings

Upper confidence bound and posterior sampling policies are robust to information arrival processes.

A new policy using dual upper confidence bounds is near-optimal when auxiliary data mappings are unknown.

Empirical analysis shows potential performance gains from auxiliary data in content recommendation.

Abstract

Sequential experiments are often characterized by an exploration-exploitation tradeoff that is captured by the multi-armed bandit (MAB) framework. This framework has been studied and applied, typically when at each time period feedback is received only on the action that was selected at that period. However, in many practical settings additional data may become available between decision epochs. We introduce a generalized MAB formulation, which considers a broad class of distributions that are informative about mean rewards, and allows observations from these distributions to arrive according to an arbitrary and a priori unknown arrival process. When it is known how to map auxiliary data to reward estimates, by obtaining matching lower and upper bounds we characterize a spectrum of minimax complexities for this class of problems as a function of the information arrival process, which captures how salient characteristics of this process impact achievable performance. In terms of achieving optimal performance, we establish that upper confidence bound and posterior sampling policies possess natural robustness with respect to the information arrival process without any adjustments, which uncovers a novel property of these popular policies and further lends credence to their appeal. When the mappings connecting auxiliary data and rewards are a priori unknown, we characterize necessary and sufficient conditions under which auxiliary information allows performance improvement. We devise a new policy that is based on two different upper confidence bounds (one that accounts for auxiliary observation and one that does not) and establish the near-optimality of this policy. We use data from a large media site to analyze the value that may be captured in practice by leveraging auxiliary data for designing content recommendations.

Tables

Table 1. Table 1 : Notations for counters and empirical means

Definition	Description
$n_{k,t}^{\pi }≔{\sum }_{s=1}^{t-1}\mathrm{𝟙}\left\{{\pi }_s=k\right\}$	Number of pulls
${\overline{X}}_{k,t}^{\pi }≔{\overline{X}}_{k,n_{k,t}^{\pi }}^{\pi }≔\left({\sum }_{s=1}^{t-1}\mathrm{𝟙}\left\{{\pi }_s=k\right\}{X}_{k,s}\right)/\left(\max \{1,n_{k,t}^{\pi }\}\right)$	Empirical mean reward
$n_{k,t}^{\mathrm{aux}}≔{\sum }_{s=1}^t{h}_{k,s}$	Number of auxiliary observations
${\overline{Y}}_{k,t}≔{\overline{Y}}_{k,n_{k,t}^{\mathrm{aux}}}≔\left({\sum }_{s=1}^t{\sum }_{m=1}^{h_{k,s}}{Y}_{k,s,m}\right)/\left(\max \{1,n_{k,t}^{\mathrm{aux}}\}\right)$	Empirical mean of auxiliary observations
$n_{k,t}^{\pi ,\mathrm{aux}}≔n_{k,t}^{\pi }+\frac{{\sigma }^2}{{\overline{\alpha }}^2{\hat{\sigma }}^2}{n}_{k,t}^{\mathrm{aux}}$	Weighted number of observations
${\overline{X}}_{k,t}^{\pi ,\mathrm{aux}}≔{\overline{X}}_{k,n_{k,t}^{\pi },n_{k,t}^{\mathrm{aux}}}^{\pi ,\mathrm{aux}}≔\left(n_{k,t}^{\pi }\cdot {\overline{X}}_{k,n_{k,t}^{\pi }}^{\pi }+\frac{{\sigma }^2}{{\overline{\alpha }}^2{\hat{\sigma }}^2}\cdot n_{k,t}^{\mathrm{aux}}\cdot \overline{\alpha }{\overline{Y}}_{k,t}\right)/\left(\max \{1,n_{k,t}^{\pi ,\mathrm{aux}}\}\right)$	Optimistic empirical mean reward

Table 2. Table 2: Notations for counters and empirical means

Definition	Description
$n_{k,t}^{\mathrm{known}}≔\sum _{s=1}^{t-1}\mathrm{𝟙}\left\{{\pi }_s=k\right\}+\sum _{s=1}^t\frac{{\sigma }^2}{{\hat{\sigma }}^2}{h}_{k,s}$	Number of pulls (known mappings case)
${\overline{X}}_{k,t}^{\mathrm{known}}≔{\overline{X}}_{k,n_{k,t}^{\mathrm{known}}}^{\mathrm{known}}≔\frac{\sum _{s=1}^{t-1}\frac{1}{{\sigma }^2}\mathrm{𝟙}\left\{{\pi }_s=k\right\}{X}_{k,s}+\sum _{s=1}^t\frac{1}{{\hat{\sigma }}^2}{h}_{k,s}{Z}_{k,s}}{\sum _{s=1}^{t-1}\frac{1}{{\sigma }^2}\mathrm{𝟙}\left\{{\pi }_s=k\right\}+\sum _{s=1}^t\frac{1}{{\hat{\sigma }}^2}{h}_{k,s}}$	Number of pulls (known mappings case)
$n_{k,t}^{\pi }≔{\sum }_{s=1}^{t-1}\mathrm{𝟙}\left\{{\pi }_s=k\right\}$	Number of pulls
${\overline{X}}_{k,t}^{\pi }≔{\overline{X}}_{k,n_{k,t}^{\pi }}^{\pi }≔\left({\sum }_{s=1}^{t-1}\mathrm{𝟙}\left\{{\pi }_s=k\right\}{X}_{k,s}\right)/\left(\max \{1,n_{k,t}^{\pi }\}\right)$	Empirical mean reward
$n_{k,t}^{\mathrm{aux}}≔{\sum }_{s=1}^t{h}_{k,s}$	Number of auxiliary observations
${\overline{Y}}_{k,t}≔{\overline{Y}}_{k,n_{k,t}^{\mathrm{aux}}}≔\left({\sum }_{s=1}^t{\sum }_{m=1}^{h_{k,s}}{Y}_{k,s,m}\right)/\left(\max \{1,n_{k,t}^{\mathrm{aux}}\}\right)$	Empirical mean of auxiliary obs.
$n_{k,t}^{\pi ,\mathrm{aux}}≔n_{k,t}^{\pi }+\frac{{\sigma }^2}{{\overline{\alpha }}^2{\hat{\sigma }}^2}{n}_{k,t}^{\mathrm{aux}}$	Weighted number of observations
${\overline{X}}_{k,t}^{\pi ,\mathrm{aux}}≔{\overline{X}}_{k,n_{k,t}^{\pi },n_{k,t}^{\mathrm{aux}}}^{\pi ,\mathrm{aux}}≔\left(n_{k,t}^{\pi }\cdot {\overline{X}}_{k,n_{k,t}^{\pi }}^{\pi }+\frac{{\sigma }^2}{{\overline{\alpha }}^2{\hat{\sigma }}^2}\cdot n_{k,t}^{\mathrm{aux}}\cdot \overline{\alpha }{\overline{Y}}_{k,t}\right)/\left(\max \{1,n_{k,t}^{\pi ,\mathrm{aux}}\}\right)$	Optimistic empirical mean reward

Table 3. Table 3: Cumulative regret of UCB1, aUCB1, and 2-UCBs normalized by the cumulative regret of UCB1 in the case of Δ = 0.03 Δ 0.03 \Delta=0.03 and c = 0.05 𝑐 0.05 c=0.05 , under different choices of the gap Δ Δ \Delta and tuning parameter c 𝑐 c .

	$𝒄=0.03$			$𝒄=0.05$			$𝒄=0.1$
	UCB1	aUCB1	2-UCBs	UCB1	aUCB1	2-UCBs	UCB1	aUCB1	2-UCBs
$𝚫=0.01$	0.44538	0.42288	0.43988	0.44567	0.43286	0.43855	0.46515	0.45524	0.45252
$𝚫=0.02$	0.90161	0.82164	0.86636	0.89996	0.84585	0.83899	1.0251	0.95521	0.94564
$𝚫=0.03$	1.0107	0.90583	0.95628	1	0.93476	0.91303	1.1764	1.0839	1.0611
$𝚫=0.04$	1.1403	1.0023	1.0732	1.1373	1.0461	1.0057	1.4188	1.2591	1.2093
$𝚫=0.2{\mathrm{𝐂𝐕𝐑}}_{𝒂,𝒅}$	1.1679	1.0093	1.1354	1.0984	1.0115	1.0242	1.2726	1.159	1.1108

Table 4. Table 4: Average Relative Improvement (RI) for aUCB1 and 2-UCBs under different choices of the gap Δ Δ \Delta and tuning parameter c 𝑐 c .

	$𝚫=0.01$		$𝚫=0.02$		$𝚫=0.03$		$𝚫=0.04$		$𝚫=0.2{\mathrm{𝐂𝐕𝐑}}_{𝒂,𝒅}$
	aUCB1	2-UCBs	aUCB1	2-UCBs	aUCB1	2-UCBs	aUCB1	2-UCBs	aUCB1	2-UCBs
$𝒄=0.03$	5.0%	1.2%	8.8%	3.9%	10.3%	5.3%	12.1%	5.8%	13.5%	2.7%
$𝒄=0.05$	2.8%	1.5%	5.0%	7.7%	6.5%	8.6%	8.0%	11.5%	6.7%	8.9%
$𝒄=0.1$	2.1%	2.7%	6.8%	7.7%	7.8%	9.8%	11.2%	14.7%	8.9%	12.7%

Table 5. Table 5: Average No Harm (NH) for aUCB1 and 2-UCBs under different choices of the gap Δ Δ \Delta and tuning parameter c 𝑐 c .

	$𝚫=0.01$		$𝚫=0.02$		$𝚫=0.03$		$𝚫=0.04$		$𝚫=0.2{\mathrm{𝐂𝐕𝐑}}_{𝒂,𝒅}$
	aUCB1	2-UCBs	aUCB1	2-UCBs	aUCB1	2-UCBs	aUCB1	2-UCBs	aUCB1	2-UCBs
$𝒄=0.03$	66.6%	60.6%	66.1%	60.6%	65.1%	65.6%	73.2%	68.6%	67.1%	62.6%
$𝒄=0.05$	53.0%	60.6%	58.5%	72.7%	59.5%	69.1%	64.6%	73.7%	61.6%	71.7%
$𝒄=0.1$	58.5%	72.2%	54.5%	77.2%	58.5%	77.2%	59.0%	74.2%	57.5%	76.7%

Equations

$$\sum _{k=1}^K\log \left(\sum _{t=1}^T\text{vspace}-0.0cm\exp \left(-c\sum _{s=1}^t{h}_{k,s}\right)\right),\text{vspace}-0.15cm$$

$${\mu }^{\ast }:=\underset{k\in \mathcal{K}}{\max }\{{\mu }_k\},k^{\ast }:=\underset{k\in \mathcal{K}}{\mathrm{argmax}}{\mu }_k.\text{vspace}-0.0cm$$

$$\mathcal{S}=\left\{(\mathbf{\nu },{\mathbf{\nu }}^{\mathrm{aux}})\in \prod _{k\in \mathcal{K}}(\mathcal{V}\times {\mathcal{V}}^{\mathrm{aux}}):\forall k\in \mathcal{K}\exists {\varphi }_k\in \Phi \text{ s.t. }\mathbb{E}\left[{\varphi }_k(Y_{k,t,m})\right]={\mu }_k,\text{and }{\Delta }_k\ge \Delta \text{ for all }k\ne k^{\ast }\right\}\text{vspace}-0.15cm$$

$${\mathcal{R}}_{\mathbf{\nu },{\mathbf{\nu }}^{\mathrm{aux}}}^{\pi }(T,\mathrm{H}):={\mathbb{E}}_{\mathbf{\nu },{\mathbf{\nu }}^{\mathrm{aux}}}^{\pi }\left[\sum _{t=1}^T({\mu }^{\ast }-{\mu }_{{\pi }_t})\right],\text{vspace}-0.2cm$$

$${\mathcal{R}}_{\mathcal{S}}^{\pi }(T,\mathrm{H}):=\underset{\left(\mathbf{\nu },{\mathbf{\nu }}^{\mathrm{aux}}\right)\in \mathcal{S}}{\sup }{\mathcal{R}}_{\mathbf{\nu },{\mathbf{\nu }}^{\mathrm{aux}}}^{\pi }(T,\mathrm{H});\text{ and }{\mathcal{R}}_{\mathcal{S}}^{\ast }(T,\mathrm{H}):=\underset{\pi \in \mathcal{P}}{\inf }{\mathcal{R}}_{\mathcal{S}}^{\pi }(T,\mathrm{H})\text{vspace}-0.2cm$$

$${\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\ge \frac{\text{ltx@label}lowe{r}_{b}ound1C_1}{\Delta }\sum _{k=1}^K\log \left(\frac{\text{ltx@label}lowe{r}_{b}ound2C_2{\Delta }^2}{K}\sum _{t=1}^T\exp \left(-\text{ltx@label}lowe{r}_{b}ound3C_3{\Delta }^2\sum _{s=1}^t{h}_{k,s}\right)\right),\text{vspace}-0.1cm$$

$$\mathrm{KL}(\rho ,{\rho }^{\prime }):=\int \log \left(\frac{d\rho }{d{\rho }^{\prime }}\right)d\rho ={\mathbb{E}}_{\rho }\log \left(\frac{d\rho }{d{\rho }^{\prime }}(X)\right),\text{vspace}-0.15cm$$

$$p_t^{\mathrm{sub}}=\frac{1}{4}\exp \left(-\frac{2{\Delta }^2}{{\sigma }^2}\left({\mathbb{E}}_{\mathbf{\nu },{\mathbf{\nu }}^{\mathrm{aux}}}[{\tilde{n}}_{1,T}]+\sum _{s=1}^t\frac{{\sigma }^2}{{\hat{\sigma }}^2}{h}_{1,s}\right)\right),\text{vspace}-0.15cm$$

$${\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\ge \frac{{\sigma }^2(K-1)}{4K\Delta }\sum _{k=1}^K\log \left(\frac{{\Delta }^2}{{\sigma }^{2}K}\sum _{t=1}^T\exp \left(-\frac{2{\Delta }^2}{{\hat{\sigma }}^2}\sum _{s=1}^t{h}_{k,s}\right)\right),\text{vspace}-0.15cm$$

$${\mathbb{E}}_{\mathrm{H}}\left[{\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\right]\ge \frac{{\sigma }^2(K-1)}{4\Delta }\log \left(\frac{(1-e^{-1/2}){\Delta }^{2}T}{K}\right).\text{vspace}-0.2cm$$

$${\mathbb{E}}_{\mathrm{H}}\left[{\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\right]\ge \frac{{\sigma }^2(K-1)}{4\Delta }\log \left(\frac{1-e^{-1/2}}{2\lambda K{\sigma }^2/{\hat{\sigma }}^2}\right).\text{vspace}-0.1cm$$

$$\mathbb{E}\left[\sum _{s=1}^t{h}_{k,s}\right]=\lfloor \frac{{\hat{\sigma }}^2\kappa }{2{\Delta }^2}\log t\rfloor ,\text{vspace}-0.15cm$$

$${\mathbb{E}}_{\mathrm{H}}\left[{\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\right]\ge \frac{{\sigma }^2(K-1)}{4\Delta }\log \left(\frac{{\Delta }^2/K{\sigma }^2}{1-\kappa }\left((T+1{)}^{1-\kappa }-1\right)\right).\text{vspace}-0.2cm$$

$${\mathbb{E}}_{\mathrm{H}}\left[{\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\right]\ge \frac{{\sigma }^2(K-1)}{4\Delta }\log \left(\frac{{\Delta }^2/K{\sigma }^2}{\kappa -1}\left(1-\frac{1}{(T+1{)}^{\kappa -1}}\right)\right).\text{vspace}-0.2cm$$

$$\mathbb{E}\left[\sum _{s=1}^t{h}_{k,s}\right]=\lambda T\frac{t^{1-\gamma }-1}{T^{1-\gamma }-1}.\text{vspace}-0.10cm$$

$$n_{k,t}^{\text{known}}:=\sum _{s=1}^{t-1}\text{mathbbm}1\{{\pi }_s=k\}+\sum _{s=1}^t\frac{{\sigma }^2}{{\hat{\sigma }}^2}{h}_{k,s},{\bar{X}}_{k,t}^{\text{known}}:={\bar{X}}_{k,n_{k,t}^{\text{known}}}^{\text{known}}:=\frac{\sum _{s=1}^{t-1}\frac{1}{{\sigma }^2}\text{mathbbm}1\{{\pi }_s=k\}{X}_{k,s}+\sum _{s=1}^t\frac{1}{{\hat{\sigma }}^2}{h}_{k,s}{Z}_{k,s}}{\sum _{s=1}^{t-1}\frac{1}{{\sigma }^2}\text{mathbbm}1\{{\pi }_s=k\}+\sum _{s=1}^t\frac{1}{{\hat{\sigma }}^2}{h}_{k,s}}.\text{vspace}-0.8cm$$

$$\pi_{t}=\begin{cases*}t&if $t\leq K$\\ \arg\max_{k\in\mathcal{K}}\left\{U_{k,t}\coloneqq\bar{X}_{k,t}^{\text{\tiny$\mathrm{known}$}}+\sqrt{\frac{c\sigma^{2}\log t}{n^{\text{\tiny$\mathrm{known}$}}_{k,t}}}\right\}&if $t>K$\end{cases*}\vspace{-0.05cm}$$

$${\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\le \sum _{k\in \mathcal{K}}\frac{\text{ltx@label}ucb1C_4}{{\Delta }^2}\log \left(\sum _{t=0}^T\exp \left(\frac{-{\Delta }^2}{\text{ref}\ast ucb1}\sum _{s=1}^t{h}_{k,s}\right)\right)+\text{ltx@label}ucb2C_5\text{vspace}-0.05cm,$$

$${\mathcal{R}}_{\mathcal{S}}^{\ast }(\mathrm{H},T)\asymp \sum _{k\in \mathcal{K}}\log \left(\sum _{t=1}^T\text{vspace}-0.0cm\exp \left(-c\sum _{s=1}^t{h}_{k,s}\right)\right),\text{vspace}-0.1cm$$

$${\tilde{n}}_{k,t-1}\le {\hat{l}}_{k,t}-\sum _{s=1}^t\frac{{\sigma }^2}{{\hat{\sigma }}^2}{h}_{k,s},\text{vspace}-0.2cm$$

$$\sum _{t=1}^T\text{mathbbm}1\left\{{\pi }_t=k,{\tilde{n}}_{k,t-1}\le {\hat{l}}_{k,t}-\sum _{s=1}^t\frac{{\sigma }^2}{{\hat{\sigma }}^2}{h}_{k,s}\right\}\le \underset{1\le t\le T}{\max }\left\{{\hat{l}}_{k,t}-\sum _{s=1}^t\frac{{\sigma }^2}{{\hat{\sigma }}^2}{h}_{k,s}\right\}\text{vspace}-0.05cm$$

$${\pi }_t=\arg \underset{k}{\max }{\theta }_{k,t}.\text{vspace}-0.15cm$$

$${\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\le \sum _{k\in \mathcal{K}\setminus \{k^{\ast }\}}\left(\frac{\text{ltx@label}TS1C_6}{\Delta }\log \left(\sum _{t=0}^T\exp \left(\frac{-{\Delta }^2}{\text{ref}\ast TS1}\sum _{s=1}^t{h}_{k,s}\right)\right)+\frac{\text{ltx@label}TS2C_7}{{\Delta }^3}+\text{ltx@label}TS3C_8\right),\text{vspace}-0.15cm$$

$$\mathbb{E}\left[N_t\right]\le {\log }_{\frac{1}{p}}\left(1+\sum _{s=1}^{t-1}{p}^{\sum _{\tau =1}^s{w}_{\tau }}\right)+\sum _{s=1}^t{w}_s.\text{vspace}-0.05cm$$

$$\mathbb{E}\left[p^{-N_t}|p^{-N_{t-1}}\right]$$

$$\le p^{N_{t-1}+1}\cdot p^{-\left(N_{t-1}+w_t+1\right)}+p^{-\left(N_{t-1}+w_t\right)}$$

$$=p^{-w_t}\cdot \left(1+p^{-N_{t-1}}\right).$$

$$\mathbb{E}\left[N_t\right]\le {\log }_{\frac{1}{p}}\left(\sum _{s=1}^t{p}^{-\sum _{\tau =s}^t{w}_{\tau }}\right)={\log }_{\frac{1}{p}}\left(1+\sum _{s=1}^{t-1}{p}^{\sum _{\tau =1}^s{w}_{\tau }}\right)+\sum _{s=1}^t{w}_s.\text{vspace}-0.3cm$$

$${\mathcal{R}}_{\mathcal{S}}^{\pi }(\mathrm{H},T)\ge \frac{\text{ref}\ast lowe{r}_{b}ound1(K-1)}{\Delta }\log \left(\frac{\text{ref}\ast lowe{r}_{b}ound2{\Delta }^2}{K}T\right),\text{vspace}-0.05cm$$

$${\mathbb{E}}_{\mathbf{\nu },{\mathbf{\nu }}^{\mathrm{aux}}}^{\pi }[n_{k,T+1}^{\pi }]\le \frac{C_{\pi ,\mathcal{S}}}{{\Delta }_k^2}\log T,\text{vspace}-0.175cm$$

$$U_{k,t}=\min \left\{{\bar{X}}_{k,t}^{\pi }+\sqrt{\frac{c{\sigma }^2\log t}{n_{k,t}^{\pi }}},\bar{\alpha }\cdot y_k\right\},\text{vspace}-0.2cm$$

Full text

Keywords: sequential experiments, data-driven decisions, product recommendations, online learning, transfer learning, adaptive algorithms, multi-armed bandits, exploration-exploitation, minimax regret.

1 Introduction

1.1 Background and Motivation

Effective design of sequential experiments requires balancing between new information acquisition (exploration), and optimizing payoffs based on available information (exploitation). A well-studied framework that captures the trade-off between exploration and exploitation is the one of multi-armed bandits (MAB) that first emerged in Thompson (1933) in the context of drug testing, and was later extended by Robbins (1952) to a more general setting. In this framework, an agent repeatedly chooses between different actions (arms), where each action is associated with an a priori unknown reward distribution, and the objective is to maximize cumulative return over a certain time horizon.

The MAB framework focuses on balancing exploration and exploitation under restrictive assumptions on the future information collection process: in every round noisy observations are collected only on the action that was selected at that period. Correspondingly, policy design is typically predicated on that information collection procedure. However, in many practical settings to which bandit models have been applied for designing sequential experimentation, additional information might be realized and leveraged over time. Examples of such settings include dynamic pricing (Bastani et al. 2019, Bu et al. 2019), retail management (Zhang et al. 2010), clinical trials (Bertsimas et al. 2016, Anderer et al. 2019), as well as many machine learning domains (see a list of relevant settings in the survey by Pan and Yang 2009).

To illustrate one particular such setting, consider the problem of designing product recommendations with limited data. Product recommendation systems are widely deployed over the web with the objective of helping users navigate through content and consumer products while increasing volume and revenue for service and e-commerce platforms. These systems commonly apply techniques that leverage information such as explicit and implicit user preferences, product consumption, and consumer ratings (see, e.g., Hill et al. 1995, Konstan et al. 1997, Breese et al. 1998). While effective when ample relevant information is available, these techniques tend to under-perform when encountering products that are new to the system and have little or no trace of activity. This phenomenon, known as the cold-start problem, has been documented and studied in the literature; see, e.g., Schein et al. (2002), Park and Chu (2009).

In the presence of new products, a recommendation system needs to strike a balance between maximizing instantaneous performance indicators (such as revenue) and collecting valuable information that is essential for optimizing future recommendations. To illustrate this tradeoff (see Figure 1), consider a consumer (A) that is searching for a product using the organic recommendation system of an online retailer (e.g., Amazon). The consumer provides key product characteristics (e.g., “headphones” and “bluetooth”), and based on this description (and, perhaps, additional factors such as the browsing history of the consumer) the retailer recommends products to the consumer. While some of the candidate products that fit the desired description may have already been recommended many times to consumers, and the mean returns from recommending them are known, other candidate products might be new brands that were not recommended to consumers before. Evaluating the mean returns from recommending new products requires experimentation that is essential for improving future recommendations, but might be costly if consumers are less likely to purchase these new products.

Several MAB settings were suggested and applied for designing recommendation algorithms that effectively balance information acquisition and instantaneous revenue maximization, where arms represent candidate recommendations; see the overview in Madani and DeCoste (2005), as well as later studies by Agarwal et al. (2009), Caron and Bhagat (2013), Tang et al. (2014), and Wang et al. (2017). Aligned with traditional MAB frameworks, these studies consider settings where in each time period information is obtained only on items that are recommended at that period, and suggest recommendation policies that are predicated on this information collection process. However, in many instances additional browsing and consumption information may be maintained in parallel to the sequential recommendation process, as a significant fraction of website traffic may take place through means other than its organic recommender system; see browsing data analysis in Sharma et al. (2015) that estimate that product recommendation systems are responsible for at most 30% of site traffic and revenue and that substantial fraction of remaining traffic arrives to product pages directly from search engines, as well as Besbes et al. (2016) that report similar findings for content recommendations in media sites.

To illustrate an additional source of traffic and revenue information, let us revert to Figure 1 and consider a different consumer (B) that does not use the organic recommender system of the retailer, but rather arrives to a certain product page directly from a search engine (e.g., Google) by searching for that particular brand. While consumers (A) and (B) may have inherently different preferences, the actions taken by consumer (B) after arriving to the product page (e.g., whether or not she bought the product) can be informative about the returns from recommending that product to other consumers. This additional information could potentially be used to improve the performance of recommendation algorithms, especially when the browsing history that is associated with some products is limited.

The above example illustrates that additional information often becomes available between decision epochs of recommender systems, and that this information might be essential for designing effective product recommendations. In comparison with the restrictive information collection process that is assumed in most MAB formulations, the availability of such auxiliary information (and even the prospect of obtaining it in the future) might impact the achievable performance and the design of sequential experiments and learning policies. When additional information is available one may potentially need to “sacrifice” less decisions for exploration in order to obtain effective estimators for the mean rewards. While this intuition suggests that exploration can be reduced in the presence of additional information, it is a priori not clear what type of performance improvement can be expected given different information arrival processes, and how policy design should depend on these processes.

Moreover, monitoring exploration rates in real time in the presence of an arbitrary information arrival process introduces additional challenges that have distinct practical relevance. Most importantly, an optimal exploration rate may depend on several characteristics of the information arrival process, such as the amount of information that arrives and the time at which it appears (e.g., early on versus later on along the decision horizon). Since it may be hard to predict upfront the salient characteristics of the information arrival process, an important challenge is to identify the extent to which it is possible to adapt in real time to an a priori unknown information arrival process, in the sense of approximating the performance that is achievable under prior knowledge of the sample path of information arrivals. This paper is concerned with addressing these challenges.

The research questions that we study in this paper are along two lines. First, we study how the information arrival process and its characteristics impact the performance one could aspire to when designing a sequential experiment. Second, we study what type of policies should one adopt for designing sequential experiments in the presence of general and unknown information arrival processes.

1.2 Main Contributions

The main contribution of this paper lies in (1) introducing a new, generalized MAB framework with unknown and arbitrary information arrival process; (2) characterizing the minimax regret complexity of this broad class of problems as a function of the information arrival process; and (3) identifying policies (and proposing new ones) that are rate-optimal in the presence of general and a priori unknown information arrival processes. More specifically, our contribution is along the following lines.

(1) Modeling. To capture the salient features of settings with auxiliary information arrivals we formulate a new class of MAB problems that generalizes the classical stochastic MAB framework, by relaxing strong assumptions that are typically imposed on the information collection process. Our formulation considers a broad class of distributions that are informative about mean rewards, and allows observations from these distributions to arrive at arbitrary and a priori unknown rates and times.

More particularly, given available actions $k=1,\ldots,K$ and horizon $t=1,\ldots T$, we consider an a priori unknown information arrival matrix $\bm{\mathrm{H}}$, with each integer entry $h_{k,t}$ capturing the number of auxiliary observations collected on action $k$ between time $t-1$ and time $t$. We assume that for each action $k$ there are mappings $\left\{\phi_{k}:\;k=1,\ldots,K\right\}$ that map auxiliary observations to unbiased estimates of mean rewards, where we distinguish between the cases where these mappings are known or unknown a priori to the decision maker. Our model captures a large variety of real-world phenomena, yet maintains tractability that allows for a crisp characterization of the performance improvement that can be expected when more information might become available throughout the sequential decision process.

(2) Analysis of achievable performance. When it is known how to map auxiliary data to reward estimates, we establish lower bounds on the performance that is achievable by any non-anticipating policy, where performance is measured in terms of regret relative to the performance of an oracle that constantly selects the action with the highest mean reward. We further show that our lower bounds can be achieved through suitable policy design. These results identify the spectrum of minimax complexities associated with the MAB problem with unknown information arrival processes, as the following function of the realized information arrival process itself:

[TABLE]

where $c$ is a constant that depends on other problem parameters. The identified spectrum of minimax complexities ranges from the classical regret rates that appear in the stochastic MAB literature when there is no auxiliary information, to regret that is uniformly bounded over time when information arrives frequently and/or early enough. This provides a yardstick for evaluating the performance of policies, and a sharp criterion for optimality in the presence of auxiliary information arrivals.

When the mappings $\{\phi_{k}\}_{k=1}^{K}$ are unknown, one faces a difficult problem that essentially requires to learn synchronously from auxiliary data and on that data and how to interpret it. Nevertheless, focusing on the practical family of linear mappings, we identify a necessary and sufficient condition delineating, on the one hand, when auxiliary data still allows the performance improvement that is characterized by the case where these mappings are known, and on the other hand, when performance cannot improve based on auxiliary information, regardless of the timing and frequency of information arrivals.

(3) Policy design. We observe that, whether the mappings $\{\phi_{k}\}_{k=1}^{K}$ are known or unknown, policies that are rate optimal in the absence of auxiliary data may not be rate optimal anymore in the presence of auxiliary data. Nevertheless, when it is known how to map auxiliary data to reward estimates, we establish that polices that are based on upper confidence bounds and posterior sampling leverage auxiliary data naturally, without any prior knowledge on the information arrival process. In that sense, polices such as UCB1 and Thompson sampling exhibit remarkable robustness with respect to the information arrival process: without any adjustments, these policies achieve the best performance that can be achieved, guaranteeing rate optimality uniformly over a general class of information arrival processes. Moreover, the regret rates they guarantee cannot be improved even with prior knowledge of the information arrival process. This identifies a new important property of these popular policies, extends the class of problems to they apply, and further lends credence to their appeal.

When the mappings $\{\phi_{k}\}_{k=1}^{K}$ connecting auxiliary data to reward estimates are a priori unknown, we devise a new policy (termed 2-UCBs) that is adaptive with respect to both the unknown information arrival process and the unknown mappings, and establish its near optimality for the class of linear mappings. 2-UCBs is based on two different upper confidence bounds: one that accounts only for decisions-driven observations, and one that also accounts (in an optimistic manner) for auxiliary data.

Using data from a large media site, we analyze the value that may be captured by leveraging auxiliary information to recommend content with unknown impact on the future browsing path of readers. The auxiliary data we utilize consist of users that arrived to media pages directly from search, and the browsing path of these users following their arrival. We compare the performance of the 2-UCBs algorithm relative to non-adaptively relaying on estimates of $\left\{\phi_{k}\right\}_{k=1}^{K}$ formed based on historical data.

1.3 Related Work

Multi-Armed Bandits. Since its inception, the MAB framework has been adopted for studying a variety of applications including clinical trials (Zelen 1969), strategic pricing (Bergemann and Välimäki 1996), assortment selection (Caro and Gallien 2007), online auctions (Kleinberg and Leighton 2003), online advertising (Pandey et al. 2007), and product recommendations (Madani and DeCoste 2005, Li et al. 2010). For an overview of MAB formulations we refer the readers to Berry and Fristedt (1985) and Gittins et al. (2011) for Bayesian settings, as well as Cesa-Bianchi and Lugosi (2006) and Bubeck and Cesa-Bianchi (2012) for the so-called adversarial setting. A regret characterization for the more traditional formulation adopted in this paper, often referred to as the stochastic MAB problem, was first established by Lai and Robbins (1985), followed by analysis of policies such as $\epsilon_{t}$-greedy, UCB1, and Thompson sampling; see, e.g., Auer et al. (2002) and Agrawal and Goyal (2013a).

The MAB framework focuses on balancing exploration and exploitation under restrictive assumptions on the information collection process. Correspondingly, optimal policy design is typically predicated on the assumption that at each period a reward observation is collected only on the action that is selected at that time period (exceptions to this common information collection process are discussed below). Such policy design does not account for information that may become available between decision epochs, and that might be essential for achieving good performance in many practical settings. In the current paper we relax these assumptions on the information arrival process by allowing arbitrary information arrival processes. Our focus is on studying the impact of the information arrival characteristics on achievable performance and policy design when the information arrival process is a priori unknown.

Few MAB settings deviate from the information collection process described above by considering specific information arrival processes that are known a priori, as opposed to our formulation where the information arrival process is arbitrary and a priori unknown. In a well-studied contextual MAB setting (Woodroofe 1979) at each trial the decision maker observes a context carrying information about other arms. In the full-information adversarial MAB setting, rewards are arbitrary and can even be selected by an adversary (Auer et al. 1995, Freund and Schapire 1997). In this setting, after each decision epoch the agent has access to a reward observation from each arm. The adversarial nature of this setting makes it fundamentally different from ours in terms of achievable performance, analysis, and policy design. Shivaswamy and Joachims (2012) consider what could be viewed as a special case of our framework, where some initial reward observations are available at the beginning of the horizon.

Balancing and Regulating Exploration. Several papers have considered settings of dynamic optimization with partial information and distinguished between cases where myopic policies guarantee optimal performance, and cases where exploration is essential, in the sense that myopic policies may lead to incomplete learning and large losses; examples include several application domains, including dynamic pricing (Araman and Caldentey 2009, Farias and Van Roy 2010, Harrison et al. 2012, den Boer and Zwart 2013), inventory management (Besbes and Muharremoglu 2013), product recommendations (Bastani et al. 2018), and technology development (Lee et al. 2003). Furthermore it was shown that myopic policies maintain notions of optimality when future decisions are discounted (e.g., Woodroofe 1979, Sarkar 1991), or with access to sufficiently diverse contexts (Bastani et al. 2020). On the other hand, few papers have studied cases where exploration is not only essential but should be enhanced in order to maintain optimality; prominent examples include the partial monitoring feedback structure by Cesa-Bianchi et al. (2006), and the non-stationary MAB framework by Besbes et al. (2019).

The above studies demonstrate a variety of practical settings where the extent of exploration that is required to maintain optimality depends on problem characteristics that may often be a priori unknown to the decision maker. This introduces the challenge of dynamically endogenizing the rate at which a decision-making policy explores to approximate the best performance that is achievable under ex ante knowledge of the underlying problem characteristics. In this paper we address this challenge from an information collection perspective. We identify the connection between the information arrival process and the degree of exploration that is required to maintain rate optimality, and identify adaptive policies that guarantee rate optimality without prior knowledge on the information arrival process.

Recommender Systems. An active stream of literature has been focusing on modelling and maintaining connections between users and products in recommender systems; see, e.g., Ansari et al. (2000), the survey by Adomavicius and Tuzhilin (2005), and a book by Ricci et al. (2011). One key element that impacts the performance of recommender systems is the often limited data that is available. Focusing on the prominent information acquisition aspect of the problem, several studies (to which we referred earlier) have addressed sequential recommendation problems using a MAB framework where at each time period information is obtained only on items that are recommended at that period. Another approach is to identify and leverage additional sources of relevant information. Following that avenue, Farias and Li (2019) consider the problem of estimating user-item propensities, and propose a method to incorporate auxiliary data such as browsing and search histories to enhance the predictive power of recommender systems. While their work concerns with an offline prediction setting, our paper focuses on the impact of auxiliary information streams on policy design and performance in a sequential dynamic setting.

Conference Proceedings. An extended abstract that appeared in Gur and Momeni (2018) included some initial findings for the special case where at most one auxiliary observation is obtained for each arm at each time period, and where auxiliary data must consist of iid reward observations. Considering that setting, the focus of Gur and Momeni (2018) is on using virtual time indexes for developing an $\varepsilon_{t}$-greedy-type policy that adapts to an unknown information arrival matrix, albeit predicated on prior knowledge of $\Delta$, the minimal separation in mean rewards between the optimal action and suboptimal ones.111For completeness, in Appendix D we describe the virtual time indexes method and detail its analysis. The current paper considers a general framework where, in particular, auxiliary data can be realized from a general family of distributions that could be mapped to rewards estimates, and analyzes the cases where the underlying mappings are known or unknown a priori. Furthermore, our focus is on algorithms based on upper confidence bounds and posterior sampling that guarantee near optimality without requiring prior knowledge of $\Delta$, including in the setting considered in Gur and Momeni (2018).

2 Problem Formulation

We next formulate the class of multi-armed bandit problems with auxiliary information arrivals. Let $\mathcal{K}=\{1,\dots,K\}$ be a set of arms (actions) and let $\mathcal{T}=\{1,\dots,T\}$ denote a sequence of decision epochs. At each time period $t\in\cal{T}$, a decision maker selects one of the $K$ arms. When selecting an arm $k\in\cal{K}$ at time $t\in\cal{T}$, a reward $X_{k,t}\in\mathbb{R}$ is realized and observed. For each $t\in\cal T$ and $k\in\cal K$, the reward $X_{k,t}$ is assumed to be independently drawn from a distribution $\nu_{k}\in\mathcal{V}$ with mean $\mu_{k}$, where $\mathcal{V}$ is the collection of $\sigma^{2}$-sub-Gaussian distributions.222A real-valued random variable $X$ is said to be sub-Gaussian if there is some $\sigma>0$ such that for every $\lambda\in\mathbb{R}$ one has $\mathbb{E}e^{\lambda\left(X-\mathbb{E}X\right)}\leq e^{\sigma^{2}\lambda^{2}/2}$. This broad class of distributions includes, for instance, Gaussian random variables, as well as any random variable with a bounded support (if $X\in[a,b]$ then $X$ is $\frac{(a-b)^{2}}{4}$-sub-Gaussian) such as Bernoulli random variables. In particular, if a random variable is $\sigma^{2}$-sub-Gaussian, it is also $\tilde{\sigma}^{2}$-sub-Gaussian for all $\tilde{\sigma}>\sigma$. We denote by $\bm{\nu}=\left(\nu_{1},\dots,\nu_{K}\right)^{\top}$ the distribution of the rewards profile $\left(X_{1,t},\dots,X_{K,t}\right)^{\top}$. We assume that rewards are independent across time periods and arms. For simplicity, we assume that there exists a unique best action, and denote the highest expected reward and the best arm by $\mu^{\ast}$ and $k^{\ast}$, respectively:333For simplicity, when using the $\operatorname*{arg\,min}$ and $\operatorname*{arg\,max}$ operators we assume ties to be broken in favor of the smaller index.

[TABLE]

We denote by $\Delta_{k}=\mu^{\ast}-\mu_{k}$ the difference in expected reward between the best arm and arm $k$ and assume it is bounded. We denote by $\Delta$ a lower bound such that $0<\Delta\leq\min\limits_{k\in\mathcal{K}\setminus\{k^{\ast}\}}\Delta_{k}$. (We make no assumption on the separability of mean rewards and allow $\Delta$ to be a decreasing function of $T$.)

Information Arrival Process. Before each epoch $t$, the agent may or may not observe auxiliary information for some of the arms without pulling them. For each period $t$ and arm $k$, we denote by $h_{k,t}\in\mathbb{N}^{+}:=\{0\}\cup\mathbb{N}$ the number of auxiliary information arrivals on arm $k$ between decision epochs $t-1$ and $t$. We denote by $\bm{\mathrm{H}}$ the information arrival matrix with entries $h_{k,t}$; we assume that this matrix is independent of the policy’s actions. If $h_{k,t}\geq 1$, then between decision epochs $t-1$ and $t$ the agent observes $h_{k,t}$ independent random variables $Y_{k,t,m}\sim\nu_{k}^{\mathrm{aux}}\in\mathcal{V}^{\mathrm{aux}}$ with mean $y_{k}$, where $m\in\{1,\dots,h_{k,t}\}$, and where $\mathcal{V}^{\mathrm{aux}}$ is an a priori known class of allowed auxiliary observation distributions. For each arm $k$ we assume that there exists a mapping $\phi_{k}:\mathbb{R}\rightarrow\mathbb{R}$, through which an unbiased estimator for the mean reward $\mu_{k}$ can be constructed, that is, $\mathbb{E}\left[\phi_{k}(Y_{k,t,m})\right]=\mu_{k}$, and that $\phi_{k}(Y_{k,t,m})$ is $\hat{\sigma}^{2}$-sub-Gaussian for some $\hat{\sigma}>0$ and any $\nu_{k}^{\mathrm{aux}}\in\mathcal{V}^{\mathrm{aux}}$. (The mappings $\{\phi_{k}\}$ can be either known or unknown a priori to the decision maker, and we will analyze both cases; see discussion in §2.1.) We denote the class of allowed mappings that satisfy the former criterion by $\Phi$. We assume that the random variables $Y_{k,t,m}$ are independent across time periods and arms, and are also independent from reward realizations. We denote the vector of average information received between epochs $t-1$ and $t$ by $\bm{\mathrm{Z}}_{t}=\left(Z_{1,t},\dots,Z_{K,t}\right)^{\top}$ where for any $k$ one has $Z_{k,t}=\sum_{m=1}^{h_{k,t}}\phi_{k}(Y_{k,t,m})/\max\{1,h_{k,t}\}$.

Example 1

(Linear mappings)* A practical class of mappings $\Phi_{L}$ consists of linear mappings with an intercept at zero. Then, auxiliary data consists of realizations of independent random variables $Y_{k,t,m}$ for which there is a vector $(\alpha_{1},\ldots,\alpha_{K})\in\leavevmode\nobreak\ \mathbb{R}^{K}$ such that for all $k$, $t$, and $m$ one has $\mathbb{E}[\alpha_{k}Y_{k,t,m}]=\mu_{k}$.*

This simple class of mappings illustrates the possibility of utilizing available data in many domains, including maximizing conversions of product recommendations (for example, in terms of purchases of recommended products). The practicality of using linear mappings for improving product recommendations will be demonstrated empirically in §6 using data from a large media site.

Policies and Performance. \Copyadmissible policyLet $U$ be a random variable defined over a probability space $(\mathbb{U},\mathcal{U},\bm{\mathrm{P}}_{u})$, and denote $\bm{h}_{t}=\left(h_{1,t},\dots,h_{K,t}\right)^{\top}$. Let $\pi_{t}:\mathbb{R}^{t-1}\times\mathbb{R}^{K\times t}\times\{\mathbb{N}^{+}\}^{K\times t}\times\mathbb{U}\rightarrow\mathcal{K}$ for $t=1,2,\dots$ be measurable functions (with some abuse of notation we also denote the action at time $t$ by $\pi_{t}\in\mathcal{K}$) given by $\pi_{t}=\pi_{t}(X_{\pi_{t-1},t-1},\dots,X_{\pi_{1},1},\bm{\mathrm{Z}}_{t},\dots,\bm{\mathrm{Z}}_{1},\bm{h}_{t},\dots,\bm{h}_{1},U)$ for $t=2,3,\dots$, and $\pi_{1}=\pi_{1}(\bm{\mathrm{Z}}_{1},\bm{h}_{1},U)$.

The mappings $\{\pi_{t}:t=1,\dots,T\}$, together with the distribution $\bm{\mathrm{P}}_{u}$ define the class of admissible policies denoted by $\mathcal{P}$. Policies in $\mathcal{P}$ depend only on the past history of actions and observations as well as auxiliary information arrivals, and allow for randomization via their dependence on $U$. We denote by

[TABLE]

the class of problems that includes pairs of allowed reward distribution profiles $\left(\bm{\nu},\bm{\nu}^{\mathrm{aux}}\right)$.

We evaluate the performance of a policy $\pi\in\mathcal{P}$ by the regret it incurs under information arrival process $\bm{\mathrm{H}}$ relative to the performance of an oracle that selects the arm with the highest expected reward. For a given instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$, we define the cased-dependent regret by:

[TABLE]

where the expectation $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}[\cdot]$ is taken with respect to the noisy rewards and noisy auxiliary observations, as well as to the policy’s actions (throughout the paper we will denote by $\mathbb{P}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$, $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$, and $\mathbb{R}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$ the probability, expectation, and regret when the arms are selected according to policy $\pi$, rewards are distributed according to $\bm{\nu}$, and auxiliary observations are distributed according to $\bm{\nu}^{\mathrm{aux}}$). We note that regret is incurred over decisions made in epochs $t=1,\ldots,$T; the main distinction relative to classical regret formulations is that in (1) the mappings $\left\{\pi_{t};\;t=1,\ldots,T\right\}$ can be measurable with respect to the $\sigma$-field that also includes information that arrives between decision epochs, as captured by the matrix $\bm{\mathrm{H}}$. Finally, given an information arrival matrix $\bm{\mathrm{H}}$, we denote by

[TABLE]

the worst-case regret guaranteed by policy $\pi$, and the best guaranteed performance, respectively.

2.1 Discussion: Prior Knowledge on Mappings

In our model, the decision maker may estimate mean rewards based on auxiliary observations through the mappings $\left\{\phi_{k}\right\}$. In §3 and §4, to study in isolation the implications of the information arrival matrix $\bm{\mathrm{H}}$, we assume that these mappings are a priori known. The interpretability of feedback signals for the purpose of forming mean rewards estimates is a simplifying modeling assumption that is essential for tractability in many online dynamic optimization frameworks, including the MAB literature.444Typically in the MAB literature feedback is known to consist of reward observations. However, even when this relation between feedback and rewards is more general, e.g., in Russo et al. (2018), it is assumed to be a priori known. Therefore this part could be viewed as extending prior work by relaxing assumptions on the information collection process that are common in the MAB literature, while offering assumptions on the information structure (and what is known about it) that are comparable to what is assumed in existing literature.

The challenge of adapting to unknown mappings is the focus of §5, where we develop an approach for leveraging auxiliary information when these mappings are unknown. The approaches from §4 and §5 are then compared in §6 where we use data from a large US media site with the objective of leveraging auxiliary information (actions of readers that arrived to content pages directly from a search engine) to improve in-site recommendations. Performance is compared along two characteristics of problem instances: $(i)$ the effectiveness of auxiliary information, and $(ii)$ the mapping misspecification.

3 Impact of Information Arrival Process on Achievable Performance

In this section we study the impact that the information arrival process (particularly, the matrix $\bm{\mathrm{H}}$) may have on the performance one could aspire to achieve when the mappings $\{\phi_{k}\}$ are known a priori. Our first result formalizes what cannot be achieved, establishing a lower bound on the best achievable performance as a function of the information arrival process.

Theorem 1

(Lower bound on the best achievable performance)* For any $T\geq 1$ and information arrival matrix $\bm{\mathrm{H}}$, the worst-case regret for any admissible policy $\pi\in\mathcal{P}$ (with knowledge of $\{\phi_{k}\}$) is bounded from below as follows:*

[TABLE]

where LABEL:lower_bound1, LABEL:lower_bound2, and LABEL:lower_bound3 are positive constants that only depend on $\sigma$ and $\hat{\sigma}$.

The precise expressions of LABEL:lower_bound1, LABEL:lower_bound2, and LABEL:lower_bound3 are provided in the discussion below. Theorem 1 establishes a lower bound on the achievable performance that depends on an arbitrary path of information arrivals, as captured by the elements of the matrix $\bm{\mathrm{H}}$. In that sense, Theorem 1 provides a spectrum of bounds on the achievable performances, mapping many potential information arrival trajectories to the best performance they allow. When there is no additional information over what is assumed in the classical MAB setting, one recovers a lower bound of order $\frac{K}{\Delta}\log T$ that was established for that setting (see Lai and Robbins 1985, Bubeck et al. 2013). Theorem 1 further suggests that in the presence of auxiliary observations, achievable regret rates may become lower, and that the impact of information arrivals on the achievable performance depends on the frequency of these arrivals, but also on the time at which these arrivals occur; we further discuss these observations in §3.1.

Key Ideas in the Proof. The proof of Theorem 1 adapts to our framework ideas of identifying a worst-case nature “strategy” (see, e.g., Bubeck et al. 2013). While the full proof is deferred to the appendix, we next illustrate its key ideas using the special case of two arms. We assume that the decision maker is a priori informed that the first arm generates rewards according to a normal distribution with standard variation $\sigma$ and a mean that is either $-\Delta$ (according to distribution denoted by $\bm{\nu}$) or $+\Delta$ (according to a distribution denoted by $\tilde{\bm{\nu}}$), and that the second arm generates rewards with normal distribution of standard variation $\sigma$ and mean zero. To quantify a notion of distance between the possible profiles of reward distributions we use the Kullback-Leibler (KL) divergence. The KL divergence between two positive measures $\rho$ and $\rho^{\mathrm{aux}}$ with $\rho$ absolutely continuous with respect to $\rho^{\prime}$, is defined as:

[TABLE]

where $\mathbb{E}_{\rho}$ denotes the expectation with respect to $\rho$. Using Lemma 2.6 from Tsybakov (2009) that connects the KL divergence to error probabilities, we establish that at each period $t$ the probability of selecting a suboptimal arm must be at least

[TABLE]

where $\tilde{n}_{1,t}$ denotes the number of times the first arm is selected up to time $t$. Each selection of suboptimal arm contributes $\Delta$ to the regret, and therefore the cumulative regret must be at least $\Delta\sum\limits_{t=1}^{T}p^{\mathrm{sub}}_{t}$. We further observe that if arm 1 has a mean reward of $-\Delta$, the cumulative regret must also be at least $\Delta\cdot\mathbb{E}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}[\tilde{n}_{1,T}]$. Therefore the regret is lower bounded by $\frac{\Delta}{2}\left(\sum\limits_{t=1}^{T}p^{\mathrm{sub}}_{t}+\mathbb{E}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}[\tilde{n}_{1,T}]\right)$, which is greater than $\frac{\sigma^{2}}{4\Delta}\log\left(\frac{\Delta^{2}}{2\sigma^{2}}\sum\limits_{t=1}^{T}\exp\left(-\frac{2\Delta^{2}}{\hat{\sigma}^{2}}\sum\limits_{s=1}^{t}h_{1,s}\right)\right)$. The argument can be repeated by switching arms $1$ and $2$. For $K$ arms, we follow the above lines and average over the established bounds to obtain:

[TABLE]

which establishes the result.

3.1 Discussion and Subclasses of Information Arrival Processes

Theorem 1 suggests that auxiliary information arrivals may be leveraged to reduce regret rates, and that their impact on the achievable performance increases when information arrives more frequently, and earlier. We next demonstrate these implications using two information arrival processes of natural interest: a process with a fixed arrival rate, and a process with a decreasing arrival rate.

3.1.1 Stationary Information Arrival Process

Assume that $h_{k,t}$’s are i.i.d. random variables with mean $\lambda$. Then, for any $T\geq 1$ and admissible policy $\pi\in\mathcal{P}$, one obtains the following lower bound for the achievable performance:

If $\lambda\leq\frac{\hat{\sigma}^{2}}{4\Delta^{2}T}$, then

[TABLE] 2. 2.

If $\lambda\geq\frac{\hat{\sigma}^{2}}{4\Delta^{2}T}$, then

[TABLE]

This class includes instances in which, on average, information arrives at a constant rate $\lambda$. Analyzing these arrival processes reveals two different regimes. When the information arrival rate is sufficiently low (there are no more than order $\Delta^{-2}$ information arrivals over $T$ time periods), auxiliary observations become essentially ineffective, and one recovers the performance bounds established in the absence of auxiliary observations. When $\Delta$ is fixed and independent of the horizon length $T$, the lower bound scales logarithmically with $T$. When $\Delta$ can scale with $T$, a bound of order $\sqrt{T}$ is recovered when $\Delta$ is of order $T^{-1/2}$. In both cases, there are known policies that guarantee rate-optimal performance (for details see policies, analysis, and discussion in Auer et al. 2002).

On the other hand, when there are more than order $\Delta^{-2}$ observations over $T$ periods, the lower bound on the regret depends on the arrival rate $\lambda$. When the arrival rate is independent of the horizon length $T$, the regret is bounded by a constant that is independent of $T$, and a myopic policy (e.g., a policy that for the first $K$ periods pulls each arm once, and at each later period pulls the arm with the current highest estimated mean reward, while randomizing to break ties) is optimal; For analysis and details see sections C.1 and C.2 of the appendix.

3.1.2 Diminishing Information Arrival Process

Assume that $h_{k,t}$’s are random variables such that for each arm $k\in\mathcal{K}$ and time step $t$,

[TABLE]

for some fixed $\kappa>0$. Then, for any $T\geq 1$ and admissible policy $\pi\in\mathcal{P}$, one obtains the following lower bounds for the achievable performance:

If $\kappa<1$ then:

[TABLE] 2. 2.

If $\kappa>1$ then:

[TABLE]

This class includes information arrival processes under which the expected number of information arrivals up to time $t$ is of order $\log t$. Therefore, it demonstrates the impact of the timing of information arrivals on the achievable performance, and suggests that a constant regret may be achieved when the rate of information arrivals is decreasing. Whenever $\kappa<1$, the lower bound on the regret is logarithmic in $T$, and there are known policies that guarantee rate-optimal performance (see Auer et al. 2002). When $\kappa>1$, the lower bound becomes a constant, and one may observe that when $\kappa$ is large enough a myopic policy is asymptotically optimal. (In the limit $\kappa\rightarrow 1$ the lower bound is of order $\log\log T$.) For analysis and details see sections C.3 and C.4 of the appendix.

3.1.3 Discussion

One may contrast the classes of information arrival processes described in §3.1.1 and §3.1.2 by selecting $\kappa=\frac{2\Delta^{2}\lambda T}{\sigma^{2}\log T}$. Then, in both settings the total number of information arrivals for each arm is $\lambda T$ on average. However, while in the first class the information arrival rate is fixed over the horizon, in the second class this arrival rate is higher in the beginning of the horizon and decreases over time. The different timing of the $\lambda T$ information arrivals may lead to different regret rates. For example, selecting $\lambda=\frac{\sigma^{2}\log T}{\Delta^{2}T}$ implies $\kappa=2$. Then, the lower bound in §3.1.1 is logarithmic in $T$ (establishing the impossibility of constant regret in that setting), but the lower bound in §3.1.2 is constant and independent of $T$ (in §4 we show that constant regret is indeed achievable in this setting). This observation conforms to the intuition that earlier observations have higher impact on achievable performance, as at early periods there is only little information that is available (and therefore the marginal impact of an additional observation is larger), and since earlier information can be used for more decisions (as the remaining horizon is longer).555The subclasses described in §3.1.1 and §3.1.2 are special cases of the following setting. Let $h_{k,t}$’s be independent random variables such that for each arm $k$ and time $t$, the expected number of information arrivals up to time $t$ satisfies:

$\mathbb{E}\left[\sum_{s=1}^{t}h_{k,s}\right]=\lambda T\frac{t^{1-\gamma}-1}{T^{1-\gamma}-1}.\vspace{-0.10cm}$

While the expected number of total information arrivals for each arm, $\lambda T$, is determined by the parameter $\lambda$, the concentration of arrivals is governed by the parameter $\gamma$. When $\gamma=0$ the arrival rate is constant, corresponding to the class described in §3.1.1. As $\gamma$ increases, information arrivals concentrate in the beginning of the horizon, and $\gamma\rightarrow 1$ leads to $\mathbb{E}\left[\sum_{s=1}^{t}h_{k,s}\right]=\lambda T\frac{\log t}{\log T}$, corresponding to the class in §3.1.2. Then, when $\lambda T$ is of order $T^{1-\gamma}$ or higher, the lower bound is a constant independent of $T$.

The analysis above suggests that effective optimal policy design might depend on the information arrival process: while policies that explore (and in that sense are not myopic) may be rate-optimal in some cases, a myopic policy that does not explore (except perhaps in a small number of periods in the beginning of the horizon) can be rate-optimal in other cases. However, this identification of rate-optimal policies relies on prior knowledge of the information arrival process. In the following sections we therefore study the adaptation of polices to unknown information arrival processes, in the sense of guaranteeing rate-optimality without any prior knowledge on the information arrival process.

4 Natural Adaptation to the Information Arrival Process

In this section we establish that, when the mappings $\{\phi_{k}\}$ are known, policies based on upper confidence bounds and posterior sampling exhibit natural robustness to the information arrival process in the sense of achieving the lower bound of Theorem 1 uniformly over the general class of information arrival processes we consider, without any prior knowledge on the information arrival matrix $\bm{\mathrm{H}}$.

4.1 Robustness of UCB

We next consider a UCB1 policy (Auer et al. 2002) in which in addition to updating mean rewards and observation counters after the policy’s actions, these are also updated after auxiliary observations. Denote by n^{\text{\tiny\mathrm{known}}}_{k,t} and \bar{X}_{k,t}^{\text{\tiny\mathrm{known}}} the weighted number of times a sample from arm $k$ has been observed and the weighted empirical average reward of arm $k$ up to time $t$, respectively:

[TABLE]

UCB1 with Auxiliary Observations (aUCB1). Inputs: a constant $c$.

At each period $t=1,\dots,T$:

1. (a)

Observe the vectors $\bm{h}_{t}$ and $\bm{\mathrm{Z}}_{t}$ 2. (b)

Select the arm

[TABLE] 3. (c)

Receive and observe a reward $X_{\pi_{t},t}$

The next result establishes that by updating counters and reward estimates after both policy’s actions and auxiliary information arrivals, the UCB1 policy guarantees rate-optimal performance over the class of general information arrival processes defined in §2.

Theorem 2** (Near optimality of UCB1 with auxiliary observations)**

Assume that the mappings $\{\phi_{k}\}_{k=1}^{K}$ are known, and let $\pi$ be UCB1 with auxiliary observations, tuned by $c>2$. Then, for any $T\geq 1$, $K\geq 2$, and additional information arrival matrix $\bm{\mathrm{H}}$:

[TABLE]

where LABEL:ucb1, and LABEL:ucb2 are positive constants that depend only on $\sigma$ and $\hat{\sigma}$.

The upper bound in Theorem 2 holds for any arbitrary trajectory of information arrivals that is captured by the matrix $\bm{\mathrm{H}}$, and matches the lower bound in Theorem 1 with respect to that trajectory (captured by $h_{k,t}$’s), and with respect to the time horizon $T$, the number of arms $K$, the minimum expected reward difference $\Delta$, as well as $\sigma$ and $\hat{\sigma}$. This establishes that, by updating counters and reward estimates after both policy’s actions and auxiliary information arrivals, UCB1 guarantees rate optimality uniformly over the general class of information arrival processes defined in §2.

When put together, Theorems 1 and 2 establish the spectrum of minimax complexities associated with our class of problems, which is characterized as follows.

Remark 1

(Minimax complexity)* Theorems 1 and 2 together identify the spectrum of minimax regret rates for the class of MAB problem with unknown information arrival process (but with prior knowledge of mappings $\{\phi_{k}\}$), as a function of the realized information arrival process matrix $\bm{\mathrm{H}}$:*

[TABLE]

where $c$ is a constant that depends on problem parameters such as $K$, $\Delta$, and $\sigma$.

Remark 1 has a couple of important implications. First, it identifies a spectrum of minimax regret rates ranging from the classical regret rates that were established in the stochastic MAB literature when there is no auxiliary information, to regret that is uniformly bounded over time when information arrives frequently and/or early enough. Furthermore, identifying the minimax complexity provides a yardstick for evaluating the performance of policies, and a sharp criterion for identifying rate-optimal ones. This already identifies UCB1 as rate optimal, but will shortly be used also for establishing the optimality (or sub-optimality) of other policies.

Key Ideas in the Proof. The proof of Theorem 2 adjusts the analysis of UCB1 in Auer et al. (2002). Pulling a suboptimal arm $k$ at time step $t$ implies at least one of the following: $(i)$ the empirical average of the best arm deviates from its mean; $(ii)$ the empirical mean of arm $k$ deviates from its mean; or $(iii)$ arm $k$ has not been pulled sufficiently often in the sense that

[TABLE]

where $\hat{l}_{k,t}=\frac{4c\sigma^{2}\log\left(\tau_{k,t}\right)}{\Delta^{2}}$ with $\tau_{k,t}\coloneqq\sum\limits_{s=1}^{t}\exp\left({\frac{\Delta_{k}^{2}}{4c\hat{\sigma}^{2}}\sum\limits_{\tau=s}^{t}h_{k,\tau}}\right)$, and $\tilde{n}_{k,t-1}$ is the number of times arm $k$ is pulled up to time $t$. The probability of the first two events can be bounded using Chernoff-Hoeffding inequality, and the probability of the third one can be bounded using:

[TABLE]

Applying these bounds, we established the Theorem for $c>2$.

4.2 Robustness of Thompson Sampling

Consider a Thompson sampling policy with Gaussian priors (Agrawal and Goyal 2012, 2013a), but where posteriors are updated both after the policy’s actions and after auxiliary observations. We denote by n^{\text{\tiny\mathrm{known}}}_{k,t} and \bar{X}_{k,t}^{\text{\tiny\mathrm{known}}} the observation counters and mean reward estimates as defined in (2).

Thompson sampling with auxiliary observations (aTS). Inputs: a tuning constant $c$.

Initialization: set initial counters $n_{k,0}=0$ and initial empirical means $\bar{X}_{k,0}=0$ for all $k\in\cal K$ 2. 2.

At each period $t=1,\dots,T$:

1. (a)

Observe the vectors $\bm{h}_{t}$ and $\bm{\mathrm{Z}}_{t}$ 2. (b)

Sample \theta_{k,t}\sim\mathcal{N}(\bar{X}_{k,t}^{\text{\tiny\mathrm{known}}},c\sigma^{2}(n^{\text{\tiny\mathrm{known}}}_{k,t}+1)^{-1}) for all $k\in\cal K$, and select the arm

[TABLE] 3. (c)

Receive and observe a reward $X_{\pi_{t},t}$

The next result establishes that by deploying a Thompson sampling policy that updates posteriors after the policy’s actions and whenever auxiliary information arrives, one may guarantee rate optimal performance in the presence of unknown information arrival processes.

Theorem 3

(Near optimality of Thompson sampling with auxiliary observations)* Assume that the mappings $\{\phi_{k}\}_{k=1}^{K}$ are known, and let $\pi$ be a Thompson sampling with auxiliary observations policy, with $c>0$. For every $T\geq 1$, and auxiliary information arrival matrix $\bm{\mathrm{H}}$:*

[TABLE]

for some absolute positive constants LABEL:TS1, LABEL:TS2, and LABEL:TS3 that depend on only $c$, $\sigma$ and $\hat{\sigma}$.

The upper bound in Theorem 3 holds for any trajectory of information arrivals that is captured by the matrix $\bm{\mathrm{H}}$, and matches the lower bound in Theorem 1 with respect to the dependence on that trajectory (captured by $h_{k,t}$’s), as well as $T$, $K$, $\Delta$, $\sigma$, and $\hat{\sigma}$. This establishes that, similarly to UCB1, by updating posteriors after both policy’s actions and auxiliary information arrivals, Thompson sampling guarantees rate optimality uniformly over the general class of information arrival processes defined in §2.

Key Ideas in the Proof. To establish Theorem 3 we decompose the regret associated with each suboptimal arm $k$ into three components: $(i)$ Regret from selecting arm $k$ when its empirical mean deviates from its expectation; $(ii)$ Regret from selecting arm $k$ when its empirical mean does not deviate from its expectation, but $\theta_{k,t}$ deviates from the empirical mean; and $(iii)$ Regret from selecting arm $k$ when its empirical mean does not deviate from its expectation and $\theta_{k,t}$ does not deviate from the empirical mean. Following the analysis in Agrawal and Goyal (2013b) one may use concentration inequalities to bound the cumulative regret associated with $(i)$ and $(iii)$ uniformly over time.

To analyze $(ii)$, we consider an arrival process with a rate that is decreasing exponentially with each arrival. To bound the number of arrivals we establish the following lemma.

Lemma 1

Fix an integer $T\geq 1$. For every integer $1\leq t\leq T$, let $A_{t}\in\{0,1\}$ be a binary random variable. Let $w_{t}\geq 0$, $1\leq t\leq T$, be some deterministic constants, and define $w_{T}=0$. Define $N_{t}=\sum\limits_{s=1}^{t-1}\left(A_{s}+w_{s}\right)+w_{t}$, and assume $\mathbb{E}\left[A_{t}\;\middle|\;N_{t}\right]\leq p^{N_{t}+1}$ for some $p\leq 1$. Then, for every $1\leq t\leq T+1$:

[TABLE]

Proof.    Consider the sequence $\{p^{-N_{t}}:\;t=1,\ldots,T\}$. For every $1\leq t\leq T$,

[TABLE]

Using this inequality and applying a simple induction, one obtains $\mathbb{E}\left[p^{-N_{t}}\right]\leq\sum\limits_{s=1}^{t}p^{-\sum\limits_{\tau=s}^{t}w_{\tau}},$ for every $t\in\{1,\ldots,T\}$. Finally, by Jensen’s inequality, one has $p^{-\mathbb{E}\left[N_{t}\right]}\leq\mathbb{E}\left[p^{-N_{t}}\right],$ which implies

[TABLE]

This concludes the proof of the lemma.

Using Lemma 1 we establish that the probability of selecting a sub-optimal action when event (ii) occurs drops exponentially whenever that action is selected or an auxiliary observation is received (similarly to the arrival probability in the setting of Lemma 1). This, in turn, enables one to bound the expected number of times a suboptimal arm is selected when event (ii) occurs, and to establish Theorem 3.

4.3 Discussion

Theorems 2 and 3 establish that, when the mappings $\{\phi_{k}\}$ are known, both UCB1 and Thompson sampling guarantee rate optimality uniformly over the class of information arrival processes defined in §2. Figure 2 visualizes the regret incurred by UCB1 and Thompson sampling for the stationary information arrivals as the rate of arrivals increases; tight upper bounds for the settings in §3.1.1 and §3.1.2 for any values of $\lambda$ and $\kappa$ are given in Appendix C (Corollaries 1 and 2), and numerical analysis with a variety of tuning parameters and information arrival scenarios is provided in Appendix E.

Our results imply that UCB1 and Thompson sampling possess remarkable robustness with respect to the information arrival process: without any adjustments they guarantee the best achievable performance (up to some multiplicative constant). Furthermore, the regret rate guaranteed by these policies cannot be improved by any other policy, including ones that are predicated on accurate prior knowledge of the information arrival process. These results therefore uncover an important novel property of these policies that extends the class of problems to which they have been applied.

To appreciate the above robustness property, it is important to observe that, in general, policies that are rate optimal in the absence of auxiliary information arrivals do not necessarily maintain optimality under the general information arrival process considered here. For example, policies with exogenous exploration rate, such as forced sampling and $\epsilon_{t}$-greedy, do not exhibit such robustness (as we observe in Appendix D.1). The main reason for the sub-optimality of these policies in the presence of auxiliary information arrivals is that regardless of the amount of auxiliary observations collected, they always incur a certain loss due to their predetermined exploration schedule. Nevertheless, we note that it is possible to adjust such policies by endogenizing their exploration rate and adjusting it dynamically when auxiliary information is collected. For completeness, in Appendix §D we detail a method of doing so using virtual time indices that are advanced multiplicatively upon auxiliary information arrivals. We further provide an adjusted $\epsilon_{t}$-greedy-type policy with virtual time indexes, and establish it guarantees rate-optimality in the presence of auxiliary information arrivals. (We note that similarly to $\epsilon_{t}$-greedy-type policies, this policy requires the accurate minimal expected reward difference $\Delta$ as an input.)

5 Leveraging Auxiliary Information with Unknown Mappings

In the previous section we have established the rate optimality of UCB1 and Thompson sampling in the presence of auxiliary information. Importantly, the rate optimality of these policies is predicated on precise prior knowledge of the mappings $\left\{\phi_{k}\right\}$ that allow the interpretation of auxiliary data for constructing reward estimates. When the mappings $\left\{\phi_{k}\right\}$ are misspecified, these policies are not rate optimal anymore, and the performance they achieve may deteriorate (and may even become worse than the performance they would have achieved without access to auxiliary data; this phenomenon will be demonstrated in §6). In the current section, we address the challenge of designing adaptive policies that leverage auxiliary information without precise prior knowledge of the mappings $\left\{\phi_{k}\right\}$. For simplicity, we assume that mappings are known to belong to the class $\Phi_{L}$ of linear mappings described in Example 1 with $0\leq\underline{\alpha}\leq\alpha_{k}\leq\bar{\alpha}$ for some known $\bar{\alpha}$ and $\underline{\alpha}$, and assume that auxiliary observations have non-negative mean, that is, $y_{k}\geq 0$ for all $k\in\cal K$. (We note that extending the analysis of this section to the class of linear mappings that includes non-zero intercepts is immediate.)

It is important to observe that when the mappings $\{\phi_{k}\}$ are unknown, one could always construct instances where regret rates that are achievable in the absence of auxiliary information must be incurred by any admissible policy, even when auxiliary data is abundant, as we next demonstrate.

Example 2

(No performance gain with unknown mappings)* Let $K=2$ and let $\Phi=\Phi_{L}$ be a class of linear mappings from Example 1. Assume it is known that $\alpha_{k}\in\{\alpha_{1},\alpha_{2}\}$ for $k\in\{1,2\}$ and that $\mu_{1}\in\{\mu_{2}+\Delta,\mu_{2}-\Delta\}$. Suppose that for all $t\in\{1,\ldots,T\}$ and $m\in\{0,1,\ldots,h_{1,t}\}$, auxiliary observations obtain $Y_{1,t,m}=y_{1}=\frac{\mu_{2}+\Delta}{\alpha_{2}}=\frac{\mu_{2}-\Delta}{\alpha_{1}}$ with probability 1. Note that if $\mu_{1}=\mu_{2}-\Delta$, then $\mu_{1}=E[\alpha_{1}Y_{1,t,m}]$, and if $\mu_{1}=\mu_{2}+\Delta$, then $\mu_{1}=E[\alpha_{2}Y_{1,t,m}]$. Therefore auxiliary observations are uninformative for identifying $\mu_{1}$. Following the proof of Theorem 1, one obtains that for any $T\geq 1$ and information arrival matrix $\bm{\mathrm{H}}$, the regret that any policy $\pi\in\mathcal{P}$ may guarantee is bounded from below as follows:*

[TABLE]

where LABEL:lower_bound1 and LABEL:lower_bound2 are the same positive constants that appear in the statement of Theorem 1.

We therefore turn to derive instance-dependent bounds on regret as opposed to guarantees over a class of problems. To provide meaningful instance-dependent lower bounds we focus on policies that are rate-optimal in the absence of auxiliary data. This set of policies can be characterized as follows.666A similar definition (super-fast convergent) is given in Garivier et al. (2019). Note that there is a variety of policies that guarantee the above regret rate; see, e.g., Auer et al. (2002) that studies a class of problem instances with bounded support.

Definition 1** (Information-agnostic-optimal policies)**

We say a policy $\pi\in\cal{P}$ is information-agnostic-optimal (IAO) over the class of problem instances ${\cal S}$ if for any problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$, any suboptimal arm $k\in\mathcal{K}$, and horizon length $T\geq 2$,

[TABLE]

for some constant $C_{\pi,{\cal S}}>0$. We denote the class of such policies by $\cal{P}^{\mathrm{IAO}}$.

5.1 Identifying Different Performance Regimes

As demonstrated in Example 2, when the mappings $\{\phi_{k}\}$ are unknown it is not clear a priori if and when one should expect performance improvement by utilizing auxiliary data. As a preliminary step, in this subsection we consider a simple version of our problem where prior to time $t=1$ the decision maker has access to the means of the auxiliary observations $\{y_{k}\}$, and study under what circumstances better performance is achievable compared to classical regret rates that are established in the absence of auxiliary information. Concretely, we assume access to a single noiseless auxiliary observation on each arm, that is, $\hat{\sigma}=0$ and $h_{k,1}=1$ for $k\in\mathcal{K}$. Upon collecting the auxiliary means $\{y_{k}\}$, the agent’s belief regarding the possible values of mean rewards changes. For each $k\in\mathcal{K}$ the agent learns that $\mu_{k}\in[\underline{\alpha}\cdot y_{k},\bar{\alpha}\cdot y_{k}]$; Figure 3 depicts different scenarios in the case of two arms and $y_{1}\geq y_{2}$. Depending on the value of $\mu_{1}$, we consider two different regimes. In Regime 1, $\mu_{1}$ is not in the intersection of the posterior support of the mean rewards: $\mu_{1}\in(\max\{\underline{\alpha}\cdot y_{1},\bar{\alpha}\cdot y_{2}\},\bar{\alpha}\cdot y_{1}]$. In Regime 2, $\mu_{1}$ is inside that intersection: $\mu_{1}\in[\underline{\alpha}\cdot y_{1},\max\{\underline{\alpha}\cdot y_{1},\bar{\alpha}\cdot y_{2}\}]$.

Consider the following variant of UCB1, to which we refer as UCB1+, that conditional on observing the auxiliary means $\{y_{k}\}$, deploys for each arm $k\in\cal K$ an upper confidence bound that is bounded from above by $\bar{\alpha}\cdot y_{k}$, the upper bound on the posterior support of $\mu_{k}$. Formally, for each $k$ and $t$ we define:

[TABLE]

where $\bar{X}^{\pi}_{k,t}$ and $n^{\pi}_{k,t}$ are the empirical mean reward and the number of pulls (see Table 2). We next characterizes the performance of this policy when the auxiliary means $\{y_{k}\}$ are observed before $t=1$.

Theorem 4

(Near optimality of UCB1+)* Let $\Phi=\Phi_{L}$ with $0\leq\underline{\alpha}\leq\alpha_{k}\leq\bar{\alpha}$ for some known $\underline{\alpha},\bar{\alpha}$, where $\hat{\sigma}=0$. Let $\pi$ be the UCB1+ policy, tuned by $c>2$. Then, for every $T\geq 1$, auxiliary information arrival matrix $\bm{\mathrm{H}}$ such that $h_{k,1}=1$ for $k\in\mathcal{K}$, and problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$, one has:*

[TABLE]

where the constants LABEL:ucb1 and LABEL:ucb2 appear in Theorem 2. Furthermore, this policy is optimal among the class $\cal{P}^{\mathrm{IAO}}$ up to a multiplicative constant that is independent of $T$, and logarithmic in $K$ and $\Delta_{k}$’s.

Theorem 4 suggests that when the mappings $\{\phi_{k}\}$ are unknown, auxiliary information on arm $k$ may still be leveraged to reduce the loss from selecting that arm (potentially to a regret that is uniformly bounded over time by a constant), but only when $\bar{\alpha}y_{k}<\mu^{\ast}$; only then, auxiliary information can effectively reduce the upper confidence bound associated with that arm. Recalling Figure 3, one may observe that performance improvement is therefore only possible in Regime 1, and the value of $\underline{\alpha}$ does not impact the regret rate. Henceforth we simply let $\underline{\alpha}=0$.

5.2 Impact of Unknown Mappings on Achievable Performance

In this subsection, we study the performance that one could aspire to achieve in the absence of prior knowledge of true mappings $\{\phi_{k}\}$, and under an arbitrary and unknown information arrival matrix $\bm{\mathrm{H}}$. The next result formalizes what cannot be achieved in the presence of unknown mappings, establishing instance-dependent lower bounds on the best achievable performance.

Theorem 5

(Lower bound on achievable performance with unknown mappings)* Let $\Phi=\leavevmode\nobreak\ \Phi_{L}$ with $0\leq\alpha_{k}\leq\bar{\alpha}$ for some known $\bar{\alpha}$. Then, for any $T\geq 1$ and $k\in\cal{K}$ and for any policy $\pi\in\cal{P}^{\mathrm{IAO}}$, the followings hold with $\delta_{k}=\mu^{\ast}-\bar{\alpha}\cdot y_{k}$:*

If $\bar{\alpha}\cdot y_{k}>\mu^{\ast}$ then, $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t}}[n_{k,T+1}^{\pi}]\geq\frac{\ref*{lower-bound-unknown1}}{\Delta_{k}^{2}}\log\left(\frac{\ref*{lower-bound-unknown2}\min\{4\Delta_{k}^{4},(\Delta_{k}-\delta_{k})^{2}\delta_{k}^{2}\}}{K\log T}T\right);$ 2. 2.

If $\bar{\alpha}\cdot y_{k}<\mu^{\ast}$ then, $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t}}[n_{k,T+1}^{\pi}]\geq\frac{\ref*{lower-bound-unknown1}}{\Delta_{k}^{2}}\log\left(\frac{\ref*{lower-bound-unknown2}(\Delta_{k}+\delta_{k})^{2}\delta^{2}_{k}}{K\log T}\sum_{t=1}^{T}\exp\left(-\ref*{lower-bound-unknown3}\delta_{k}^{2}\cdot\sum\limits_{s=1}^{t}h_{k,s}\right)\right),$

where $\ltx@label{lower-bound-unknown1}C_{9}$, $\ltx@label{lower-bound-unknown2}C_{10}$, and $\ltx@label{lower-bound-unknown3}C_{11}$ are positive constants that only depend on $\sigma$, $\hat{\sigma}$, and $\bar{\alpha}$.

The product $\bar{\alpha}\cdot y_{k}$ represents an optimistic mean reward estimate based on auxiliary information. Depending on whether it is larger or smaller than $\mu^{\ast}$, the above lower bound can be logarithmic in the horizon length $T$ or a decreasing log-sum-exp function of the cumulative information arrivals, respectively.

Key Ideas in the Proof. The proof of Theorem 5 relies on the following lemma which provides regret lower bound for any general class of bandit problems.

Lemma 2

(General Lower Bound with Unknown Mappings)* For any $T\geq 2$ and $k\in\cal{K}$, any information arrival matrix $\bm{\mathrm{H}}$, any information-agnostic-optimal policy $\pi\in\cal{P}^{\mathrm{IAO}}$, any problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$, one has for any $\epsilon_{k}>0$:*

[TABLE]

Lemma 2 implies that in the absence of precise knowledge on mappings it is not always possible to achieve performance improvement. For example, consider arm $k$ for which there exists some problem instance $(\tilde{\bm{\nu}},\tilde{\bm{\nu}}^{\mathrm{aux}})\in{\cal S}$ such that $\mathbb{E}_{X\sim\tilde{\nu}_{k}}[X]>\mu^{\ast}$ and $\mathrm{KL}(\nu_{k}^{\mathrm{aux}},\tilde{\nu}_{k}^{\mathrm{aux}})=0$. Then, the regret would be at least logarithmic in $T$, a regret rate one would incur without accounting for auxiliary information.

5.3 Adapting to Unknown Mappings Based on Upper Confidence Bounds

We next develop the 2-UCBs policy that adapts to a priori unknown linear mappings and information arrival processes, and then establish its near optimality. (Relevant notation is provided in Table 2.)

2-UCBs Policy. The 2-UCBs policy deploys the following two upper confidence bounds:

[TABLE]

and tightens the final UCB by using the minimum of the two,

[TABLE]

The first upper confidence bound is typical in the MAB literature (see, e.g., Auer et al. 2002) and is based on reward observations. The second one incorporates both reward observations and auxiliary observations. Note that $\bar{X}^{\pi,\mathrm{aux}}_{k,t}$ is an optimistic estimator for the mean reward of arm $k$ since it linearly combines the estimators $\bar{X}^{\pi}_{k,t}$ and $\bar{\alpha}\bar{Y}_{k,t}$, where the latter over-estimates $\mu_{k}$ whenever $\alpha_{k}<\bar{\alpha}$. Therefore $\bar{X}^{\pi,\mathrm{aux}}_{k,t}$ is a biased estimate, while $\bar{X}^{\pi}_{k,t}$ is unbiased. On the other hand, $\bar{X}^{\pi,\mathrm{aux}}_{k,t}$ incorporates a larger number of samples compared to $\bar{X}^{\pi}_{k,t}$, and hence has a smaller stochastic error.

With only few observations, the stochastic error is a key driver of experimentation with sub-optimal actions. On the other hand, when there is abundant auxiliary data on a suboptimal arm $k$ for which $\alpha_{k}y_{k}\geq\mu^{\ast}$, then the upward bias of the optimistic empirical mean $\bar{X}^{\pi,\mathrm{aux}}_{k,t}$ can be substantial. Selecting the minimum of the two UCB’s efficiently balances these opposing risks.

Theorem 6

(Near optimality of 2-UCBs)* Assume that $\Phi=\Phi_{L}$ with $0\leq\alpha_{k}\leq\bar{\alpha}$ for some known $\bar{\alpha}$, and let $\pi$ be the 2-UCBs policy tuned by $c>2$. Then, for any $\epsilon>\frac{1}{2}$, and every $T\geq 1$, auxiliary information arrival matrix $\bm{\mathrm{H}}$, and problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$, and without prior knowledge of the mappings, one has for $\delta_{k}=\mu^{\ast}-\bar{\alpha}\cdot y_{k}$:*

If $\bar{\alpha}\cdot y_{k}\geq\mu^{\ast}$, then $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t}}[n_{k,T+1}^{\pi}]\leq\frac{\ltx@label{ucb1-unknown}C_{12}}{\Delta_{k}^{2}}\log T;\vspace{-0.05cm}$ 2. 2.

If $\mu^{\ast}>\bar{\alpha}\cdot y_{k}>\mu^{\ast}-\epsilon\Delta_{k}$, then $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t}}[n_{k,T+1}^{\pi}]\leq\min\left\{\frac{\ref*{ucb1-unknown}}{\Delta_{k}^{2}}\log T,\;\frac{\ref*{ucb1-unknown}}{\delta_{k}^{2}}\log\left(\sum\limits_{t=0}^{T}\exp\left(-\ref*{ucb2-unknown}\delta_{k}^{2}\sum\limits_{s=1}^{t}h_{k,s}\right)\right)+\ltx@label{ucb3-unknown}C_{13}\right\};\vspace{-0.1cm}$ 3. 3.

If $\bar{\alpha}\cdot y_{k}\leq\mu^{\ast}-\epsilon\Delta_{k}$, then $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t}}[n_{k,T+1}^{\pi}]\leq\frac{\ref*{ucb1-unknown}}{\Delta_{k}^{2}}\log\left(\sum\limits_{t=0}^{T}\exp\left(-\ltx@label{ucb2-unknown}C_{14}\delta_{k}^{2}\sum\limits_{s=1}^{t}h_{k,s}\right)\right)+\ref*{ucb3-unknown};$

where $\ref*{ucb1-unknown},\ref*{ucb2-unknown}$, and LABEL:ucb3-unknown are positive constants that only depend on $\sigma$, $\hat{\sigma}$, $\bar{\alpha}$, $\epsilon$, and $c$.

Theorem 6 implies that when both the information arrival matrix $\mathbf{H}$ and the mappings $\{\phi_{k}\}$ are a priori unknown, 2-UCBs guarantees rate-optimality with respect to arms for which either $\bar{\alpha}\cdot y_{k}>\mu^{\ast}$ or $\bar{\alpha}\cdot y_{k}\leq\mu^{\ast}-\epsilon\Delta_{k}$, uniformly over $(i)$ the class of arbitrary arrival processes, and $(ii)$ the class $\Phi_{L}$ of linear mappings. In particular, when $\bar{\alpha}\cdot y_{k}\geq\mu^{\ast}$ for some suboptimal arm $k$, then 2-UCBs guarantees the regret that is achievable without auxiliary information, and when $\bar{\alpha}\cdot y_{k}\leq\mu^{\ast}-\epsilon\Delta_{k}$, it guarantees regret that is a decreasing log-sum-exp function of the auxiliary information arrivals.

Recalling Figure 3, one may observe that the case $\mu^{\ast}>\bar{\alpha}\cdot y_{k}>\mu^{\ast}-\epsilon\Delta_{k}$ corresponds to the part of Regime 1 that is alongside Regime 2. In that case, 2-UCBs still guarantees a regret rate that is a decreasing log-sum-exp function of auxiliary information arrivals, but with a multiplicative factor that is proportional to $\delta_{k}^{-2}$ as opposed to $\Delta_{k}^{-2}$ (but never worse than the rate that is achievable without auxiliary information).

6 Simulation Using Content Recommendations Data

To demonstrate the value that may be captured by leveraging auxiliary data, we use data from a large US media site to analyze performance when recommending articles that have unknown impact on the future browsing path of readers. For complete description of the setup and analysis see Appendix F.

Background. Content recommendations, which point readers to content they “may like,” are a form of dynamic service provided by media sites and third-party providers, with the objective of increasing readership and revenue streams for media sites. In that context, there are two key performance indicators that are fundamental for recommendations that guarantee good performance along the reading path of readers: $(i)$ the likelihood of a reader to click on a recommendation, and $(ii)$ when clicking on it, the likelihood of the reader to continue and consume additional content afterwards (see discussion and analysis in Besbes et al. 2016). The former indicator is the click-through rate (CTR) of the recommendation, and the latter indicator can be viewed as the conversion rate (CVR) of the recommendation.

The likelihood of readers to continue consuming content after reading an article is affected by design features of that article (e.g., its organization or the number of photos it features) that are not indicated by the recommendation linked to that article (and therefore typically do not affect CTR). These features are subject to editorial adjustments that may take place after CTR estimates are already formed (say, a few hours or days after the article’s release), with the objective of increasing readership.

Rather than which article to recommend, our setup describes sequential experiments designed for evaluating which version of a given article to recommend, and how changes in the design of the article impact the likelihood of readers to continue consuming content after reading it. Based on our data, we simulate the extent to which the performance of such experiments could be improved when utilizing auxiliary observations, available in the form of the browsing path of readers that arrived to these articles directly from external search (and not by clicking a recommendation). Our data consists of a list of articles, together with: $(i)$ times at which these articles were recommended and the in-site browsing path that followed these recommendations; and $(ii)$ times at which readers arrived to these articles directly from a search engine (such as Google) and the browsing path that followed these visits.777The media site at hand follows a non-subscription model, and hence common user-idiosyncratic contexts (such as age, or preferences over content) are not available to the recommender system in real time.

Setup. For each article in our data we considered a one-armed bandit setting with a known outside option to simulate experimentation with a new version of that article. Each bandit experiment was based on one day of data. We constructed a decision horizon $t=1,2,\ldots,2000$ based on the first 2,000 time-stamps of that day at which the article was recommended from the highest position (out of 5 links that are presented in each page). For each article-day pair, we assume that the CTR is identical across both versions of the article and denote it by $\mathrm{CTR}_{a,d}$; for simplicity we assume that it is known. We calculated from the data the fraction of occasions where readers continued to read another article after arriving to article $a$ by clicking on a content recommendation, and denote it by $\mathrm{CVR}^{recom}_{a,d}$. We denote by $\mathrm{CVR}^{\text{recom}}_{a,d,0}$ the known conversion rate of article $a$ at its current design, and by $\mathrm{CVR}^{\text{recom}}_{a,d,1}$ a new (unknown) conversion rate that corresponds to a new structure that is under consideration. We assume that $\mathrm{CVR}^{recom}_{a,d,1}=\mathrm{CVR}^{recom}_{a,d}$ and that $\mathrm{CVR}^{recom}_{a,d,0}=\mathrm{CVR}^{recom}_{a,d}+s\Delta_{a,d}$, where at each replication we set $s\in\{-1,1\}$ with equal probabilities, and where $\Delta_{a,d}$ was selected in the range $[0.01,0.04]$.

Given an article-day pair, we define the independent random variables $W_{t}\sim\mathrm{Ber}(\mathrm{CTR}_{a,d})$ and $X_{k,t}\sim\leavevmode\nobreak\ \mathrm{Ber}(\mathrm{CVR}^{recom}_{a,d,k})$, for $k\in\{0,1\}$. The average regret incurred by policy $\pi\in\cal{P}$ is denoted by $\mathcal{R}^{\pi}_{a,d}$, and calculated by averaging over replications of the expression888As click-through rates are identical across article versions, this performance measure coincides with maximizing the (revenue-normalized) one-step lookahead objective, which was shown by Besbes et al. (2016) to approximate the recommendation value in the context of content recommendations in media sites.

[TABLE]

We assume that $X_{\pi_{t},t}$ is observed only when the recommendation is clicked, that is, when $W_{t}=1$.

Auxiliary Information. For each article-day pair, the matrix $\mathbf{H}$ and the sample path of auxiliary observations were fixed and determined as follows. We extracted the trajectory of information arrivals $\left\{h_{1,t}\right\}$ from the number of readers that arrived to the article from a search engine between consecutive decision epochs. For each epoch $t$ and arrival-from-search $m\in\left\{1,\ldots,h_{1,t}\right\}$ we denote by $Y_{1,t,m}\in\left\{0,1\right\}$ an indicator of whether the reader continued to read additional content in the media site after visiting the article. We denote by $\mathrm{CVR}^{\text{search}}_{a,d}$ the fraction of readers that continued to another article after arriving to article $a$ from search during day $d$. We let

[TABLE]

denote the fraction of conversion rates for users that arrive to article $a$ by clicking a recommendation, and for users that arrive to it from search.999Conversion rates of readers that clicked on a recommendation are typically higher than conversion rates of readers that arrived from search (values of $\alpha_{a,d}$ are in the range $[1,16]$). See related discussions in Besbes et al. (2016) on experienced versus inexperienced readers, and in Caro and Martínez de Albéniz (2020) on followers versus new readers. Given $\alpha_{a,d}$, auxiliary observations $\left\{Y_{1,t,m}\right\}$ can be mapped to reward observations using a mapping $\phi$ that belongs to the class of linear mappings $\Phi_{L}$ from Example 1.

We compare the performance of UCB1 without auxiliary data, with the one achieved by utilizing auxiliary data based on estimated mappings (aUCB1, see §4.1), and the adaptive 2-UCBs policy (see §5.3); a description of how these policies are updated under the information structure at hand is provided in Appendix F.1. It is important to clarify that the purpose of our analysis is not to study effective methods of estimating the mapping $\phi$ from data, but rather, to ($i$) demonstrate that mappings could be indeed estimated from historical data in a simple manner, and ($ii$) investigate the performance sensitivity of aUCB1 to mapping misspecification and, correspondingly, the potential benefit of deploying an adaptive policy such as 2-UCBs. With that in mind, as an estimator of $\phi$, we simply used $\hat{\alpha}_{a,d}=\alpha_{a,d-1}$, the fraction of the two conversion rates from the previous day, and deployed that estimate in the aUCB1 policy. We used $\bar{\alpha}_{a,d}=1.1\hat{\alpha}_{a,d}$ as the upper bound that is used in the 2-UCBs policy.

Evaluating Performance. For each article-day pair we denote the Relative Improvement (RI) achieved by policy $\pi$ relative to a UCB1 policy (that does not account for auxiliary information) by

[TABLE]

Since each article-day pair is associated with a different auxiliary information trajectory and estimation of $\alpha_{a,d}$, we present the Relative Improvement of each policy along two key dimensions. The first one is the Auxiliary Information Effectiveness ($\mathrm{AIE}$) index, defined as follows:

[TABLE]

where $c=\tilde{c}\cdot(\Delta_{a,d}/\hat{\sigma}\alpha_{a,d})^{2}$, and where $\hat{\sigma}$ is set globally to $1/4$ for Bernoulli observations, $\Delta_{a,d}$ and $\alpha_{a,d}$ were set as described above, and $\tilde{c}$ is a scaling parameter.101010For consistency with our analytical bounds we set $\tilde{c}$ to $4\times(\text{UCB tuning parameter})$, which, in Figure 4, equals $0.2$.

The $\mathrm{AIE}$ index captures the effectiveness of the information arrival trajectory, which is impacted by frequency and timing of auxiliary information arrivals (as captured by the matrix $\bm{\mathrm{H}}$), and other parameters that determine the informativeness of each auxiliary observation (recall that in order to focus on the impact of auxiliary information arrivals we set $T=2,000$ in all the experiments). Note that the $\mathrm{AIE}$ index equals [math] in the absence of auxiliary information and is positive otherwise. When auxiliary observations arrive more frequently and earlier this index increases according to the log-sum-exp rate that determines the minimax complexity of the problem (see Remark 1). The second dimension we consider is the Relative Mapping Misspecification (RMM), defined as follows:

[TABLE]

The RMM captures the bias in auxiliary observations that is caused by misspecification of the mappings. It equals [math] when the true mappings are accurately specified, and is increasing with the mapping misspecification and the true conversion rate.

Results and Discussion. The Relative Improvements as a function of the Auxiliary Information Effectiveness and the Relative Mapping Misspecification appear in Figure 4 (For additional summary statistics see Appendix F). We note that for a given AIE, which is determined by the trajectory of auxiliary information arrivals, performance is sensitive to the particular order of auxiliary observations along that trajectory.111111We further note that each of the scatter plots in Figure 4 depicts a projection of performances onto a single dimension: AIE or RMM; a 3D plot of RI as a function of both AIE and RMM can be viewed in https://ygur.people.stanford.edu/adaptive-sequential-experiments-plot.

Figure 4 reveals few important occurrences. Relative to ignoring auxiliary data, aUCB1 and 2-UCBs achieved averaged performance improvement of $7.1\%$ and $8.8\%$, respectively. With little to moderate AIE, aUCB1 achieved notable RI, but as AIE grew large the variance of RI values increased and average RI decreased. This trend demonstrates that little auxiliary data can be leveraged to significantly to improve performance even when mappings are estimated from historical data using simple methods, but also that abundant auxiliary data could lead to performance deterioration whenever auxiliary data is misinterpreted (mappings are misspecified). On the other hand, 2-UCBs adapts to unknown mappings, and its performance improves quite consistently with the effectiveness of auxiliary data (with $\text{AIE}\geq 1.5$, 2-UCBs outperformed aUCB1 and achieved average RI of $19.2\%$).

Reviewing the RI as a function of RMM reveals that the RI values of aUCB1 consistently decrease with the mapping misspecification. When the misspecification is small aUCB1 achieves significant improvement ($18\%$ when $\text{MMR}\leq 0.025$), but when the mapping misspecification grows large the performance of aUCB1 deteriorates and is eventually dominated by ignoring auxiliary data. The RI achieved by 2-UCBs also decreases with the MMR (as estimates of mappings are used to construct the upper bound $\bar{\alpha}$) but more moderately, and on average, 2-UCBs outperformed aUCB1 when $\text{MMR}\geq 0.025$.

7 Concluding Remarks

Summary and Implications. We considered an extension of the MAB framework by allowing for arbitrary and unknown auxiliary information arrival processes that are relevant to many application domains, including product recommendations, pricing, health care, and machine learning. We studied the impact of the information arrival process on policy design and the performance that can be achieved. Through matching lower and upper bounds we identified a spectrum of minimax (regret) complexities for this class of problems as a function of the information arrival process, which provides a sharp criterion for identifying rate-optimal policies.

When it is known how to map auxiliary observations to reward estimates, we established that Thompson sampling and UCB1 can be leveraged to uniformly guarantee rate-optimality and, in that sense, possess natural robustness to the information arrival process (robustness which is not possessed by policies with exogenous exploration rate). Moreover, the regret rate that is achieved by Thompson sampling and UCB1 cannot be improved even with prior knowledge of the information arrival process.

When it is a priori unknown how to map auxiliary observations to reward estimates, we provide necessary and sufficient conditions that delineate when auxiliary data can still lead to performance improvement, and devise a new (and simple) 2-UCBs policy that is near optimal in that setting.

Using content recommendations data, we demonstrated the value that can be captured in practice by leveraging auxiliary information. Our analysis implies that adaptivity with respect to the mappings that interpret auxiliary data is particularly crucial in settings characterized by $(i)$ significant uncertainty about these mappings; or $(ii)$ access to abundant auxiliary data.

7.1 Avenues for Future Research

Endogenous Information Arrival Processes. One may consider information arrival processes that may depend on past decisions and observations. A class of endogenous arrival processes may include various instances, such as information arrivals that are decoupled across arms (selecting an action can impact only future information on that action), and interconnected information arrivals where each action may impact future information arrivals on all actions. An equilibrium analysis with endogenous information arrival processes is an interesting and challenging research avenue.

Adapting to Unknown Mappings Using Posterior Sampling. In addition to the approach taken at §5, which is based on upper confidence bounds, performance guarantees with unknown mappings could potentially be established also based on posterior sampling. For example, one may consider the following two-dimensional Thompson sampling policy that is based on joint posterior updates of mean rewards and the means of auxiliary observations, $(\mu_{k},y_{k})$. For each pair $(\mu,y)$, define the weight functions

[TABLE]

Denote by $P_{k,t}$ the posterior of action $k$ at time $t$. Then, its density is defined as

[TABLE]

At each period, after observing the last reward and any realized auxiliary observations, one would sample $(\mu^{(\theta)}_{k,t},y^{(\theta)}_{k,t})\sim P_{k,t}$ for all $k\in\cal K$, and select the action that maximizes $\mu^{(\theta)}_{k,t}$. While numerical analysis suggests that performance guarantee that is similar to the one established for 2-UCBs could be established for the above policy, we note that the analysis of this policy is challenging, and requires establishing concentration inequalities across two dimensions under various conditions.

Extending Upper Bounds with Unknown Mappings. Additional interesting paths are to extend analysis with unknown mappings to guarantee rate optimality with respect to all problem parameters (beyond $\bm{\mathrm{H}}$ and $T$ that are at the focus of the current work) for the case $\mu^{\ast}>\bar{\alpha}\cdot y_{k}>\mu^{\ast}-\epsilon\Delta_{k}$; to design a policy that infers the upper bound $\bar{\alpha}$ throughout the decision horizon and adapts to it without prior knowledge; and to consider other classes of mappings (beyond the linear class on which we focused here).

Appendix A Proofs of Main Results

A.1 Notations

For any policy $\pi$ and profiles $\bm{\nu}$ and $\bm{\nu}^{\prime}$, let $\mathbb{P}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$, $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$, and $\mathbb{R}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$ denote the probability, expectation, and regret when rewards are distributed according to $\bm{\nu}$, and auxiliary observations are distributed according to $\bm{\nu}^{\mathrm{aux}}$. Notation of counters and empirical means is provided in the following table:

A.2 Proof of Theorem 1

Step 1 (Preliminaries).

The proof adapts ideas of identifying worst-case nature strategy (see, e.g., Bubeck et al. 2013) to our setting in order to identify the precise change in the achievable performance as a function of the entries of information arrival matrix $\bm{\mathrm{H}}$. For $m,q\in\{1,\dots,K\}$ define the distribution profiles $\bm{\nu}^{(m,q)}$:

[TABLE]

For example, for $m=1$, one has

[TABLE]

Similarly, assume that the auxiliary information $Y_{k,t,m}$ is distributed according to the reward distribution $\hat{\nu}_{k}^{(m,q)}\overset{d}{=}\nu_{k}^{(m,q)}$, and hence we use the notation $\mathbb{P}^{\pi}_{\bm{\nu}}$, $\mathbb{E}^{\pi}_{\bm{\nu}}$, and $\mathbb{R}^{\pi}_{\bm{\nu}}$ instead of $\mathbb{P}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$, $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$, and $\mathbb{R}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}$.

Step 2 (Lower bound decomposition).

We note that

[TABLE]

Step 3 (A naive lower bound for $\max\limits_{q\in\{1,\dots,K\}}\{\mathcal{R}^{\pi}_{\bm{\nu}^{(m,q)}}(\bm{\mathrm{H}},T)\}$).

We note that

[TABLE]

Step 4 (An information theoretic lower bound).

For any profile $\bm{\nu}$, denote by $\bm{\nu}_{t}$ the distribution of the observed rewards up to time $t$ under $\bm{\nu}$. By Lemma 8, for any $q\neq m$, one has

[TABLE]

One obtains:

[TABLE]

where (a) follows from Lemma 7, (b) holds by (5), and (c) follows from $n^{\pi}_{q,T+1}\geq n^{\pi}_{q,t}$ for $t\in\mathcal{T}$.

Step 5 (Unifying the lower bounds in steps 3 and 4).

Using (4), and (A.2), we establish

[TABLE]

where (a) follows from $x+\gamma e^{-\kappa x}\geq\frac{\log\gamma\kappa}{\kappa}$ for $\gamma,\kappa,x>0$. (Note that the function $x+\gamma e^{-\kappa x}$ is a convex function and we can find its minimum by finding the root of its derivative) The result is then established by putting together (3), and (A.2).

A.3 Proof of Theorem 2

Fix a problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$ with the mean rewards $\bm{\mu}$ and the mean $\bm{\mathrm{y}}$ for auxiliary observations. Consider a suboptimal arm $k\neq k^{\ast}$. If arm $k$ is pulled at time $t$, then

[TABLE]

Therefore, at least one of the following three events must occur:

[TABLE]

To see why this is true, assume that all the above events fail. Then, we have

[TABLE]

which is in contradiction with the assumption that arm $k$ is pulled at time $t$. For any sequence $\{l_{k,t}\}_{t\in\mathcal{T}}$, and $\{\hat{l}_{k,t}\}_{t\in\mathcal{T}}$, such that $\hat{l}_{k,t}\geq l_{k,t}$ for all $t\in\mathcal{{T}}$, one has

[TABLE]

Set the values of $l_{k,t}$ and $\hat{l}_{k,t}$ as follows:

[TABLE]

where

[TABLE]

To have n^{\text{\tiny\mathrm{known}}}_{k,t}>l_{k,t}, it must be the case that $E_{3,t}$ does not occur. Therefore,

[TABLE]

where the last inequality follows from Lemma 9 and the union bound. Plugging the value of $\hat{l}_{k,t}$ into (A.3), one obtains:

[TABLE]

which concludes the proof.

A.4 Proof of Theorem 3

We adapt the proof of the upper bound for Thompson sampling with Gaussian priors in Agrawal and Goyal (2013a).

Step1 (Notations and definitions).

Fix a problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$ with the mean rewards $\bm{\mu}$ and the mean $\bm{\mathrm{y}}$ for auxiliary observations. For every suboptimal arm $k$, we consider three parameters $x_{k}$, $y_{k}$, and $u_{k}$ such that $\mu_{k}<x_{k}<y_{k}<u_{k}<\mu_{k^{\ast}}$. We will specify these three parameters at the end of the proof. Also, define the events $E_{k,t}^{\mu}$, and $E^{\theta}_{k,t}$ to be the events on which \bar{X}_{k,n^{\text{\tiny\mathrm{known}}}_{k,t}}\leq x_{k}, and $\theta_{k,t}\leq y_{k}$, respectively. In words, the events $E_{k,t}^{\mu}$, and $E^{\theta}_{k,t}$ happen when the estimated mean reward, and the sample mean reward do not deviate from the true mean, respectively. Define the history $\mathcal{H}_{t}\coloneqq\left(\{X_{\pi_{s},s}\}_{s=1}^{t-1},\{\pi_{s}\}_{s=1}^{t-1},\{\bm{\mathrm{Z}}_{s}\}_{s=1}^{t},\{\bm{h}_{s}\}_{s=1}^{t}\right)$ for all $t=1,\dots$. Finally define

[TABLE]

Step2 (Preliminaries).

We will make use of the following lemmas from Agrawal and Goyal (2013a) throughout this proof. The proof of the lemmas are skipped since they are simple adaptation to our setting with auxiliary information arrival.

Lemma 3

For any suboptimal arm $k$,

[TABLE]

where $t_{0}=0$ and for $j>0$, $t_{j}$ is the time step at which the optimal arm $k^{\ast}$ is pulled for the $j$th time.

Proof.    The proof can be found in the analysis of Theorem 1 in Agrawal and Goyal (2013a). However, we bring the proof for completeness.

[TABLE]

where (a) follows from Lemma 1 in Agrawal and Goyal (2013a).

Lemma 4** (Agrawal and

Goyal 2013a, Lemma 2)**

For any suboptimal arm $k$,

[TABLE]

Step 3 (Regret decomposition).

Fix a profile $\bm{\nu}$. For each suboptimal arm $k$, we will decompose the number of times it is pulled by the policy as follows and upper bound each term separately:

[TABLE]

Step 4 (Analysis of $J_{k,1}$).

Given Lemma 3, $J_{k,1}$ can be upper bounded by analyzing $\frac{1-p_{k,t_{j}+1}}{p_{k,t_{j}+1}}$:

[TABLE]

First, we analyze $Q_{1}$:

[TABLE]

where the last inequality follows from Chernoff-Hoeffding bound in Lemma 9. The term $Q_{2}$ can also be bounded from below using Chernoff-Hoeffding in Lemma 9 bound as follows:

[TABLE]

Define $\delta_{k}\coloneqq\min\{(\mu_{1}-u_{k}),(u_{k}-y_{k})\}$. The last three displays along with Lemma 4 yield

[TABLE]

Step 5 (Analysis of $J_{k,2}$).

First, we upper bound the following conditional probability

[TABLE]

where (a) follows from Chernoff-Hoeffding’s bound in Lemma 9. Define

[TABLE]

By (A.4), and the definition above, one obtains:

[TABLE]

for every $t\in\cal T$. Note that by definition,

[TABLE]

By the above two displays, we can conclude that

[TABLE]

The following lemma will be deployed to analyze $n_{k,T+1}^{\theta}$:

Now, we can apply Lemma 1 to the sequence $\left\{n_{k,t}^{\theta}\right\}_{0\leq t\leq T}$, and obtain

[TABLE]

By this inequality and the definition of $n_{k,t}^{\theta}$, we have

[TABLE]

Step 6 (Analysis of $J_{k,3}$).

The term $J_{k,3}$ can be upper bounded using Lemma 4.

Step 7 (Determining the constants)

Finally, let $x_{k}=\mu_{k}+\frac{\Delta_{k}}{4}$, $y_{k}=\mu_{k}+\frac{2\Delta_{k}}{4}$, and $u_{k}=\mu_{k}+\frac{3\Delta_{k}}{4}$. Then, by putting (9), (A.4), (12), and Lemma 4 back together, the result is established.

A.5 Proof of Theorem 4

The proof of the upper bound is a simple extension of the proof of Theorem 2. In fact, for the suboptimal arms $k\in\mathcal{K}\setminus\{k^{\ast}\}$ for which $\bar{\alpha}y_{k}\geq\mu^{\ast}$, one may repeat the same line of proof replacing . For the suboptimal arms $k\in\mathcal{K}\setminus\{k^{\ast}\}$ for which $\bar{\alpha}y_{k}<\mu^{\ast}$, one may repeat the same line of proof replacing \bar{X}_{k,t}^{\text{\tiny\mathrm{known}}}+\sqrt{\frac{c\sigma^{2}\log t}{n^{\text{\tiny\mathrm{known}}}_{k,t}}} with $\bar{\alpha}\cdot y_{k}$.

The proof of optimality among IAO policies, is an application of Theorem 5. When $\hat{\sigma}=0$, Theorem 5 reads as follows with $\delta_{k}=\mu^{\ast}-\bar{\alpha}\cdot y_{k}$:

• •

If $\bar{\alpha}\cdot y_{k}>\mu^{\ast}$ then, $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t}}[n_{k,T+1}^{\pi}]\geq\ref*{lower-bound-unknown1}\Delta_{k}^{-2}\log\left(\frac{\ref*{lower-bound-unknown2}\min\{4\Delta_{k}^{4},(\Delta_{k}-\delta_{k})^{2}\delta_{k}^{2}\}}{K\log T}T\right);$

• •

If $\bar{\alpha}\cdot y_{k}<\mu^{\ast}$ then, $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t}}[n_{k,T+1}^{\pi}]\geq 0,$

where $\ltx@label{lower-bound-unknown1}C_{15}$ and $\ltx@label{lower-bound-unknown2}C_{16}$ are positive constants that only depend on $\sigma$, and $\bar{\alpha}$.

A.6 Proof of Theorem 5

In order to prove the theorem, we need to repeat the following argument for each arm $k$:

Consider a new problem instance $(\tilde{\bm{\nu}},\tilde{\bm{\nu}}^{\mathrm{aux}})\in{\cal S}$ in Lemma 2, for which arm $k$ is the optimal arm. The only difference between $(\tilde{\bm{\nu}},\tilde{\bm{\nu}}^{\mathrm{aux}})$ and $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})$ is the distributions of rewards and auxiliary observations of arm $k$:

• •

If $\bar{\alpha}\cdot y_{k}<\mu^{\ast}$ then, $(\tilde{\nu}_{k},\tilde{\nu}_{k}^{\mathrm{aux}})=(\mathcal{N}(\mu^{\ast}+\epsilon_{k},\sigma^{2}),\mathcal{N}(\frac{\mu^{\ast}+\epsilon_{k}}{\bar{\alpha}},\hat{\sigma}^{2}))$ with $\epsilon_{k}=\delta_{k}=\mu^{\ast}-\bar{\alpha}\cdot y_{k}$;

• •

If $\mu^{\ast}+\Delta_{k}\geq\bar{\alpha}\cdot y_{k}>\mu^{\ast}$ then, $(\tilde{\nu}_{k},\tilde{\nu}_{k}^{\mathrm{aux}})=(\mathcal{N}(\mu^{\ast}+\epsilon_{k},\sigma^{2}),\mathcal{N}(y_{k},\hat{\sigma}^{2}))$ with $\epsilon_{k}=-\delta_{k}=\bar{\alpha}\cdot y_{k}-\mu^{\ast}$;

• •

If $\bar{\alpha}\cdot y_{k}>\mu^{\ast}+\Delta_{k}$ then, $(\tilde{\nu}_{k},\tilde{\nu}_{k}^{\mathrm{aux}})=(\mathcal{N}(\mu^{\ast}+\epsilon_{k},\sigma^{2}),\mathcal{N}(y_{k},\hat{\sigma}^{2}))$ with $\epsilon_{k}=\Delta_{k}$.

This concludes the proof.

A.7 Proof of Lemma 2

Step 1 (Preliminaries).

The proof adapts ideas of identifying worst-case nature strategy to our setting in order to identify the precise change in the achievable performance as a function of the entries of information arrival matrix $\bm{\mathrm{H}}$. Fix a problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$ with the mean rewards $\bm{\mu}$ and the mean $\bm{\mathrm{y}}$ for auxiliary observations, and recall that $\Delta_{k}=\mu^{\ast}-\mu_{k}$ and $\delta_{k}=\mu^{\ast}-\bar{\alpha}\cdot y_{k}$. Consider a suboptimal arm $k$. We consider a new problem instance $(\tilde{\bm{\nu}},\tilde{\bm{\nu}}^{\mathrm{aux}})\in{\cal S}$ for which arm $k$ is the optimal arm such that $\mathbb{E}_{X\sim\tilde{\nu}_{k}}[X]=\mu^{\ast}+\epsilon_{k}$.

Step 2 (Distance between problem instances).

This step is based on the following lemma that states how informative the history can be about the true underlying problem instance.

Lemma 5

Let $\mathcal{H}_{t}\coloneqq(\bm{h}_{1},\bm{\mathrm{Z}}_{1},\pi_{1},X_{\pi_{1},1},\dots,\bm{h}_{t-1},\bm{\mathrm{Z}}_{t-1},\pi_{t-1},X_{\pi_{t-1},t-1},\bm{h}_{t},\bm{\mathrm{Z}}_{t})$ be the history up to time step $t$. For any problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})$, denote by $({\bbnu},\bbnu^{\mathrm{aux}})_{t}$ the law of $\mathcal{H}_{t}$ under the problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})$. Then, for any problem instances $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})$ and $(\tilde{\bm{\nu}},\tilde{\bm{\nu}}^{\mathrm{aux}})$ and any time step $t\in\mathcal{T}$,

[TABLE]

Proof. This type of statement is well-known in the MAB literature (see, e.g., Auer et al. (2002), Garivier et al. (2019), or Gerchinovitz and Lattimore (2016)). However, we show how it is derived for the sake of completeness. For any random variables $X$ and $Y$, we denote by $(\bbnu,\bbnu^{\mathrm{aux}})^{(X)}$ the law of $X$, and by $(\bbnu,\bbnu^{\mathrm{aux}})^{(X|Y)}$ the law of $X$ conditional on $Y$ under the problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})$. By the chain rule of KL-divergence, one has

[TABLE]

Applying the chain rule to the second quantity on the right hand side of the above equality, one obtains

[TABLE]

By reiterating the above two displays, the desired result is derived. This concludes the proof.

The above lemma yields

[TABLE]

Step 3 (A constraint for $\mathbb{E}^{\pi}_{(\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t})}[n_{k,T+1}^{\pi}]$).

Note that by the assumption that $\pi$ is IAO, one has

[TABLE]

where (a) follows from Lemma 7 and (b) holds by (13). The above inequality provides a constraint over the quantity of interest, $\mathbb{E}^{\pi}_{(\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t})}[n_{k,T+1}^{\pi}]$.

Step 4 (Lower bounding $\mathbb{E}^{\pi}_{(\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t})}[n_{k,T+1}^{\pi}]$).

Given the above inequality one can see that $\mathbb{E}^{\pi}_{(\bm{\nu},\bm{\nu}^{\mathrm{aux}}_{t})}[n_{k,T+1}^{\pi}]$ is lower bounded by the solution of the following convex optimization problem:

[TABLE]

The solution of the above optimization problem is then, lower bounded by that of the following for any $\lambda\geq 0$:

[TABLE]

Letting $\lambda=\frac{\epsilon^{2}}{C_{\pi,{\cal S}}K\sigma^{2}\log T}$ and noting that $x+\gamma e^{-\kappa x}\geq\frac{\log\gamma\kappa}{\kappa}$ for $\gamma,\kappa,x>0$, one obtains

[TABLE]

This concludes the proof.

A.8 Proof of Theorem 6

Step 1 (Preliminaries).

Fix a profile $\left(\bm{\nu},\bm{\nu}^{\mathrm{aux}}\right)$. Note that it is unlikely for the UCB of the optimal arm to be less than the largest mean reward. We will carry out the analysis based on this observation. To be more precise, define

[TABLE]

to be the event on which the UCB of the optimal arm is less than the largest mean reward at time $t$. Note that by the Chernoff-Hoefding bound in Lemma 9, one has

[TABLE]

Hence, defining $\mathcal{E}^{\ast}\coloneqq\bigcup_{t=1}^{T}\mathcal{E}^{\ast}_{t}$, one obtains

[TABLE]

where we have used the assumption that $c>2$. Consider a suboptimal arm $k\neq k^{\ast}$. We will bound $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}\left[n^{\pi}_{k,T+1}\;\middle|\;\bar{\mathcal{E}}^{\ast}\right]$, the expected number of times the suboptimal arm $k$ is pulled conditional on the event $\bar{\mathcal{E}}^{\ast}$. Our analysis is based on the fact that if arm $k$ is pulled at time $t$, then

[TABLE]

We give two separate upper bounds for $\mathbb{E}^{\pi}_{\left(\bm{\nu},\bm{\nu}^{\mathrm{aux}}\right)}\left[n^{\pi}_{k,T+1}\;\middle|\;\bar{\mathcal{E}}^{\ast}\right]$. The first upper bound holds for any suboptimal arm $k$, and the second one is specific to the suboptimal arms $k\in\mathcal{K}^{\ast}$. The following lemma will be the main tool in our analysis.

Lemma 6

Let $\{f_{n}\}_{n=1}^{N}$ and $\{g_{n}\}_{n=1}^{N}$ be two sequences such that for all $n\in\{1,\dots,N\}$, $f_{n}=\sum_{j=1}^{n-1}x_{j}$ with $0\leq x_{j}\leq M$ for all $j\in\{1,\dots,N\}$ and some $M\geq 0$. Then,

[TABLE]

Proof.* If $\max\limits_{1\leq n\leq N}g_{n}+M<0$ then, the result is immediate. Assume $\max\limits_{1\leq n\leq N}g_{n}+M\geq 0$. We prove the result through contradiction. Suppose that $\sum_{n=1}^{N}x_{n}\cdot\mathbbm{1}\left\{f_{n}\leq g_{n}\right\}>\max\limits_{1\leq n\leq N}g_{n}+M$. Let*

[TABLE]

Note that one must have $f_{\tau}\leq g_{\tau}$. Since all $x_{j}$’s are non-negative, one has

[TABLE]

*which is a contradiction. This concludes the proof.   * **

Step 2 (Regret upper bound for any suboptimal arms $k\in\mathcal{K}\setminus\{k^{\ast}\}$).

In this step, we deploy the UCB constructed based on reward observations. That is, (17) yields that if arm $k$ is pulled at time $t$ then, conditional on the event $\bar{\mathcal{E}}^{\ast}$, one has

[TABLE]

Therefore, at least one of the following events must occur:

[TABLE]

To see why this is true, assume that all the above events fail. Then, we have

[TABLE]

which is in contradiction with the assumption that arm $k$ is pulled at time $t$ conditional on the event $\bar{\mathcal{E}}^{\ast}$. For any sequence $\{l_{k,t}\}_{t\in\mathcal{T}}$, one has

[TABLE]

where the last inequality follows from Lemma 6. Set the values of $l_{k,t}$ as follows:

[TABLE]

To have $n^{\pi}_{k,t}>l_{k,t}$, it must be the case that $\mathcal{E}^{\pi}_{2,t}$ does not occur. Therefore,

[TABLE]

where (a) follows from the Chernoff-Hoeffding bound in Lemma 9 and (b) follows from the assumption that $c>2$. The last two displays yield

[TABLE]

Step 3 (Regret upper bound for suboptimal arms $k\in\mathcal{K}\setminus\{k^{\ast}\}$ which satisfy $\delta_{k}\geq 0$)).

For this set of arms, we deploy the UCB that incorporate both reward observations and auxiliary observations. That is, (17) yields that if arm $k$ is pulled at time $t$ then, conditional on the event $\bar{\mathcal{E}}^{\ast}$, one has

[TABLE]

Therefore, at least one of the following events must occur:

[TABLE]

A similar analysis as in the proof of Theorem 2, except for replacing $\Delta_{k}$ with $\delta_{k}$ throughout the analysis, results in the following upper bound:

[TABLE]

Step 4 (Regret upper bound for suboptimal arms $k\in\mathcal{K}\setminus\{k^{\ast}\}$ which satisfy $\delta_{k}\geq\frac{\Delta_{k}}{2}$)).

For this set of arms, we deploy the UCB that incorporate both reward observations and auxiliary observations. That is, (17) yields that if arm $k$ is pulled at time $t$ then, conditional on the event $\bar{\mathcal{E}}^{\ast}$, one has

[TABLE]

Therefore, at least one of the following events must occur:

[TABLE]

where we define

[TABLE]

To see why this is true, assume that all the above events fail. Then, we have

[TABLE]

which is in contradiction with the assumption that arm $k$ is pulled at time $t$ conditional on the event $\bar{\mathcal{E}}^{\ast}$. For any sequence $\{l_{k,t}\}_{t\in\mathcal{T}}$, one has

[TABLE]

We will upper bound each term on the right hand side of the above inequality separately. Set the values of $l_{k,t}$ as $l_{k,t}=\frac{4c\sigma^{2}\log\left(t\right)}{(\Delta^{\pi,\mathrm{aux}}_{k,t})^{2}}.$ In order to upper bound the term $\mathbb{E}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}\left[\sum\limits_{t=1}^{T}\mathbbm{1}\left\{\pi_{t}=k,n^{\pi,\mathrm{aux}}_{k,t}\leq l_{k,t}\right\}\;\middle|\;\bar{\mathcal{E}}^{\ast}\right]$ in (A.8), note that the equality $\Delta^{\pi,\mathrm{aux}}_{k,t}=\frac{\Delta_{k}\cdot n^{\pi}_{k,t}+\frac{\delta_{k}\sigma^{2}}{\bar{\alpha}^{2}\hat{\sigma}^{2}}n^{\mathrm{aux}}_{k,t}}{n^{\pi}_{k,t}+\frac{\sigma^{2}}{\bar{\alpha}^{2}\hat{\sigma}^{2}}n^{\mathrm{aux}}_{k,t}}$ yields

[TABLE]

Since the left-hand side of the above inequality is quadratic in $n^{\pi}_{k,t}$, with a positive coefficient for the quadratic term then, the above inequality holds only if

[TABLE]

Now, in order to upper bound the right hand side of this inequality, note that if $\delta_{k}=\xi_{k}\Delta_{k}$, $\frac{1}{2}\leq\xi_{k}\leq 1$, one obtains

[TABLE]

That is, the last two displays yield that (A.8) holds only if

[TABLE]

where we define $\tau_{k,t}\coloneqq\sum\limits_{s=1}^{t}\exp\left(\frac{\left(\frac{2\xi_{k}-1}{\xi_{k}^{2}}\right)\delta_{k}^{2}}{4c\bar{\alpha}^{2}\hat{\sigma}^{2}}\sum\limits_{m=s}^{t-1}h_{k,m}\right)\geq t$, and the equality follows from $n^{\mathrm{aux}}_{k,t}=\sum\limits_{s=1}^{t}h_{k,s}$. One obtains

[TABLE]

where (a) follows from Lemma 6, and (b) follows from the fact that $\{\hat{l}_{k,t}\}_{t}$ is an increasing sequence.

In order to upper bound the term $\sum\limits_{t=1}^{T}\mathbb{P}^{\pi}_{\bm{\nu},\bm{\nu}^{\mathrm{aux}}}\left\{\pi_{t}=k,n^{\pi,\mathrm{aux}}_{k,t}>l_{k,t}\;\middle|\;\bar{\mathcal{E}}^{\ast}\right\}$ in (A.8), we note that to have $n^{\pi,\mathrm{aux}}_{k,t}>l_{k,t}$, it must be the case that $\bar{\mathcal{E}}^{\pi,\mathrm{aux}}_{2,t}$ does not occur. On the other hand, if arm $k$ is pulled and $\mathcal{E}^{\pi,\mathrm{aux}}_{2,t}$ does not occur, then it must be the case that $\mathcal{E}^{\pi,\mathrm{aux}}_{1,t}$ occurs. Therefore,

[TABLE]

Putting back together (A.8), (A.8), and (A.8), one obtains

[TABLE]

This concludes the proof.

Appendix B Auxiliary Lemmas

In this section of the appendix, we provide new lemmas that are used in the proofs of the main results of the paper, and some auxiliary results that appear in the literature.

Lemma 7

Let $\rho_{0},\rho_{1}$ be two probability distributions supported on some set $\mathbb{X}$, with $\rho_{0}$ absolutely continuous with respect to $\rho_{1}$. Then for any measurable function $\Psi:\mathbb{X}\rightarrow\{0,1\}$, one has:

[TABLE]

Proof.      Define $\mathcal{A}$ to be the event that $\Psi(X)=1$. One has

[TABLE]

where the last inequality follows from Tsybakov 2009, Lemma 2.6.

Lemma 8

Consider two profiles $\bm{\nu}$, and $\hat{\bm{\nu}}$. Denote by $\bm{\nu}_{t}$ ($\hat{\bm{\nu}}_{t}$) the distribution of the observed rewards up to time $t$ under $\bm{\nu}$ ($\hat{\bm{\nu}}$ respectively). Let ${n}_{k,t}$ be the number of times a sample from arm $k$ has been observed up to time $t$, that is n^{\text{\tiny\mathrm{known}}}_{k,t}=h_{k,t}+\sum\limits_{s=1}^{t-1}\left(h_{k,s}+\mathbbm{1}\{\pi_{s}=k\}\right). For any policy $\pi$, we have

[TABLE]

Proof.    The proof is a simple adaptation of the proof of Lemma 1 in Gerchinovitz and Lattimore (2016); hence, it is omitted.

Lemma 9** (Chernoff-Hoeffding bound)**

Let $X_{1},\dots,X_{n}$ be random variables such that $X_{t}$ is a $\sigma^{2}$-sub-Gaussian random variable conditioned on $X_{1},\dots,X_{t-1}$ and $\mathbb{E}\left[X_{t}\,|\,X_{1},\dots,X_{t-1}\right]=\mu$. Let $S_{n}=X_{1}+\dots+X_{n}$. Then for all $a\geq 0$

[TABLE]

Lemma 10** (Bernstein inequality)**

Let $X_{1},\dots,X_{n}$ be random variables with range $[0,1]$ and

[TABLE]

Let $S_{n}=X_{1}+\dots+X_{n}.$ Then for all $a\geq 0$

[TABLE]

Lemma 11** (Pinsker’s inequality)**

Let $\rho_{0},\rho_{1}$ be two probability distributions supported on some set $\mathbb{X}$, with $\rho_{0}$ absolutely continuous with respect to $\rho_{1}$ then,

[TABLE]

where $\left\|\rho_{0}-\rho_{1}\right\|_{1}=\sup_{A\subset\mathbb{X}}\left|\rho_{0}(A)-\rho_{1}(A)\right|$ is the variational distance between $\rho_{0}$ and $\rho_{1}$.

Lemma 12

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli random variable with parameters $p_{1},\dots,p_{n}$, respectively, then, for any $\kappa>0$:

[TABLE]

Proof.    Define event $\mathcal{E}$ to be the event that $\sum\limits_{j=1}^{n}X_{j}<\sum\limits_{j=1}^{n}p_{j}/2$. By Lemma 10, we have:

[TABLE]

By the law of total expectation:

[TABLE]

Lemma 13

For any $\kappa>0$ and integers $n_{1}$ and $n_{2}$, one has: $\sum\limits_{j=n_{1}}^{n_{2}}e^{-\kappa j}\leq\frac{1}{\kappa}(e^{-\kappa(n_{1}-1)}-e^{\kappa n_{2}})$.

Proof.      We can upper bound the summation using an integral as follows:

[TABLE]

Appendix C Additional Auxiliary Analysis

Corollary 1

(Near optimality under stationary information arrival process)* Let $\pi$ be Thompson sampling with auxiliary observations tuned by $c>0$. If $h_{k,t}$’s are i.i.d. Bernoulli random variables with parameter $\lambda$ then, for every $T\geq 1$:*

[TABLE]

for some constant $C$ independent of $T$.

Proof.

[TABLE]

where: (a) holds by Theorem 3, (b) follows from Jensen’s inequality and the concavity of $log(\cdot)$, and (c) holds by Lemmas 12 and 13. Noting that $\sum\limits_{t=0}^{T}\exp\left({\frac{-\Delta^{2}}{18c\hat{\sigma}^{2}}\sum\limits_{s=1}^{t}h_{k,s}}\right)<T+1$, the result is established.

Corollary 2

(Near optimality under diminishing information arrival process)* Let $\pi$ be Thompson sampling with auxiliary observations tuned by $c>0$. If $h_{k,t}$’s are random variables such that for some $\kappa\in\leavevmode\nobreak\ \mathbb{R}^{+}$, $\mathbb{E}\left[\sum_{s=1}^{t}h_{k,s}\right]=\lceil\frac{\hat{\sigma}^{2}\kappa}{2\Delta^{2}}\log t\rceil$ for each arm $k\in\mathcal{K}$ at each time step $t$, then for every $T\geq 1$:*

[TABLE]

for some constant $C$ independent of $T$.

Proof.

[TABLE]

where: (a) holds by Theorem 3, (b) follows from Jensen’s inequality and the concavity of $log(\cdot)$, and (c) holds by Lemma 12.

C.1 Proof for Subsection 3.1.1

We note that:

[TABLE]

where (a) holds by Theorem 1, (b) follows from the fact that log-sum-exp is a convex function (see (Boyd and Vandenberghe 2004, Example 3.1.5)), and (c) follows from $1-e^{-\frac{2\Delta^{2}}{\hat{\sigma}^{2}}\lambda}\leq 2\Delta^{2}\lambda/\hat{\sigma}^{2}$. Now we consider the following cases:

If $2\Delta^{2}T\lambda\leq\hat{\sigma}^{2}/2$ then, by inequality $1-e^{-x}\geq 2(1-e^{-1/2})x$ for $0\leq x\leq 1/2$, and (C.3), we have

[TABLE] 2. 2.

If $2\Delta^{2}T\lambda\geq\sigma^{2}/2$ then, by inequality $1-e^{-\frac{2\Delta^{2}}{\sigma^{2}}T\lambda}\geq 1-e^{-1/2}$, and (C.3), we have

[TABLE]

C.2 Analysis of a Myopic Policy under the Setting of Example 3.1.1.

Assume that $\pi$ is a myopic policy. Consider a suboptimal arm $k$. One has:

[TABLE]

We will upper bound \mathbb{P}\left\{\bar{X}_{k,t}^{\text{\tiny\mathrm{known}}}>\mu_{k}+\frac{\Delta_{k}}{2}\right\}. \mathbb{P}\left\{\bar{X}_{k^{\ast},t}^{\text{\tiny\mathrm{known}}}<\mu_{k^{\ast}}+\frac{\Delta_{k}}{2}\right\} can be upper bounded similarly.

[TABLE]

where (a) follows from Lemma 9, and Lemma 10. As a result the cumulative regret is upper bounded by

[TABLE]

Therefore, if $\Delta$ is a constant independent of $T$ then the regret is upper bounded by a constant.

C.3 Proof for Subsection 3.1.2.

We note that:

[TABLE]

where (a) holds by Theorem 1, (b) follows from the fact that log-sum-exp is a convex function (see (Boyd and Vandenberghe 2004, Example 3.1.5)).

C.4 Analysis of a Myopic Policy under the Setting of Example 3.1.2.

In this section we will shortly prove that if $\mathbb{E}\left[\sum\limits_{s=1}^{t}h_{k,s}\right]=\lceil\frac{\hat{\sigma}^{2}\kappa}{2\Delta^{2}}\log t\rceil$ for each arm $k$ and time step $t$ then a myopic policy achieves an asymptotic constant regret if $\Delta_{k}^{2}\kappa>16\hat{\sigma}^{2}$. For any profile $\bm{\nu}$, we have:

[TABLE]

Appendix D Endogenizing Exploration via Virtual Time Indices

D.1 Policy Design Intuition

In this subsection we first demonstrate, through the case of $\epsilon_{t}$-greedy policy, that policies with exogenous exploration rate may fail to achieve the lower bound in Theorem 1 in the presence of auxiliary information arrivals, and then develop intuition for how such policies may be adjusted to better leverage auxiliary information through the virtual time indices method. Formal policy design and analysis follow in §D.2.

The Inefficiency of “Naive” Adaptations of $\bm{\epsilon_{t}}$-greedy. Despite the robustness that was established in §4 for Thompson sampling and UCB, in general, accounting for auxiliary information while otherwise maintaining the policy structure may not suffice for achieving the lower bound established in Theorem 1. To demonstrate this, consider the $\epsilon_{t}$-greedy policy (Auer et al. 2002), which at each period $t$ selects an arm randomly with probability $\epsilon_{t}$ that is proportional to $t^{-1}$, and with probability $1-\epsilon_{t}$ selects the arm with the highest reward estimate. Without auxiliary information, $\epsilon_{t}$-greedy guarantees rate-optimal regret of order $\log T$, but this optimality does not carry over to settings with other information arrival processes: as was visualized in Figure 7, using auxiliary observations to update estimates without appropriately adjusting the exploration rate leads to sub-optimality.121212A natural alternative is to increase the time index by one (or any other constant) each time auxiliary information arrives. Such an approach, that could be implemented by arm-dependent time indices with update rule $\tau_{k,t}=\tau_{k,t-1}+1+h_{k,t}$, may lead to sub-optimality as well. For example, suppose $h_{k,1}=\left\lfloor\frac{C}{\Delta_{k}^{2}}\log T\right\rfloor$ for some constant $C>0$, and $h_{k,t}=0$ for all $t>1$. By Theorems 2 and 3, when the constant $C$ is sufficiently large constant regret is achievable. However, following the above update rule the regret incurred due to exploration is at least of order $\log(\frac{\Delta^{2}_{k}T}{\log T})$. For example, consider stationary information arrivals (described in §3.1.1), with arrival rate $\lambda\geq\frac{\hat{\sigma}^{2}}{4\Delta^{2}T}$. While constant regret is achievable in this case, without adjusting its exploration rate, $\epsilon_{t}$-greedy explores suboptimal arms at a rate that is independent of the number of auxiliary observations, and still incurs regret of order $\log T$.

Over-Exploration in the Presence of Additional Information. For simplicity, consider a 2-armed bandit problem with $\mu_{1}>\mu_{2}$. At each time $t$ the $\epsilon_{t}$-greedy policy explores over arm 2 independently with probability $\epsilon_{t}=ct^{-1}$ for some constant $c>0$. As a continuous-time proxi for the minimal number of times arm $2$ is selected by time $t$, consider the function

[TABLE]

The probability of best-arm misidentification at period $t$ can be bounded from above by:131313This upper bound on the probability of misidentifying the best arm can be obtained using standard concentration inequalities and is formalized, for example, in Step 5 of the proof of Theorem 7.

[TABLE]

for some constants $\bar{c}$, $\tilde{c}>0$. Thus, setting $\epsilon_{t}=ct^{-1}$ balances losses from exploration and exploitation.141414See related discussions on balancing losses from experimenting with suboptimal arms and from best arm misidentification, e.g., in Auer et al. (2002), Langford and Zhang (2008), Goldenshluger and Zeevi (2013), and Bastani and Bayati (2020).

Next, assume that just before time $t_{0}$, an additional independent reward observation of arm 2 is collected. Then, at time $t_{0}$, the minimal number of observations from arm 2 increases to ($1+c\log t_{0}$), and the upper bound on the probability of best-arm misidentification decreases by factor $e^{-\bar{c}}$:

[TABLE]

Therefore, when there are many auxiliary observations, the loss from best-arm misidentification is guaranteed to diminish, but performance might still be sub-optimal due to over-exploration.

Endogenizing the Exploration Rate. Note that there exists a future time period $\hat{t}_{0}\geq t_{0}$ such that:

[TABLE]

In words, the minimal number of observations from arm 2 by time $t_{0}$, including the one that arrived just before $t_{0}$, equals (in expectation) the minimal number of observations from arm 2 by time $\hat{t}_{0}$ without any additional information arrivals. Therefore, replacing $\epsilon_{t_{0}}=ct_{0}^{-1}$ with $\epsilon_{t_{0}}=c\hat{t}_{0}^{-1}$ would adjust the exploration rate to fit the amount of information actually collected at $t=t_{0}$, and the respective loss from exploitation. The regret reduction by this adjustment is illustrated in Figure 6. We therefore adjust the exploration rate to be $\epsilon_{t}=c(\tau(t))^{-1}$ for some virtual time $\tau(\cdot)$. We set $\tau(t)=t$ for all $t<t_{0}$, and $\tau(t)=\hat{t}_{0}+(t-t_{0})$ for $t\geq t_{0}$.

Solving (28) for $\hat{t}_{0}$, we write $\tau(t)$ in closed form:

[TABLE]

for some constant $c_{0}>1$. Therefore, the virtual time grows together with $t$, and advanced by a multiplicative constant whenever an auxiliary observation is collected.

D.2 A Rate-Optimal Adaptive ${\epsilon_{t}}$-greedy Policy

We apply the ideas discussed in §D.1 to design an $\epsilon_{t}$-greedy policy with adaptive exploration that dynamically adjusts the exploration rate in the presence of an unknown information arrival process. For simplicity, and consistent with standard versions of $\epsilon_{t}$-greedy (see, e.g., Auer et al. 2002), the policy below assumes prior knowledge of the parameter $\Delta$.

Define n^{\text{\tiny\mathrm{known}}}_{k,t} and $\bar{X}_{k,t}$ as in (2), and consider the following adaptation of the $\epsilon_{t}$-greedy policy.

$\epsilon_{t}$-greedy with adaptive exploration. Input: a tuning parameter $c>0$.

Initialization: set initial virtual times $\tau_{k,0}=0$ for all $k\in\cal K$ 2. 2.

At each period $t=1,2,\dots,T$:

1. (a)

Observe the vectors $\bm{h}_{t}$ and $\bm{\mathrm{Z}}_{t}$, and update virtual time indices for all $k\in\mathcal{K}$:

[TABLE] 2. (b)

With probability $\min\left\{1,\frac{c\sigma^{2}}{\Delta^{2}}\sum\limits_{k^{\mathrm{aux}}=1}^{K}\tau_{k,t}^{-1}\right\}$ select an arm at random: (exploration)

[TABLE]

Otherwise, select an arm with the highest estimated reward: (exploitation)

[TABLE] 3. (c)

Receive and observe a reward $X_{\pi_{t},t}$

At every period $t$, the $\epsilon_{t}$-greedy with adaptive exploration policy dynamically reacts to the information sample path by advancing virtual time indices associated with different arms based on auxiliary observations that were collected since the last period. Then, the policy explores with probability that is proportional to $\sum\limits_{k^{\mathrm{aux}}=1}^{K}\tau_{k^{\mathrm{aux}},t}^{-1}$, and otherwise pulls the arm with the highest empirical mean reward.

Every time additional information on arm $k$ is observed, a carefully selected multiplicative factor is used to advance the virtual time index $\tau_{k,t}$ according to the update rule $\tau_{k,t}=\left(\tau_{k,t-1}+1\right)\cdot\exp\left(\delta\cdot h_{k,t}\right)$, for some suitably selected $\delta$. In doing so, the policy effectively reduces exploration rates in order to explore over each arm $k$ at a rate that would have been appropriate without auxiliary information arrivals at a future time step $\tau_{k,t}$. This guarantees that the loss due to exploration is balanced with the loss due to best-arm misidentification throughout the horizon. Advancing virtual times based on the information sample path and the impact on the exploration rate of a policy are illustrated in Figure 6.

The following result characterizes the guaranteed performance and establishes the rate optimality of $\epsilon_{t}$-greedy with adaptive exploration in the presence of unknown information arrival processes.

Theorem 7

(Near optimality of $\epsilon_{t}$-greedy with adaptive exploration)* Let $\pi$ be an $\epsilon_{t}$-greedy with adaptive exploration policy, tuned by $c>\max\left\{16,\frac{10\Delta^{2}}{\sigma^{2}}\right\}$. Then, for every $T\geq 1$, and for any information arrival matrix $\bm{\mathrm{H}}$, and under prior knowledge of mappings $\{\phi_{k}\}$:*

[TABLE]

where LABEL:eps-greedy1, and LABEL:eps-greedy2 are positive constants that depend only on $\sigma$ and $\hat{\sigma}$.

The upper bound in Theorem 7 holds for any sample path of information arrivals as captured by the matrix $\bm{\mathrm{H}}$. This establishes that, similarly to Thompson sampling and UCB policies (but unlike the standard $\epsilon_{t}$-greedy) the $\epsilon_{t}$-greedy with adaptive exploration (through virtual time indices) guarantees rate optimality uniformly over the general class of information arrival processes defined in §2.

Key Ideas in the Proof. The proof decomposes regret into exploration and exploitation time periods. To bound the regret at exploration time periods express virtual time indices as

[TABLE]

Denoting by $t_{m}$ the time step at which the $m^{th}$ auxiliary observation for arm $k$ was collected, we establish an upper bound on the expected number of exploration time periods for arm $k$ in the time interval $[t_{m},t_{m+1}-1]$, which scales linearly with $\frac{c\sigma^{2}}{\Delta^{2}}\log\left(\frac{\tau_{k,t_{m+1}}}{\tau_{k,t_{m}}}\right)-1$. Summing over all values of $m$, we obtain that the regret over exploration time periods is bounded from above by

[TABLE]

To analyze regret at exploitation time periods we first lower bound the number of observations of each arm using Bernstein inequality, and then apply Chernoff-Hoeffding inequality to bound the probability that a sub-optimal arm would have the highest estimated reward, given the minimal number of observations on each arm. When $c>\max\left\{16,\frac{10\Delta^{2}}{\sigma^{2}}\right\}$, this regret component decays at rate of at most $t^{-1}$.

Numerical Analysis. Figure 7 follows the settings described in Figure 2 to visualize the impact of adjusting the exploration rate of an $\varepsilon_{t}$-greedy policy using virtual time indices as the amount of auxiliary information increases. Appendix E includes further numerical analysis of $\epsilon_{t}$-greedy with adaptive exploration with a variety of tuning parameter and information arrival scenarios, relative to other policies. We also test robustness with respect to misspecification of the gap parameter $\Delta$.

D.3 Proof of Theorem 7

Let $t^{\ast}=\lceil\frac{Kc\sigma^{2}}{\Delta^{2}}\rceil$, and fix $T\geq t^{\ast}$, $K\geq 2$, $\bm{\mathrm{H}}\in\{0,1\}^{K\times T}$, and fix a problem instance $(\bm{\nu},\bm{\nu}^{\mathrm{aux}})\in{\cal S}$ with the mean rewards $\bm{\mu}$ and the mean $\bm{\mathrm{y}}$ for auxiliary observations.

Step 1 (Decomposing the regret).

We decompose the regret of a suboptimal arm $k\neq k^{\ast}$ as follows:

[TABLE]

The component $J_{k,1}$ is the expected number of times arm $k$ is pulled due to exploration, while the the component $J_{k,2}$ is the one due to exploitation.

Step 2 (Analysis of $\tau_{k,t}$).

We will later find upper bounds for the quantities $J_{k,1}$, and $J_{k,2}$ that are functions of the virtual time $\tau_{k,T}$. A simple induction results in the following expression:

[TABLE]

Step 3 (Analysis of $J_{k,1}$).

Let $\bar{M}\coloneqq\sum\limits_{t=1}^{T}h_{k,t}$. For $0\leq m\leq\bar{M}$, let $t_{m}$ be the time step at which the $m$th auxiliary observation for arm $k$ is received, that is,

[TABLE]

Note that we dropped the index $k$ in the definitions above for simplicity of notation. Also define $\tau_{k,T+1}\coloneqq\left(\tau_{k,T}+1\right)\cdot\exp\left(\frac{h_{k,t}\Delta^{2}}{c\hat{\sigma}^{2}}\right)$. One obtains:

[TABLE]

where (a), (b), and (c) follow from the following lemma, the fact that $\exp\left(\frac{\Delta^{2}}{c\hat{\sigma}^{2}}\right)\left(\tau_{k,t_{m}}+t_{m+1,k}-t_{m}\right)=\tau_{k,t_{m+1}}$, and (30), respectively:

Lemma 14

For any $t>1/2$, and $n\in\{0,1,2,\dots\}$, we have $\log\frac{t+n+1}{t}\leq\sum\limits_{s=0}^{n}\frac{1}{t+s}\leq\log\frac{t+n+1/2}{t-1/2}$.

Proof.    We first, show the lower through an integral:

[TABLE]

Now, we show the upper bound through a similar argument:

[TABLE]

where (a) holds because we have $\frac{1}{t+s}\leq\int_{t+s-1/2}^{t+s+1/2}\frac{dx}{x}$ by Jensen’s inequality.

Step 4 (Analysis of n^{\text{\tiny\mathrm{known}}}_{k,t}).

To analyze $J_{k,2}$ we bound n^{\text{\tiny\mathrm{known}}}_{k,t}, the number of samples of arm $k$ at each time $t\geq t^{\ast}$. Fix $t\geq t^{\ast}$. Then, n^{\text{\tiny\mathrm{known}}}_{k,t} is a summation of independent Bernoulli r.v.’s. One has

[TABLE]

The term $\sum\limits_{s=t^{\ast}}^{t-1}\frac{c\sigma^{2}}{\Delta^{2}\tau_{k,s}}$ can be bounded from below similar to Step 3. Let $\bar{M}_{t}=\sum\limits_{s=1}^{t-1}h_{k,s}$ and $\underline{M}=\sum\limits_{s=1}^{t^{\ast}}h_{k,s}$. For $\underline{M\mkern-4.0mu}\mkern 4.0mu\leq m\leq\bar{M}_{t}$, let $t_{m,t}$ be the time step at which the $m$th auxiliary observation for arm $k$ is received up to time $t-1$, that is,

[TABLE]

One has that

[TABLE]

where (a) follows from Lemma 14. Putting together (32) and (D.3), one obtains:

[TABLE]

By Bernstein’s inequality in Lemma 10, we have

[TABLE]

where the last inequality follows from (30).

Step 5 (Analysis of $J_{k,2}$).

We note that

[TABLE]

We upper bound each summand as follows:

[TABLE]

We next bound the first term on the right-hand side of the above inequality; the second term can be treated similarly. One obtains:

[TABLE]

where, (a) holds by the conditional probability definition, (b) follows from Lemma 9 together with (35), and (c) holds by (30). Putting together (D.3), (D.3), (36), (D.3), and (D.3), the result is established.

Appendix E Numerical Analysis

E.1 Comparing Performance of Various Policies

Setup. \Copynumerics-setupWe simulate the performance of different policies using MAB instances with three arms. The reported results correspond to rewards that are Gaussian with means $\bm{\mu}=\begin{pmatrix}0.7&0.5&0.5\end{pmatrix}^{\top}$ and standard deviation $\sigma=0.5$, but we note that results are very robust with respect to the reward distributions. For simplicity we drew auxiliary information from the same distributions as the rewards. \Copyinformation-arrival-processesWe considered two information arrival processes: stationary information arrival process with $h_{k,t}$’s that are i.i.d. Bernoulli random variables with mean $\lambda$, where we considered values of $\lambda\in\{500/T,100/T,10/T\}$; and diminishing information arrival process where $h_{k,t}$’s are i.i.d. Bernoulli random variables with mean $\frac{\kappa^{\mathrm{aux}}}{t}$, where we considered values of $\kappa^{\mathrm{aux}}\in\{4,2,1\}$ for each time period $t$ and for each arm $k$.

In this part we experimented with three policies: UCB1, Thompson sampling, and $\epsilon_{t}$-greedy. We considered several variants of each policy: $(i)$ versions that ignores auxiliary information arrivals (EG, UCB1, TS); $(ii)$ version that updates empirical means and cumulative number of observations upon information arrivals but uses a standard time index (nEG); and $(iii)$ version that updates empirical means and cumulative number of observations upon information arrivals and uses virtual time indices (aEG, aUCB, aTS1). The difference between these versions is essentially in manner in which empirical means and reward observation counters are computed, as well as in the type of time index that is used. Namely, for the original versions:

[TABLE]

for the versions that account for auxiliary information while using standard time index:

[TABLE]

and for the versions with virtual time indices:

[TABLE]

where $c_{\pi}$ depends on the policy type $\pi$. We measured the average empirical regret $\sum\limits_{t=1}^{T}(\mu^{\ast}-\mu_{\pi_{t}})$, that is, the average difference between the performance of the best arm and the performance of the policy, over a decision horizon of $T=10^{4}$ periods, and averaging over 400 replications. \Copydetermine-tuning-parameter To determine the tuning parameter $c$ used by the policies we simulated their original versions that do not account for auxiliary information arrivals with $c=0.1\times m$, where we considered values of $m\in\{1,2,\dots,20\}$. The results presented below include the tuning parameter that resulted in the lowest cumulative empirical regret over the entire time horizon: $c=1.0$ for $\epsilon_{t}$-greedy, $c=1.0$ for UCB1, and $c=0.5$ for Thompson sampling (robustness with respect to this tuning parameter is discussed in E.2).

Results and Discussion. Plots comparing the empirical regret of the different algorithms for stationary and diminishing information arrival processes appear in Figures 8 and 9, respectively. For a stationary information arrival process with large $\lambda$, the regret is bounded uniformly by a constant and eventually stops growing with $T$, and when $\lambda$ is smaller we transition back to logarithmic regret, as discussed in §3.1.1. The impact of the timing of information arrivals on the achievable performance can be viewed by comparing the plots that correspond to the stationary and the diminishing information arrival processes. The expected total number of information arrivals in the first, second, and third columns with diminishing information arrival process is less than or equal to that of the first, second, and third column with stationary information arrival process, and the cumulative regrets for both cases are close to one another.

E.2 Different Values of the Tuning Parameter c

Setup. We use the setup discussed in §E.1 to study the effect of using different values for the tuning parameter $c$. The plots here detail results for parametric values of $c=\{0.4,1.0,1.6\}$ for $\epsilon_{t}$-greedy, $c\in\{0.4,1.0,1.6\}$ for UCB1, and $c=\{0.1,0.5,0.7\}$ for Thompson sampling, and using four different information arrival instances: stationary with $\lambda\in\{500/T,10/T\}$ and diminishing with $\kappa^{\mathrm{aux}}\in\{4.0,1.0\}$.

Results and Discussion. Plots comparing the empirical regret accumulation of policies with different tuning parameters $c$ for the stationary and diminishing information arrival processes appear in Figure 10. \Copymisspecification-tuning-parameterFor $\epsilon_{t}$-greedy and Thompson sampling, one may observe that when there is a lot of auxiliary information (the first and third columns) lower values of $c$ lead to better performance. However, when there is less auxiliary information (the second and fourth columns) very small values of $c$ may result in high regret. The reason for this is that when $c$ is small these policies tend to explore less, hence, in the presence of a lot of auxiliary information they incur little regret but if auxiliary information is rare these policies under-explores and incur a lot of regret. We observe a similar effect for UCB1 as well.

E.3 Misspecification of the Minimal Gap $\Delta$ for $\epsilon_{t}$-greedy with Adaptive Exploration

Setup. We use the setup from §E.1 to study the effect of misspecifying the minimum gap $\Delta$ for $\epsilon_{t}$-greedy with virtual time indices. We experiment with values of $\hat{\Delta}$ that are higher than, lower than, and equal to the true value of $\Delta=0.2$. The representative results below detail the empirical regret incurred by using $\hat{\Delta}\in\{0.05,0.2,0.35\}$, in different information arrival processes: stationary arrival processes with $\lambda\in\{500/T,10/T\}$ and diminishing arrival processes with $\kappa^{\mathrm{aux}}\in\{8.0,1.0\}$. We used the tuning parameter $c=1.0$ in all cases.

Results and discussion. Plots comparing the regret accumulation with different specified gaps $\hat{\Delta}$ appear in Figure 11. \Copymisspecification-Delta-e-greedyIn all cases, a conservative choice of the gap $\hat{\Delta}$ (that is, $\hat{\Delta}=0.05$) results in over-exploration and additional regret compared with the case that the exact value of the gap is used. On the other hand, a lower value of $\hat{\Delta}$ (that is, $\hat{\Delta}=0.4$) results in lower regret when the amount of information is high. The reason is that a higher value of $\hat{\Delta}$ leads to less exploration, which, when compensated by the presence of many auxiliary observations, leads to better performance overall. However, a higher value of $\hat{\Delta}$ can result in high regret when the amount of auxiliary information is low.

Appendix F Analysis with Content Recommendations Data

F.1 Description of Policies

We compare the performance of UCB1 when ignoring auxiliary data, with the one achieved by utilizing auxiliary data based on estimated mappings (aUCB1, see §4.1), and the adaptive 2-UCBs policy (see §5.3). Let $\tilde{t}_{t}\coloneqq\sum_{s=1}^{t-1}\mathcal{C}_{s}$ be the number of clicks up to time $t$. For the standard UCB1, the variables in (2) are set as follows:

[TABLE]

For the aUCB1 that utilizes auxiliary observations, they are set as follows:

[TABLE]

where $\sigma^{2}=1/4$, $\hat{\sigma}^{2}=\hat{\alpha}_{a,d-1}^{2}/4$, and $\hat{Z}_{k,t,m}=\hat{\alpha}_{a,d}Y_{k,t,m}$. The variables in 2-UCBs (in Table 2) are defined as follows:

[TABLE]

where $\hat{\bar{\sigma}}^{2}=\bar{\alpha}_{a,d}^{2}/4$ and $\hat{\bar{Z}}_{k,s,m}=\bar{\alpha}_{a,d}Y_{k,t,m}$. All algorithmic variants make decisions based on the following rule:

[TABLE]

All algorithms were tuned using the parametric value $c=0.05$, which was identified through exhaustive search as the value that optimizes the performance of the standard UCB1 over the data set.

F.2 Summary of Results

Figure 12 depicts the regret of the policies aUCB1 and 2-UCBs as function of the AIE and RMM. The color of each point is proportional to the regret of the corresponding policy for a certain day and article. The scaling of the colors are provided for each scatter plot. As you can see for both policies mapping misspecification deteriorates performance the most when auxiliary information is more effective. Furthermore, for the cases with mapping misspecification, 2-UCBs policy experiences less performance loss.

We used different selections of the tuning parameter $c\in\{0.03,0.05,0.1\}$ and the gap parameter $\Delta\in\{0.01,0.02,0.03,0.04\}$ or when $\Delta$ is $20\%$ of the $\mathrm{CVR}$ of each article and day pair. Table 3, 4, and 5 present the cumulative regret (normalized by the cumulative regret of UCB1 with $\Delta=0.03$ and $c=0.05$), the average Relative Improvement (RI), and the average No Harm (NH) rate, respectively; where the No Harm (NH) metric for policy $\pi$ is the fraction of cases in which $\pi$ has a better performance than UCB1:

[TABLE]