Practical Contextual Bandits with Regression Oracles

TL;DR

This paper introduces a new practical and theoretically sound approach for contextual bandits that uses regression oracles, generalizes existing methods, and performs well empirically.

Contribution

It proposes a novel technique combining realizability and agnostic methods using regression oracles, extending UCB and LinUCB to more expressive models.

Findings

Achieves low regret under certain assumptions

Performs comparably or better than existing baselines

Generalizes UCB and LinUCB to complex models

Abstract

A major challenge in contextual bandits is to design general-purpose algorithms that are both practically useful and theoretically well-founded. We present a new technique that has the empirical and computational advantages of realizability-based approaches combined with the flexibility of agnostic methods. Our algorithms leverage the availability of a regression oracle for the value-function class, a more realistic and reasonable oracle than the classification oracles over policies typically assumed by agnostic methods. Our approach generalizes both UCB and LinUCB to far more expressive possible model classes and achieves low regret under certain distributional assumptions. In an extensive empirical evaluation, compared to both realizability-based and agnostic baselines, we find that our approach typically gives comparable or superior results.

Figures

Tables

Table 1. Table 1: UCI datasets (before validation split).

Dataset	$n$	$d$	$K$
letter	20000	17	26
optdigits	5620	65	10
adult	45222	105	2
page-blocks	5473	11	5
pendigits	10992	17	10
satimage	6430	37	6
vehicle	846	19	4
yeast	1479	9	9

Table 2. Table 2: Mean reward matrix: yeast

1.00	0.02	0.26	0.89	0.20	0.31	0.97	0.34	0.39
0.25	1.00	0.65	0.29	0.03	0.52	0.30	0.10	0.09
0.49	0.92	1.00	0.61	0.33	0.84	0.22	0.37	0.62
0.21	0.67	0.13	1.00	0.27	0.69	0.39	0.97	0.15
0.26	0.63	0.11	0.39	1.00	0.25	0.34	0.17	0.69
0.78	0.11	0.22	0.16	0.22	1.00	0.68	0.39	0.55
0.40	0.71	0.40	0.45	0.23	0.48	1.00	0.99	0.43
0.41	0.48	0.85	0.87	0.22	0.46	0.33	1.00	0.96
0.07	0.76	0.49	0.74	0.44	0.85	0.09	0.23	1.00

Table 3. Table 3: Mean reward matrix: letter

1.00	0.33	0.82	0.04	0.11	0.60	0.53	0.42	0.34	0.62	0.44	0.74	0.52	0.58	0.65	0.99	0.82	0.41	0.88	0.82	0.05	0.72	0.80	0.74	0.71	0.54
0.12	1.00	0.40	0.22	0.72	0.99	0.26	0.67	0.60	0.72	0.94	0.35	0.25	0.40	0.75	0.72	0.41	0.99	0.45	0.37	0.71	0.08	0.40	0.77	0.76	0.28
0.19	0.47	1.00	0.73	0.19	0.33	0.84	0.62	0.89	0.98	0.84	0.18	0.62	0.48	0.40	0.74	0.83	0.68	0.14	0.70	0.06	0.19	0.92	0.41	0.15	0.68
0.16	0.65	0.25	1.00	0.96	0.07	0.51	0.34	0.66	0.84	0.60	0.59	0.12	0.71	0.20	0.49	0.04	0.32	0.86	0.56	0.55	0.37	0.83	0.28	0.13	0.56
0.27	0.78	0.18	0.78	1.00	0.04	0.56	0.67	0.94	0.79	0.75	0.50	0.04	0.82	0.01	0.55	0.57	0.11	0.06	0.57	0.49	0.30	0.04	0.63	0.12	0.01
0.28	0.30	0.18	0.07	0.78	1.00	0.25	0.52	0.25	0.85	0.48	0.62	0.97	0.35	0.22	0.98	0.59	0.98	0.97	0.71	0.02	0.61	0.25	0.13	0.37	0.20
0.77	0.93	0.03	0.26	0.27	0.14	1.00	0.25	0.36	0.05	0.24	0.88	0.96	0.66	0.30	0.06	0.86	0.16	0.27	0.55	0.25	0.84	0.50	0.48	0.91	0.92
0.24	0.02	0.67	0.27	0.01	0.10	0.42	1.00	0.21	0.75	0.46	0.11	0.22	0.93	0.01	0.64	0.64	0.68	0.58	0.78	0.82	0.65	0.18	0.73	0.28	0.84
0.57	0.09	0.91	0.46	0.94	0.04	0.11	0.76	1.00	0.45	0.82	0.42	0.19	0.84	0.11	0.29	0.22	0.46	0.32	0.91	0.79	0.71	0.14	0.61	0.85	0.92
0.66	0.26	0.28	0.64	0.72	0.31	0.68	0.51	0.83	1.00	0.91	0.12	0.84	0.95	0.57	0.00	0.03	0.41	0.46	0.48	0.68	0.75	0.82	0.35	0.61	0.39
0.73	0.56	0.59	0.39	0.63	0.87	0.65	0.13	0.09	0.68	1.00	0.31	0.89	0.86	0.81	0.36	0.64	0.60	0.24	0.59	1.00	0.05	0.24	0.33	0.80	0.44
0.06	0.32	0.83	0.74	0.28	0.73	0.32	0.15	0.98	0.26	0.61	1.00	0.64	0.43	0.40	0.05	0.08	0.45	0.92	0.23	0.87	0.81	0.17	0.31	0.43	0.86
0.63	0.82	0.50	0.58	0.45	0.26	0.62	0.58	0.87	0.92	0.57	0.69	1.00	0.68	1.00	0.94	0.14	0.94	0.04	0.03	0.18	0.31	0.98	0.94	0.76	0.62
0.97	0.57	0.21	0.13	0.76	0.53	0.82	0.79	0.67	0.78	0.69	0.43	0.83	1.00	0.78	0.09	0.95	0.48	0.89	0.08	0.94	0.31	0.42	0.69	0.09	0.21
0.58	0.39	0.11	0.01	0.90	0.67	0.32	0.89	0.97	0.08	0.26	0.53	0.92	0.23	1.00	0.90	0.34	0.23	0.18	0.05	0.96	0.15	0.96	0.34	0.06	0.82
0.80	0.46	0.77	0.75	0.45	0.28	0.14	0.91	0.08	0.73	0.08	0.67	0.06	0.11	0.48	1.00	0.03	0.64	0.90	0.48	0.84	0.71	0.93	0.97	0.59	0.95
0.71	0.46	0.92	0.58	0.24	0.39	0.42	0.16	0.02	0.05	0.68	0.25	0.15	0.20	0.82	0.89	1.00	0.74	0.58	0.49	0.64	0.95	0.80	0.41	0.25	0.00
0.29	0.98	0.42	0.54	0.06	0.14	0.99	0.54	0.22	0.64	0.73	0.50	0.33	0.72	0.13	0.72	0.45	1.00	0.63	0.86	0.32	0.70	0.12	0.44	0.72	0.89
0.56	0.63	0.53	0.35	0.85	0.57	0.26	0.80	0.83	0.45	0.68	0.09	0.72	0.34	0.02	0.71	0.55	0.83	1.00	0.99	0.33	0.13	0.04	0.32	0.21	0.57
0.96	0.22	0.33	0.27	0.27	0.69	0.89	0.58	0.40	0.43	0.55	0.31	0.26	0.91	0.51	0.12	0.57	0.25	0.01	1.00	0.36	0.68	0.61	0.17	0.30	0.72
0.43	0.13	0.17	0.73	0.62	0.56	0.06	0.39	0.45	0.58	0.70	0.72	0.59	0.27	0.41	0.78	0.47	0.40	0.85	1.00	1.00	0.63	0.91	0.15	0.29	0.65
0.18	0.28	0.94	0.31	0.10	0.50	0.08	0.25	0.96	0.84	0.15	0.25	0.05	0.20	0.81	0.91	0.62	0.09	0.50	0.67	0.11	1.00	0.76	0.39	0.83	0.17
0.26	0.80	0.68	0.78	0.18	0.95	0.18	0.70	0.31	0.51	0.91	0.78	0.75	0.11	0.91	0.90	0.98	0.11	0.38	0.27	0.85	0.90	1.00	0.22	0.05	0.88
0.95	0.75	0.82	0.31	0.13	0.10	0.67	0.14	0.92	0.24	0.75	0.61	0.34	0.63	0.02	0.76	0.17	0.61	0.12	0.57	0.73	0.80	0.14	1.00	0.41	0.40
0.83	0.19	0.76	0.74	0.42	0.14	0.70	0.88	0.18	0.12	0.21	0.44	0.46	0.76	0.16	0.90	0.52	0.28	0.02	0.59	0.20	0.44	0.96	0.20	1.00	0.84
0.03	0.67	0.47	0.34	0.50	0.43	0.56	0.11	0.36	0.93	0.50	0.64	0.47	0.97	0.12	0.35	0.68	0.79	0.40	0.74	0.37	0.10	0.02	0.14	0.99	1.00

Table 4. Table 4: Mean reward matrix: optdigits

1.00	0.56	0.12	0.40	0.78	0.51	0.18	0.85	0.96	0.98
0.19	1.00	0.23	0.03	0.95	0.92	0.29	0.17	0.40	0.51
0.31	0.43	1.00	0.56	0.83	1.00	0.33	0.09	0.77	0.15
0.73	0.96	0.07	1.00	0.84	0.15	0.77	0.78	0.68	0.13
0.04	0.66	0.25	0.99	1.00	0.06	0.70	0.63	0.90	0.16
0.61	0.32	0.76	0.16	0.93	1.00	0.83	0.23	0.11	0.67
0.58	0.88	1.00	0.28	0.74	0.28	1.00	0.49	0.87	0.16
0.97	0.05	0.70	0.65	0.05	0.20	0.33	1.00	0.37	0.53
0.35	0.51	0.26	0.85	0.62	0.30	0.78	0.90	1.00	0.86
0.82	0.87	0.38	0.61	0.42	0.24	0.06	0.82	0.38	1.00

Table 5. Table 5: Mean reward matrix: page-blocks

1.00	0.38	0.66	0.16	0.96
0.35	1.00	0.24	0.59	0.41
0.14	0.54	1.00	0.77	0.93
0.09	0.20	0.99	1.00	0.24
0.63	0.73	0.69	0.03	1.00

Table 6. Table 6: Mean reward matrix: pendigits

1.00	0.37	0.56	0.96	0.74	0.82	0.10	0.93	0.61	0.60
0.09	1.00	0.66	0.44	0.55	0.70	0.59	0.05	0.56	0.77
0.91	0.09	1.00	0.46	0.45	1.00	0.16	0.71	0.16	0.81
0.04	0.53	0.17	1.00	0.05	0.24	0.67	0.78	0.70	0.33
0.49	0.52	0.30	0.46	1.00	0.50	0.40	0.73	0.86	0.03
0.29	0.79	0.46	0.01	0.42	1.00	0.60	0.32	0.98	0.59
0.13	0.52	0.36	0.01	0.10	0.78	1.00	0.20	0.62	0.64
0.27	0.13	0.47	0.39	0.41	0.38	0.29	1.00	0.43	0.78
0.70	0.78	0.29	0.21	0.50	0.13	0.17	0.25	1.00	0.23
0.63	0.63	0.53	0.74	0.82	0.37	0.80	0.88	0.59	1.00

Table 7. Table 7: Mean reward matrix: satimage

1.00	0.06	0.12	0.79	0.98	0.27
0.87	1.00	0.64	0.78	0.63	0.13
1.00	0.63	1.00	0.62	0.34	0.76
0.11	0.52	0.63	1.00	0.11	0.29
0.07	0.67	0.23	0.52	1.00	0.45
0.73	0.97	0.20	0.72	0.79	1.00

Table 8. Table 8: Mean reward matrix: vehicle

1.00	0.36	0.18	0.52
0.01	1.00	0.80	0.76
0.67	0.03	1.00	0.40
0.19	0.77	0.62	1.00

Table 9. Table 9: Mean reward matrix: adult

1.00	0.61
0.66	1.00

Equations

$$E\left[r(a)\mid x,a\right]=f^{\star }(x,a)\text{for all x∈X and a∈A.}$$

$${\text{Reg}}_T=\sum _{t=1}^T{r}_t({\pi }^{\star }(x_t))-\sum _{t=1}^T{r}_t(a_t),$$

$$\text{textsc}Oracle(H)=\underset{f\in \mathcal{F}}{\mathrm{argmin}}\sum _{(w,x,a,y)\in H}w(f(x,a)-y{)}^2.$$

$${\text{High}}_{{\mathcal{F}}^{\prime }}(x,a)$$

$${\text{Low}}_{{\mathcal{F}}^{\prime }}(x,a)$$

$$a_{t}\sim\text{Unif}\left(\Bigl{\{}a\mathbin{\Bigm{|}}\text{High}_{\mathcal{F}_{m}}(x_{t},a)\geq{}\max_{a^{\prime}\in\mathcal{A}}\text{Low}_{\mathcal{F}_{m}}(x_{t},a^{\prime})\Bigr{\}}\right),$$

$$a_t=\underset{a\in \mathcal{A}}{\mathrm{argmax}}{\text{High}}_{{\mathcal{F}}_m}(x_t,a)$$

$$z_{\text{High}}$$

$$z_{\text{Low}}$$

$$\left|{\text{High}}_{{\hat{\mathcal{F}}}_m(\beta )}(x,a)-z_{\text{High}}\right|$$

$$\left|{\text{Low}}_{{\hat{\mathcal{F}}}_m(\beta )}(x,a)-z_{\text{Low}}\right|$$

$$\displaystyle\mathcal{F}(\varepsilon)=\Bigl{\{}f\in\mathcal{F}:\>\operatorname{\mathbb{E}}[r(\pi_{f}(x))]\geq\operatorname{\mathbb{E}}[r(\pi^{\star}(x))]$$

$$W_{\mathcal{F}}(x,a)={\text{High}}_{\mathcal{F}}(x,a)-{\text{Low}}_{\mathcal{F}}(x,a).$$

$$\mathrm{Dis}(\mathcal{F})$$

$$\displaystyle=\Bigl{\{}x\;\mathbin{\Bigm{|}}\;\exists{}f,f^{\prime}\in\mathcal{F}:\operatorname*{arg\,max}_{a\in\mathcal{A}}f(x,a)\neq{}\operatorname*{arg\,max}_{a\in\mathcal{A}}f^{\prime}(x,a)\Bigr{\}}.$$

$$A_{\mathcal{F}}(x)=\bigcup _{f\in \mathcal{F}}\underset{a\in \mathcal{A}}{\mathrm{argmax}}f(x,a).$$

$${\theta }_0:=\underset{\delta >0,\epsilon >0}{\sup }\frac{\delta }{\epsilon }{\mathrm{Pr}}_{D_{\mathcal{X}}}$$

$$\displaystyle\quad\exists{}a\in{}A_{\mathcal{F}(\varepsilon)}(x):W_{\mathcal{F}(\varepsilon)}(x,a)>\delta\Bigr{]}.$$

$${\text{Reg}}_T=\tilde{O}\left(T^{\frac{3}{4}}{\left(\log \left|\mathcal{G}\right|\right)}^{\frac{1}{4}}\sqrt{{\theta }_{0}K}\right).$$

$$\displaystyle\bigl{(}f(x,a)-f^{\star}(x,a)\bigr{)}^{2}$$

$$\displaystyle\quad{}\leq L_{1}\operatorname{\mathbb{E}}_{x^{\prime}\sim D_{\mathcal{X}}}\operatorname{\mathbb{E}}_{a^{\prime}\sim\text{Unif}\left(\mathcal{A}\right)}\Bigl{[}\bigl{(}f(x^{\prime},a^{\prime})-f^{\star}(x^{\prime},a^{\prime})\bigr{)}^{2}\Bigr{]}\enspace.$$

$${\Vert h\Vert }_{L_{\infty }({\mathcal{D}}^{\prime })}\le L_1\cdot {\Vert h\Vert }_{L_2({\mathcal{D}}^{\prime })}.$$

$$U_{\lambda}(a)\vcentcolon=\Bigl{\{}x\;\mathbin{\Bigm{|}}\;f^{\star}(x,a)\geq f^{\star}(x,a^{\prime})+\lambda\text{ for all $a^{\prime}\neq a$}\Bigr{\}}.$$

$$\displaystyle\bigl{(}f(x,a)-f^{\star}(x,a)\bigr{)}^{2}$$

$$\displaystyle\leq L_{2,\lambda}\operatorname{\mathbb{E}}_{x^{\prime}\sim D_{\mathcal{X}}}\operatorname{\mathbb{E}}_{a^{\prime}\sim{}\text{Unif}\left(\mathcal{A}\right)}\Bigl{[}\mathbf{1}\bigl{\{}x^{\prime}\in U_{\lambda}(a^{\prime})\bigr{\}}$$

$$\displaystyle\hskip 115.63243pt{}\cdot\bigl{(}f(x^{\prime},a^{\prime})-f^{\star}(x^{\prime},a^{\prime})\bigr{)}^{2}\Bigr{]}.$$

$$\frac{L_{2,0}}{K}{E}_{x\sim D_{\mathcal{X}}}[(f(x,{\pi }^{\star }(x))-f^{\star }(x,{\pi }^{\star }(x)){)}^2].$$

$${\text{Reg}}_T=\tilde{O}\left(\sqrt{T{L}_{2,0}\log \left|\mathcal{F}\right|}\right).$$

$${\text{Reg}}_T=\tilde{O}\left(\frac{L_1\log \left|\mathcal{F}\right|}{{\lambda }^2}+\sqrt{T{L}_{2,\lambda }\log \left|\mathcal{F}\right|}\right).$$

$$\frac{K}{\lambda_{\min}\left(\sum_{a\in\mathcal{A}}\operatorname{\mathbb{E}}_{x}\bigl{[}\mathbf{1}\left\{x\in{}U_{\lambda}(a)\right\}\,\phi(x,a)\phi(x,a)^{\top}\bigr{]}\right)},$$

$$\frac{K}{{\lambda }_{\min }\left({\sum }_{a\in \mathcal{A}}{E}_x\left[\varphi (x,a)\varphi (x,a{)}^{\top }\right]\right)}.$$

$$\frac{2Ks}{\psi_{\min}\left(\sum_{a\in\mathcal{A}}\operatorname{\mathbb{E}}_{x}\bigl{[}\mathbf{1}\left\{x\in{}U_{\lambda}(a)\right\}\,\phi(x,a)\phi(x,a)^{\top}\bigr{]}\right)},$$

$$\frac{2Ks}{\psi_{\min}\left(\sum_{a\in\mathcal{A}}\operatorname{\mathbb{E}}_{x}\bigl{[}\phi(x,a)\phi(x,a)^{\top}\bigr{]}\right)}.$$

$$\widetilde{O}\left(s\sqrt{KT\log{}d\,\bigm{/}\,\psi_{\min}\left(\operatorname{\mathbb{E}}_{x}\bigl{[}\phi(x,\pi^{\star}(x))\phi(x,\pi^{\star}(x))^{\top}\bigr{]}\right)}\right).$$

Full text

\xpatchcmd

1 Introduction

We study the design of practically useful, theoretically well-founded, general-purpose algorithms for the contextual bandits problem. In this setting, the learner repeatedly receives context, then selects an action, resulting in a received reward. The aim is to learn a policy, a rule for choosing actions based on context, so as to maximize the long-term cumulative reward. For instance, a news portal must repeatedly choose articles to present to each user to maximize clicks. Here, the context is information about the user, the actions represent the choice of articles, and the reward indicates if there was a click. We refer the reader to a recent ICML 2017 tutorial (http://hunch.net/~rwil/) for further examples and motivation.

Approaches to contextual bandit learning can broadly be put into two groups. Some methods (Langford & Zhang, 2008; Agarwal et al., 2014) are agnostic in the sense that they are provably effective for any given policy class and data distribution. In contrast, realizability-based approaches such as LinUCB and variants (Chu et al., 2011; Li et al., 2017; Filippi et al., 2010) or Thompson Sampling (Thompson, 1933) assume the data is generated from a particular parametrized family of models. Computationally tractable realizability-based algorithms are only known for specific model families, such as when the conditional reward distributions come from a generalized linear model.

The two groups of approaches seem to have different advantages and disadvantages. Empirically, in the contextual semibandit setting, Krishnamurthy et al. (2016) found that the realizability-based LinUCB approach outperforms all agnostic baselines using a linear policy class. However, the agnostic approaches were able to overcome this shortcoming by using a more powerful policy class. Computationally, previous realizability-based approaches have been limited by their reliance on either closed-form confidence bounds (as in LinUCB variants), or the ability to efficiently sample from and frequently update the posterior (as in Thompson sampling). Agnostic approaches, on the other hand, typically assume an oracle for cost-sensitive classification, which is in general computationally intractable, though often practically feasible for many natural policy classes.

In this paper, we aim to develop techniques that combine what is best about both of these approaches. To this end, in Section 3, we propose computationally efficient and practical realizability-based algorithms for arbitrary model classes. As is often done in agnostic approaches, we assume the availability of an oracle which reduces to a standard learning setting and knows how to efficiently leverage the structure of the model class. Specifically, we require access to a squared regression oracle over the model class that we use for predicting rewards, given contexts. Since regression can often be solved efficiently, the availability of such an oracle is a rather mild assumption, far more reasonable than the kind of cost-sensitive classification oracle more commonly assumed, which typically must solve NP-hard problems. In fact, for this reason, even the classification oracles are typically approximated by regression oracles in practice (see, e.g., Beygelzimer & Langford, 2009). Our main algorithmic components here are motivated by and adapted from a recent work of Krishnamurthy et al. (2017) on cost-sensitive active learning.

In Section 4, we prove that our algorithms are effective in the sense of achieving low regret under certain favorable distributional assumptions. Specifically, we show that our methods enjoy low regret so long as certain quantities like the disagreement coefficient (Hanneke et al., 2014; Krishnamurthy et al., 2017) are bounded. We also present a second set of bounds in terms of certain distributional coefficients which generalize the exploration parameters introduced by Bastani & Bayati (2015) from linear to general function classes. As a special consequence, we obtain nearly dimension-free results for sparse linear bandits in high dimensions.

Finally, in Section 5, we conduct a very extensive empirical evaluation of our algorithms on a number of datasets and against both realizability-based and agnostic baselines. In this test of practical effectiveness, we find that our approach gives comparable or superior results in nearly all cases, and we also validate the distributional assumptions required for low-regret guarantees on these datasets.

2 Preliminaries

We consider the following contextual bandit protocol. Contexts are drawn from an arbitrary space, $x\in\mathcal{X}$, actions are from a finite set, $a\in\mathcal{A}\vcentcolon=\{1,\ldots,K\}$, for some fixed $K$, and reward vectors are from a bounded set, $r\in[0,1]^{K}$, with component $r(a)$ denoting the reward for action $a\in\mathcal{A}$.

We consider a stochastic setting where there is a fixed and unknown distribution $D$ over the context-reward pairs $(x,r)$. Its marginal distribution over $\mathcal{X}$ is denoted by $D_{\mathcal{X}}$. The learning protocol proceeds in rounds $t=1,\ldots,T$. In each round $t$, nature samples $(x_{t},r_{t})$ according to $D$ and reveals $x_{t}$ to the learner. The learner chooses an action $a_{t}\in\mathcal{A}$ and observes the reward $r_{t}(a_{t})$. The goal of the learner is to maximize the reward and do well compared with any strategy that models the expected reward $\operatorname{\mathbb{E}}[r(a)\mathbin{|}x,a]$ via a function $f:\mathcal{X}\times\mathcal{A}\to[0,1]$. These mappings $f$ are drawn from a given class of predictors $\mathcal{F}$, such as the class of linear predictors or regression trees.

The main assumption this paper follows is that the class $\mathcal{F}$ is rich enough to contain a predictor that perfectly predicts the expected reward of any action under any context, that is:

Assumption 1** (Realizability).**

There is a predictor $f^{\star}\in\mathcal{F}$ such that

[TABLE]

Given a predictor $f\in\mathcal{F}$, the associated optimal strategy $\pi_{f}:\mathcal{X}\to\mathcal{A}$, called a policy, always picks the action with the highest predicted reward, i.e., $\pi_{f}(x)\vcentcolon=\operatorname*{arg\,max}_{a\in\mathcal{A}}f(x,a)$ (breaking ties arbitrarily). We use the abbreviation $\pi^{\star}\vcentcolon=\pi_{f^{\star}}$ for the underlying optimal policy. The formal goal of the learner is then to minimize the regret

[TABLE]

which compares the accumulated rewards between the optimal policy and the learner’s strategy. The classic Exp4 algorithm (Auer et al., 2002) achieves an optimal regret bound of order $O(\sqrt{TK\ln|\mathcal{F}|})$ (for any finite $\mathcal{F}$), but the computational complexity is unfortunately linear in $|\mathcal{F}|$.

Regression Oracle

To overcome the computational obstacle, our algorithms reduce the contextual bandit problem to weighted least-squares regression. Abstracting the computational complexity, we assume access to a weighted least-squares regression oracle over the predictor class $\mathcal{F}$, which takes any set $H$ of weighted examples $(w,x,a,y)\in{\mathbb{R}_{+}\times\mathcal{X}\times\mathcal{A}\times[0,1]}$ as input, and outputs the predictor with the smallest weighted squared loss:

[TABLE]

As mentioned, such regression tasks are very common in machine learning practice and the availability of such oracle is thus a very mild assumption.

3 Algorithms

The high-level idea of our algorithms is the following. As data is collected, we maintain a subset of $\mathcal{F}$, which we refer to as the version space, that only contains predictors with small squared loss on observed data. When a new example arrives, we construct upper and lower confidence bounds on the reward of each action based on the predictors in the version space. Finally, with these confidence bounds, we either optimistically pick the action with the highest upper bound, similar to UCB and LinUCB, or randomize among all actions that are potentially the best.

The challenge here is to maintain such version spaces and confidence bounds efficiently, and we show that this can be done using a simple binary search together with a small number of regression oracle calls.

We now describe our algorithms more formally. First, we define the upper and lower reward bounds with respect to a a subset $\mathcal{F}^{\prime}\subseteq\mathcal{F}$ as

[TABLE]

Our algorithms will induce the confidence bounds by instantiating these quantities using the version space as $\mathcal{F}^{\prime}$. To reduce computational costs, our algorithms update according to a doubling epoch schedule. Epoch $m$ will begin at time $\tau_{m}=2^{m-1}$, and $M=O(\log T)$ is the total number of epochs. At each epoch $m$ our algorithms (implicitly) construct a version space $\mathcal{F}_{m}\subseteq{}\mathcal{F}$, and then select an action based on the reward ranges defined by $\text{High}_{\mathcal{F}_{m}}\!(x,a)$ and $\text{Low}_{\mathcal{F}_{m}}\!(x,a)$ for each time $t$ that falls into epoch $m$. Specifically, we consider two algorithm variants: the first one uniformly at random picks from actions that are plausible to be the best, that is,

[TABLE]

where $\text{Unif}\left(S\right)$ denotes the uniform distribution over a set $S$; the second one simply behaves optimistically and picks the action with the highest upper reward bound, that is,

[TABLE]

(ties are broken arbitrarily). For technical reasons, the optimistic variant also spends the first few epochs doing pure exploration as a warm start for the algorithm.

To construct these version spaces, we further introduce the following least-squares notation for any $m\geq 2$:

• •

\widehat{R}_{m}(f)=\frac{1}{\tau_{m}-1}\sum_{s<\tau_{m}}\bigl{(}f(x_{s},a_{s})-r_{s}(a_{s})\bigr{)}^{2},

• •

\widehat{\mathcal{F}}_{m}(\beta)=\bigl{\{}f\in\mathcal{F}\mathbin{\bigm{|}}\widehat{R}_{m}(f)-\min_{f\in\mathcal{F}}\widehat{R}_{m}(f)\leq\beta\bigr{\}},

and also let $\widehat{\mathcal{F}}_{1}(\beta)=\mathcal{F}$ for any $\beta$. With this notation $\mathcal{F}_{m}$ is simply set to $\widehat{\mathcal{F}}(\beta_{m})$ for some tolerance parameter $\beta_{m}$.

Product Classes

Sometimes it is desirable to have a product predictor class, that is, $\mathcal{F}=\mathcal{G}^{\mathcal{A}}$, where $\mathcal{G}:\mathcal{X}\to\left[0,1\right]$ is a “base class” and each $f\in\mathcal{F}$, described by a $K$-tuple $(g_{a})_{a\in\mathcal{A}}$ where $g_{a}\in\mathcal{G}$, predicts according to $f(x,a)=g_{a}(x)$. Similar to the general case, we introduce the following notation for $m\geq 2$:

• •

$\widehat{\mathcal{R}}_{m}(g,a)=\frac{1}{\tau_{m}-1}\sum_{s<\tau_{m}}(g(x_{s})-r_{s}(a_{s}))^{2}\mathbf{1}\left\{a_{s}=a\right\}$,

• •

$\widehat{\mathcal{G}}_{m}(\beta,a)=\left\{g\in\mathcal{G}\mid{}\widehat{\mathcal{R}}_{m}(g,a)-\min_{g\in\mathcal{G}}\widehat{\mathcal{R}}_{m}(g,a)\leq\beta\right\}$,

and let $\widehat{\mathcal{G}}_{1}(\beta,a)=\mathcal{G}$ for any $\beta$. In this case we construct $\mathcal{F}_{m}$ as $\prod_{a\in\mathcal{A}}\widehat{\mathcal{G}}_{m}(\beta_{m},a)$ for some tolerance parameter $\beta_{m}$.

Our two procedures are described in detail in Algorithm 1 and Algorithm 2.

3.1 Efficient Reward-Range Computation

Both Algorithms 1 and 2 hinge on the computation of the reward bounds $\text{High}_{\mathcal{F}_{m}}$ and $\text{Low}_{\mathcal{F}_{m}}$. It turns out that this can be carried out efficiently via a small number of calls to the regression oracle.

Specifically, to calculate the confidence bounds for a given $x$, $a$, we augment the data set $H_{m}$ with a single example $(x,a,r)$ with a weight $w$, and set its reward $r$ beyond the reward range. For the upper confidence bound, we use $r=2$; for the lower confidence bound $r=-1$. By increasing the weight $w$, we force the regression oracle to perform better on this single example. For the upper confidence bound it means to predict higher rewards as $w$ increases, while getting worse performance on the remaining examples. The binary search over $w$ then identifies, up to a given precision, the weight $w$ and the corresponding predicted value at $x$ and $a$, at which the performance on the previous examples suffers by exactly the desired tolerance $\beta$. See Algorithm 3 for details, including the choice of the initial weight.

In Appendix A.1 we show that this strategy indeed works as intended and in $O(\log(1/\alpha))$ iterations computes the confidence bounds up to a precision of $\alpha$. The guarantee is formalized in the following theorem:

Theorem 1**.**

Let $H_{m}=\{(x_{s},a_{s},r_{s}(a_{s}))\}_{s=1}^{\tau_{m}-1}$. If the function class $\mathcal{F}$ is convex and closed under pointwise convergence, then the calls

[TABLE]

terminate after $O(\log(1/\alpha))$ oracle invocations and the returned values satisfy

[TABLE]

Compared to the procedure used by Krishnamurthy et al. (2017), Algorithm 3 is much simpler and achieves an exponential improvement in terms of oracle calls, namely, $O(\log(1/\alpha))$ as opposed to $O(1/\alpha)$, when $\mathcal{F}$ is convex. Compared to oracles used in cost-sensitive classification, convexity is not a strong assumption for regression oracles. Nonetheless, when $\mathcal{F}$ is not convex, the reward ranges can be computed with $O(1/\alpha)$ oracle calls using the techniques of Krishnamurthy et al. (2017).

4 Regret Guarantees

In this section we provide regret guarantees for RegCB (Algorithm 1 and Algorithm 2). Note that RegCB is not minimax optimal: while it can obtain O\bigl{(}\sqrt{KT\log\lvert\mathcal{F}\rvert}\bigr{)} regret or even logarithmic regret under certain distributional assumptions, which we describe shortly, for some instances it can make as many as $\left\lvert\mathcal{F}\right\rvert$ mistakes, which is suboptimal:

Proposition 1** (Bad instance for confidence-based strategies).**

For every $\epsilon\in(0,1]$ and $N\in\mathbb{N}$ there exists a class of reward predictors with $|\mathcal{F}|=N+1$ and a distribution for which both Algorithms 1 and 2 have regret across $T$ rounds lower bounded by $(1-\epsilon)\cdot\min\left\{N,\widetilde{\Omega}(T)\right\}$.

Proposition 1 is proved in Appendix A.2. The proof is build on a well known, albeit rather pathological instance. In contrast, our strong empirical results in the following section show that such instances are not encountered in practice. In order to understand the typical behavior of such algorithms, prior works have considered structural assumptions such as finite eluder dimension (Russo & Van Roy, 2013) or disagreement coefficients (Hanneke et al., 2014; Krishnamurthy et al., 2017). In the next two subsections, we use similar ideas to analyze the regret incurred by our algorithm. For simplicity, we assume that $\text{High}_{\mathcal{F}_{m}}$ and $\text{Low}_{\mathcal{F}_{m}}$ are computed exactly.

4.1 Disagreement-based Analysis

Disagreement coefficients come from the active learning literature (Hanneke et al., 2014), and roughly assume that given a set of functions which fit the historical data well, the probability that these functions make differing predictions on a new example is small. This rules out the bad case of Proposition 1, where a near-optimal predictor significantly disagrees from the others on each context. Our development in this subsection largely follows Krishnamurthy et al. (2017), with appropriate modifications to translate from active learning to contextual bandits. We start by recalling some formal definitions, leading up to the definition of the disagreement coefficient.

Definition 1**.**

For any $\varepsilon>0$, the policy-regret ball of radius $\varepsilon$ for $\mathcal{F}$ is defined as

[TABLE]

Definition 2** (Reward width).**

For any predictor class $\mathcal{F}$, context $x$, and action $a$, the reward width is defined as

[TABLE]

Definition 3** (Disagreement Region).**

For any predictor class $\mathcal{F}$, the disagreement region $\mathrm{Dis}(\mathcal{F})$ is defined as111When the maximizing action $\operatorname*{arg\,max}_{a\in\mathcal{A}}f(x,a)$ is not unique, the “$\neq$” in the disagreement set definition checks that the two argmax sets are identical.

[TABLE]

Definition 4** (Disagreement set).**

For a predictor class $\mathcal{F}$ and a context $x$, the disagreement set at $x$ is defined as

[TABLE]

With these preliminaries, the disagreement coefficient is defined as follows.

Definition 5** (Disagreement Coefficient).**

The disagreement coefficient for $\mathcal{F}$ (with respect to $D_{\mathcal{X}}$) is defined as

[TABLE]

Informally, the disagreement coefficient is small if on most contexts either all functions in $\mathcal{F}(\varepsilon)$ choose the same action according to their greedy policies or all actions chosen by those policies have a low range of predicted rewards.

The following theorem provides regret bounds in terms of the disagreement coefficient. In this theorem and subsequent theorems we use $\widetilde{O}$ to suppress polynomial dependence on $\log{}T$, $\log{}K$, and $\log(1/\delta)$, where $\delta$ is the failure probability. Moreover, all results can be improved to bounds that are logarithmic (in $T$) under the standard Massart noise condition (see the appendix for the definition and the complete theorem statements under this condition).

Theorem 2**.**

With $\beta_{m}=\frac{(M-m+1){C_{\delta}}}{\tau_{m}-1}$ and ${C_{\delta}}=16\log\left(\frac{2|\mathcal{G}|KT^{2}}{\delta}\right)$, Algorithm 1 with Option I ensures that with probability at least $1-\delta$,

[TABLE]

We state the theorem above for finite classes for simplicity. See Theorem 5 in Appendix A.3 for the full version of this theorem, which applies to infinite classes and additionally obtains faster rates under the Massart noise condition.

Discussion

Theorem 2 critically uses the product class structure, specifically, the fact that the set $A_{t}$ computed by the algorithm coincides with the disagreement set $A_{\mathcal{F}_{m}}(x_{t})$ for $t\in\left\{\tau_{m},\ldots,\tau_{m+1}-1\right\}$. This is true for product classes, but not necessarily for general (non-product) predictor classes. Computing the disagreement set efficiently for non-product classes is a challenge for future work.

While bounding the disagreement coefficients a priori often requires strong assumptions on the model class and the distribution, the size of disagreement set can be easily checked empirically under the product class assumption, and we include this diagnostic in our experimental results.

Finally, it is not obvious how to use the disagreement coefficient to analyze Algorithm 2. Our analysis crucially requires that any plausibly optimal action $a$ be chosen with a reasonable probability, something which the optimistic algorithm fails to ensure.

4.2 Moment-based Analysis

The disagreement-based analysis of Theorem 2 is not entirely satisfying because, even for simple linear predictors such as in LinUCB (Chu et al., 2011) it is known that fairly strong assumptions on the context distribution $D_{\mathcal{X}}$ such as log-concavity are required to bound the disagreement coefficient $\theta_{0}$ (Hanneke et al., 2014). In order to capture and extend the linear setting with distributional assumptions on the contexts, prior work has used the notion of eluder dimension (Russo & Van Roy, 2013). It remains challenging, however, to show examples with a small eluder dimension beyond linearly parameterized functions. In addition, taking the worst-case over all histories, as in the definition of eluder dimension, is overly pessimistic in the stochastic contextual-bandit setting.

To address the shortcomings of both the disagreement-based analysis as well as eluder dimension for i.i.d. settings, we next define a couple of distributional properties which we then use to analyze the regret of our both algorithms.

Definition 6** (Surprise bound).**

The surprise bound $L_{1}>0$ is the smallest constant such that for all $f\in\mathcal{F}$, $x\in\mathcal{X}$, and $a\in\mathcal{A}$,

[TABLE]

\comment

Let $\mathcal{D}^{\prime}=\mathcal{D}_{\mathcal{X}}\tens{}\textrm{Uniform}(\left[K\right])$. There exists constant $L_{1}$ such that for all $h\in\mathcal{F}-f^{\star}$,

[TABLE]

Remark 1**.**

The $L_{\infty}$ norm can be relaxed to the $\psi_{2}$ Orlicz norm at the cost of an extra concentration argument.

The surprise bound is small if functions with a small expected squared error to $f^{\star}$ (under a uniform choice of actions) do not encounter a much larger squared error on any single context-action pair.

The second quantity which we call the implicit exploration coefficient (IEC for short) relates the expected regression error under actions chosen by the optimal policy to the worst-case error on any other context-action pair. Specifically, for any $\lambda\in[0,1]$, first define $U_{\lambda}(a)$ to be the set of contexts where $a$ is the best action by a margin of $\lambda$:

[TABLE]

Definition 7** (Implicit exploration coefficient—IEC).**

For any $\lambda\in[0,1]$, the implicit exploration coefficient $L_{2,\lambda}>0$ is the smallest constant such that for all $f\in\mathcal{F}$, $x\in\mathcal{X}$, and $a\in\mathcal{A}$,

[TABLE]

We next make a couple remarks about these definitions and their impact on the performance of Algorithm 1 and Algorithm 2, and then spell them out more precisely in Theorem 3 and Theorem 4.

• •

By definition, $L_{2,\lambda}$ is non-decreasing in $\lambda$. For Algorithm 1 we can simply use $\lambda=0$, for which it is sufficient to replace right-hand side of (2) with

[TABLE]

The analysis of Algorithm 2 requires $\lambda>0$, and this $\lambda$ must be used to tune the algorithm’s warm-start period.

• •

We always have $L_{1}\leq{}L_{2,0}$, but $L_{1}$ may be much smaller. Only Algorithm 2 has a regret bound depending on $L_{1}$ directly, whereas the regret of Algorithm 1 is independent of this constant.

With this in mind, we proceed to state the regret bound for Algorithm 1 with a general predictor class $\mathcal{F}$:

Theorem 3**.**

With $\beta_{m}=\frac{(M-m+1){C_{\delta}^{\prime}}}{\tau_{m}-1}$ where ${C_{\delta}^{\prime}}=16\log\left(\frac{2|\mathcal{F}|T^{2}}{\delta}\right)$, Algorithm 1 with Option II ensures that with probability at least $1-\delta$,

[TABLE]

We now move on to describe the performance guarantee for Algorithm 2. Because this optimistic strategy does not explore as readily as the elimination-based strategy Algorithm 1, the analysis requires both that (i) the IEC $L_{2,\lambda}$ be invoked for some $\lambda>0$ and (ii) that the algorithm use a warm-start period whose size grows as $1/\lambda^{2}$.

Theorem 4**.**

With $\beta_{m}=\frac{(M-m+1){C_{\delta}^{\prime}}}{\tau_{m}-1}$ where ${C_{\delta}^{\prime}}=16\log\left(\frac{2|\mathcal{F}|T^{2}}{\delta}\right)$ and $M_{0}=2+\left\lfloor\log_{2}\left(1+\frac{(2M+3)L_{1}{C_{\delta}^{\prime}}}{\lambda^{2}}\right)\right\rfloor$ for any $\lambda\in(0,1)$, Algorithm 2 ensures that with probability at least $1-\delta$,

[TABLE]

Because Algorithm 2 requires warm start, the regret bounds of Theorem 4 for Algorithm 2 are always worse than those of Theorem 3 for Algorithm 1. Appendix A.4 contains full versions of these theorems, Theorem 6 and Theorem 7, which—as in the disagreement case—obtain faster rates under the Massart noise condition and apply to infinite classes.

We now bound the regret of both algorithms for some special cases.

Linear classes

Consider the linear setting, as for instance in LinUCB, with a fixed feature map $\phi:\mathcal{X}\times\mathcal{A}\to\mathbb{R}^{d}$ and $\mathcal{F}=\left\{(x,a)\mapsto{}w^{\top}\phi(x,a)\mathbin{|}w\in\mathcal{W}\right\}$ for some $\mathcal{W}\subseteq{}\mathbb{R}^{d}$.

Proposition 2**.**

• •

If $\left\|\phi(x,a)\right\|_{2}\leq{}1$ and $\left\|w\right\|_{2}\leq{}1$ then $L_{2,\lambda}$ is bounded by

[TABLE]

where $\lambda_{\min}(\cdot)$ is the smallest eigenvalue of a matrix, and $L_{1}$ is bounded by

[TABLE]

• •

In the sparse high-dimensional setting with $\left\|\phi(x,a)\right\|_{\infty}\leq{}1$, $\left\|w\right\|_{\infty}\leq{}1$, and $\left\|w\right\|_{0}\leq{}s$, then $L_{2,\lambda}$ is bounded by

[TABLE]

where $\psi_{\min}(A)\coloneqq\min_{w\neq{}0:\>\left\|w\right\|_{0}\leq{}2s}w^{\top}\!\!Aw\,/\,w^{\top}w$ is the minimum restricted eigenvalue for $2s$-sparse predictors (Raskutti et al., 2010). The coefficient $L_{1}$ is bounded by

[TABLE]

We emphasize again that Algorithm 1 has a better regret bound than Algorithm 2 due to the warm-start phase in Algorithm 2. This is most easily seen by noting that $L_{2,\lambda}$ is non-decreasing in $\lambda$, then observing that the regret of Algorithm 1 depends on $L_{2,0}$ while the regret of Algorithm 2 requires $\lambda>0$ due to warm start (recall also that $L_{1}\leq L_{2,0}$). For the linear example above, this can be observed more directly by noting that the moment matrices \sum_{a}\operatorname{\mathbb{E}}_{x}\bigl{[}\mathbf{1}\left\{x\in{}U_{\lambda}(a)\right\}\,\phi(x,a)\phi(x,a)^{\top}\bigr{]} that appear in $L_{2,\lambda}$ in Proposition 2 are lower bounded by \operatorname{\mathbb{E}}_{x}\bigl{[}\phi(x,\pi^{\star}(x))\phi(x,\pi^{\star}(x))^{\top}\bigr{]} in the Loewner order when $\lambda=0$.

Sparse bandits

For the sparse high-dimensional setting above, we can apply Theorem 3 by discretizing the set of weights and invoking a standard covering argument to obtain $\log\left\lvert\mathcal{F}\right\rvert={}O\left(s\log d\right)$222This is made precise via Lemma 9 in the appendix.. This yields a near dimension-independent bound on ${\text{\rm Reg}}_{T}$ of

[TABLE]

This improves upon the moment matrix conditions of Bastani & Bayati (2015), although our algorithm is only efficient in the oracle model.333Because the predictor class $\mathcal{F}$ is non-convex, this would require the slower binary search algorithm of Krishnamurthy et al. (2017). Furthermore, Algorithm 1 does not require a warm start based on distributional parameters unlike their algorithm (or our Algorithm 2). Note that without the scaling with $K$ as in our result, a $\sqrt{d}$ dependence is unavoidable (Abbasi-Yadkori et al., 2012). The result highlights the strengths of our analysis in the best case compared with eluder dimension, which does not adapt to sparsity structures. On the other hand, for the standard LinUCB setting, our result is inferior by at least a factor of $K$.

Discussion

Our moment-based analysis is influenced by the results of Bastani & Bayati (2015) for the (high-dimensional) linear setting. Our analysis extends to general classes and, when applied to Algorithm 1, it makes weaker assumptions. Similar assumptions have been used to analyze purely greedy linear contextual bandits (Bastani et al., 2017; Kannan et al., 2018); our assumptions are strictly weaker.

5 Experiments

We compared our new algorithms with existing oracle-based alternatives. In addition to showing that RegCB 444RegCB refers collectively to both Algorithms 1 and 2. has strong empirical performance, our experiments also provide a more extensive empirical study of oracle-based contextual bandit algorithms than any past works (e.g., Agarwal et al., 2014, Krishnamurthy et al., 2016). Detailed descriptions of the datasets, benchmark algorithms, and oracle configurations, as well as further experimental results are included in Appendix B.

Datasets

We begin with 10 datasets with full reward information and simulate bandit feedback by withholding the rewards for actions not selected by the algorithm. First, there are two large-scale learning-to-rank datasets, Microsoft MSLR-WEB30k (mslr) (Qin & Liu, 2010) and Yahoo! Learning to Rank Challenge V2.0 (yahoo) (Chapelle & Chang, 2011), that have previously been used to evaluate contextual semibandits (Krishnamurthy et al., 2016). Second, we use a collection of eight classification datasets from the UCI repository (Lichman, 2013), summarized in Table 1 of Appendix B.1.

The ranking datasets have natural rewards (relevances), but the rewards for the classification datasets always have multiclass structure ($1$ for the correct action and [math] for all others). Therefore, to ensure that we evaluate at the full generality of the contextual bandit setting, we create eight “noisy” UCI datasets by sampling new rewards for the datasets according to a noisy reward matrix model described in Appendix B. This yields additional 8 datasets for the total of 18.

On each dataset we consider several replicates obtained by randomly permuting examples and, on noisy UCI, also randomly generating rewards. All the methods are evaluated on the same set of replicates.

Algorithms

We evaluate both Algorithm 1 and Algorithm 2 against three baselines, all based on various optimization-oracle assumptions. First, we use the standard $\epsilon$-Greedy strategy (Langford & Zhang, 2008). Second, we use the minimax-optimal ILOVETOCONBANDITS (ILTCB) strategy of Agarwal et al. (2014).555We use an implementation available at https://github.com/akshaykr/oracle_cb, which was also used by Krishnamurthy et al. (2016).

The $\epsilon$-Greedy and ILTCB strategies both assume cost-sensitive classification oracles and come equipped with theoretical guarantees. The last baseline we consider is a bootstrapping-based exploration strategy of Dimakopoulou et al. (2017) (henceforth Bootstrap), which works in the regression-oracle model as we consider here, but without the corresponding theoretical analysis.

Note that the LinUCB algorithm (Chu et al., 2011; Abbasi-Yadkori et al., 2011), which is a natural baseline as well, coincides with our Algorithm 2 (with a linear oracle), so we only plot the performance of RegCB with a linear oracle.

All of the algorithms update on an epoch schedule with epoch lengths of $2^{i/2}$, which is a theoretically rigorous choice for each algorithm.

Oracles

We consider two baseline predictor classes $\mathcal{F}$: $\ell_{2}$-regularized linear functions (Linear) and gradient-boosted depth-$5$ regression trees (GB5). For the regularized linear class, Algorithm 2 is equivalent to LinUCB on an epoch schedule.666More precisely, it is equivalent to the well-known OFUL variant of LinUCB (Abbasi-Yadkori et al., 2011).

When running both RegCB variants with the GB5 oracle, we use a simple heuristic to substantially speed up the computation. At the beginning of each epoch $m$, we find the best regression tree ensemble on the dataset so far (i.e., with respect to $\widehat{R}_{m}$). Throughout the epoch, we keep the structure of the ensemble fixed and in each call to $\textsc{Oracle}(H)$ we only re-optimize the predictions in leaves. This can be solved in closed form, similar to LinUCB, so the full binary search procedure (Algorithm 3) does not need to be run.

Parameter Tuning

We evaluate each algorithm for eight exponentially spaced parameter values across five repetitions. For $\epsilon$-Greedy we tune the constant $\epsilon$, and for ILTCB we tune a certain smoothing parameter (see Appendix B). For Algorithm 1 and Algorithm 2 we set $\beta_{m}=\beta$ for all $m$ and tune $\beta$. For Algorithm 2 we use a warm start of [math]. We tune a confidence parameter similar to $\beta$ for Bootstrap.

Evaluation

Each dataset is split into “training data”, for which algorithm receives one example at a time and must predict online, and a holdout validation set. Validation is performed by simulating the algorithm’s predictions on examples from the holdout set without allowing the algorithm to incorporate these examples. We also plot the validation reward of a “supervised” baseline obtained by training the oracle (either Linear or GB5) on the entire training set at once (including rewards for all actions).

For Algorithms 1 and 2 we show average reward at various numbers of training examples for the best fixed parameter value in each dataset. For the baselines, we take the pointwise maximum of the average validation reward across all parameter values for each number of examples to be as generous as possible. Thus, the curves for our methods correspond to an actual run of the algorithm, while the baselines are an upper envelope aggregating multiple parameter values.

Results: Performance

Figure 1 shows average reward of each algorithm on a holdout validation set for three representative datasets, letter from UCI, letter-noise (the variant with simulated rewards), and yahoo.

RegCB (both Algorithms 1 and 2) outperforms all baselines on the unmodified UCI datasets (e.g., letter in Figure 1). On the noisy variants (e.g., letter+N in Figure 1), the performance of the ILTCB and Bootstrap benchmarks improves significantly, with Bootstrap slightly edging out the rest of the algorithms. On the yahoo ranking dataset (Figure 1, right), the ordering of the algorithms in performance is similar to noisy UCI datasets.

Validation performance plots for all datasets are in Appendix B. Overall, we see that RegCB methods and Bootstrap generally dominate the field. While Bootstrap can outperform RegCB methods when using GB5 models, the gap is typically quite small. For linear models, RegCB methods generally outperform Bootstrap. This hints that the stronger relative performance of Bootstrap under GB5 models might be partly due to the approximation we make by only considering a fixed ensemble structure in each epoch. We also observe that when RegCB methods outperform Bootstrap, the performance gap can often be quite large. We will see further evidence of this behavior in the next set of results.

Results: Aggregate Performance

To rigorously draw conclusions about overall performance, Figure 2 aggregates performance across all datasets. We compute “normalized relative loss” for each algorithm by rescaling the validation reward (computed as in Figure 1) so that, at each round, the best performing algorithm has loss [math] and the worst-performing has loss $1$. In each plot of Figure 2 we consider normalized relative losses at a specific cutoff time ($1000$ examples in the left plot, and all examples in the center and right), and for each method we plot how often, i.e., on how many datasets, it achieves any given value of loss or better, as a function of the loss value. Thus, curves towards top left corner correspond to better methods, i.e., the methods that achieve lower relative loss on more datasets. The intercept at the relative loss [math] shows the number of datasets where each algorithm is the best, and the intercept at 0.99 shows the number of datasets where the algorithm is not the worst (so the distance from top is the number of datasets where it is the worst). Solid lines correspond to runs with the GB5 oracle and dashed lines to the runs with the Linear oracle.

The aggregate performance with the GB5 oracle across all datasets can be briefly summarized as follows: RegCB always beats $\epsilon$-Greedy and ILTCB, but sometimes loses out to Bootstrap, and Bootstrap itself sometimes underperforms relative to the other baselines, especially on the UCI datasets. Even when RegCB is not the best, it is almost always within $20\%$ of the best. The elimination and optimistic variants of RegCB have comparable performance, with elimination performing slightly better in aggregate.

The RegCB algorithms with the GB5 oracle also dominate the $\epsilon$-Greedy, ILTCB, and Bootstrap baselines when they are equipped with Linear oracles (the dashed lines in Figure 2). When the RegCB algorithms use the Linear oracle they also dominate the baselines with the Linear oracle across all datasets, including Bootstrap. This suggests that the gap between RegCB and Bootstrap for the GB5 oracle may be due to the approximation we make by only considering a fixed ensemble structure in each epoch, as we noted earlier.777The aggregate plots for RegCB with the Linear oracle can be found in Appendix B along with additional aggregate plots.

Results: Confidence Width

The analysis of RegCB relies on distributional assumptions on $D$ (disagreement coefficient or moment parameters) that are not necessarily easy to verify. Note that the main role of these parameters is to control the rate at which confidence width $W_{\mathcal{F}_{m}}(x,a)=\text{High}_{\mathcal{F}_{m}}(x_{t},a)-\text{Low}_{\mathcal{F}_{m}}(x_{t},a)$ used in RegCB shrinks, since the small widths imply that the algorithm makes good decisions and thus has low regret.

To investigate whether the width $W_{\mathcal{F}_{m}}$ indeed shrinks empirically, we compute it on each dataset for Algorithm 2. We also compute an analogous width parameter for Bootstrap (see Appendix B). Finally for both Algorithm 2 and Bootstrap we compute the size of the “disagreement set” $A_{t}$, defined in Algorithm 1, which measures how many actions the algorithm thinks are plausibly best.888This set is well-defined for both RegCB-Opt and Bootstrap even through neither algorithm instantiates it explicitly. For the yahoo and mslr datasets this $\left\lvert A_{t}\right\rvert$ is technically a lower bound on the true disagreement set size $\left\lvert A_{\mathcal{F}_{m}}(x_{t})\right\rvert$ because our classes $\mathcal{F}$ do not have product structure on these datasets—see discussion in Section 4.1.

Figure 3 shows width and disagreement for a representative sample of datasets under the GB5 oracle; the remaining datasets are in Appendix B. The figure suggests that our distributional assumptions are reasonable for real-world datasets. In particular, for our algorithm, the width decays roughly as $T^{-1/3}$ for letter and $T^{-1/2}$ for letter+N and yahoo. Interestingly, the best hyper-parameter setting for Bootstrap on letter yields low but essentially constant (i.e., not shrinking) width, which in our experiments is associated with the poor validation reward. This suggests that while the Bootstrap confidence intervals are small, they may not be faithful in the sense of containing $f^{\star}(x,a)$.

6 Conclusion and Discussion

This work serves as a starting point for what we hope will be a fruitful line of research on oracle-efficient contextual bandit algorithms in realizability-based settings. We have shown that the RegCB family of algorithms have strong empirical performance and enjoy nice theoretical properties. These results suggest some compelling directions for future work:

• •

Is there a regression oracle–based algorithm that achieves the optimal $\widetilde{O}(\sqrt{KT\log\left\lvert\mathcal{F}\right\rvert})$ regret? For example, can the regressor elimination strategy of Agarwal et al. (2012) be oraclized?

• •

Given the competitive empirical performance of Bootstrap, are there reasonable distributional assumptions similar to those in Section 4 under which it can be analyzed? There is some recent work in this direction for the special case of linear models (Lu & Van Roy, 2017).

• •

Randomizing uniformly or putting all the mass on the optimistic choice are two extreme cases of choosing amongst the plausibly optimal actions. Are there better randomization schemes amongst these actions that lead to stronger regret guarantees?

Appendix A Proofs

A.1 Proofs from Section 3.1

We prove the statement of Theorem 1 for BinSearch.Unbounded.High (Algorithm 4), which does not require the predictors in $\mathcal{F}$ to be bounded in $[0,1]$. Note however that the actual rewards are still always bounded in $[0,1]$, so that $f^{\star}(x,a)$ is always bounded by the realizability assumption. Compared with Algorithm 3, the algorithm includes some handling of special cases, which are automatically excluded in Algorithm 3 by the assumption about boundedness. The performance guarantee for BinSearch.Unbounded.Low (Algorithm 5) is analogous and therefore is omitted.

Lemma 1**.**

Let $\mathcal{F}$ be convex and closed under pointwise convergence. Consider a run of Algorithm 4. Let $R(f)$ and $R_{\text{min}}$ be defined as in Algorithm 4 and let

[TABLE]

Then Algorithm 4 returns $z$ such that \bigl{\lvert}z-\min\{z^{\star},1\}\bigr{\rvert}\leq\alpha after at most O\bigl{(}\log(1/\alpha)+\log(\max\left\{2-z_{0},1\right\})\bigr{)} iterations, where $z_{0}=f_{\text{min}}(x,a)$ and $f_{\text{min}}=\operatorname*{arg\,min}_{f}R(f)$.

Corollary 1**.**

If $f(x,a)\in[0,1]$ for all $f\in\mathcal{F}$, $x\in\mathcal{X}$ and $a\in\mathcal{A}$, then Algorithm 4 returns $z$ such that $\left\lvert z-z^{\star}\right\rvert\leq{}\alpha$ after at most $O\left(\log(1/\alpha)\right)$ iterations.

Proof.

The proof works by analyzing a univariate auxiliary function $\phi:\mathbb{R}\to\mathbb{R}\cup\{\infty\}$, which maps $z\in\mathbb{R}$ to the smallest empirical error $R(f)$ among all functions that predict $f(x,a)=z$,

[TABLE]

note that we do not need to worry about the case when $f(x,a)$ might take values both larger and smaller than $z$ but not $z$ exactly due to the assumed convexity of $\mathcal{F}$. We first show that this function is well-defined (i.e., the minimum in the definition is attained), convex and lower semicontinuous. We begin by embedding the least-squares optimization in a finite dimensional space. Let $H=\{(x_{i},a_{i},r_{i})\}_{i=1}^{n}$ and define $x_{n+1}\coloneqq x$ and $a_{n+1}\coloneqq a$. We associate each $f$ with a vector $\mathbf{v}^{f}\in\mathbb{R}^{n+1}$ with entries $v^{f}_{i}=f(x_{i},a_{i})$. Let $\mathcal{V}\coloneqq\{\mathbf{v}^{f}:\>f\in\mathcal{F}\}$. Since $\mathcal{F}$ is closed under pointwise convergence and convex, the set $\mathcal{V}$ must also be closed and convex.

For $\mathbf{v}\in\mathbb{R}^{n+1}$, let

[TABLE]

where $r_{i}$ are the rewards from $H$. Thus,

[TABLE]

and therefore

[TABLE]

where we use the convention that the minimum of an empty set equals $\infty$. The attainment of the minimum now follows by convexity and continuity of $\rho$ along the affine space $\{v_{n+1}=z\}$. The convexity and lower semicontinuity of $\phi$ follows by Theorem 9.2 of Rockafellar (1970).

The upper confidence value $z^{\star}$ is then the largest $z$ for which $\phi(z)\leq R_{\text{min}}+\beta$:

[TABLE]

Furthermore, for any $w\geq 0$, define

[TABLE]

Thus, $z_{w}=f(x,a)$ where $f=\operatorname*{arg\,min}_{\tilde{f}\in\mathcal{F}}\widetilde{R}(\tilde{f},w)$ with $\widetilde{R}$ as defined in the algorithm. The algorithm maintains the identities $z_{\text{L}}=z_{w_{\text{L}}}$ and $z_{\text{H}}=z_{w_{\text{H}}}$, so it can be rewritten as follows:

1:if $z_{0}\geq 1$ or $\phi(z_{0})=\phi(z_{\beta/\alpha})$ then return $1$

2:$w_{\text{L}}\leftarrow 0,\;w_{\text{H}}\leftarrow\beta/\alpha,\;\Delta\leftarrow\alpha\beta/(2-z_{0})^{3}$

3:while $\lvert z_{w_{\text{H}}}-z_{w_{\text{L}}}\rvert>\alpha$ and $\lvert w_{\text{H}}-w_{\text{L}}\rvert>\Delta$ do

4:     $w\leftarrow(w_{\text{H}}+w_{\text{L}})/2$

5:     if $\phi(z_{w})>\phi(z_{0})+\beta$ then

6:         $w_{\text{H}}\leftarrow w$

7:     else

8:         $w_{\text{L}}\leftarrow w$

9:     end if

10:end while

11:return $\min\{z_{w_{\text{H}}},1\}$.

Note that $z_{0}=f_{\text{min}}(x,a)$ where $f_{\text{min}}$ is the minimizer of $R$, and therefore $\phi$ attains its minimum at $z_{0}$. If $z_{0}\geq 1$, then the algorithm terminates and returns $1$. Since $z^{\star}\geq z_{0}\geq 1$, in this case the lemma holds.

Also, note that if $z^{\star}=z_{0}<1$ then the algorithm immediately terminates with $z_{w_{\text{L}}}=z_{w_{\text{H}}}=z_{0}$. This is because of the fact that $z^{\star}=z_{0}$, given $\beta>0$, implies by lower semicontinuity that $\phi(z)=\infty$ for all $z>z_{0}$ and thus $z_{w}=z_{0}$ for all $w>0$.

The final special case to consider is when $\phi(2)=\phi(z_{0})$, i.e., there exist a minimizer $\tilde{f}_{\min}$ of $R$, which satisfies $\tilde{f}_{\min}(x,a)=2$ and thus for any $w$, it also minimizes $\tilde{R}(f,w)$. This is exactly the case when $R(f_{\text{L}})=R(f_{\text{H}})$ in Algorithm 4 and in this case the algorithm returns $1$ and the lemma holds.

In the remainder of the proof we assume that $\phi(2)>\phi(z_{0})$, $z_{0}<1$ and $z_{0}<z^{\star}$. By convexity of $\phi$, we know that $\phi$ is non-decreasing on $[z_{0},\infty)$, and we will argue that by performing the binary search over $w$, the algorithm is also performing a binary search over $z_{w}$ to find the point $z^{\star}$.

We begin by characterizing $z_{w}$ and showing that $z_{w}<2$ for all $w$. For any $w>0$, by first-order optimality,

[TABLE]

for some $\phi^{\prime}(z_{w})\in\partial\phi(z_{w})$, where $\partial\phi$ denotes the subdifferential. First, note that $z_{w}\geq z_{0}$, because at any $z<z_{0}\leq 1$, we have $w(2-z)>0$ while also $\phi^{\prime}(z)\leq 0$, because $\phi$ is convex and minimized at $z_{0}$. Therefore, at $z<z_{0}$, we have $\phi^{\prime}(z)-w(2-z)<0$, so Eq. (4) can only be satisfied by $z_{w}\geq z_{0}$. Rearranging, we obtain

[TABLE]

Since $z_{w}\geq z_{0}$, the convexity of $\phi$ implies that $\phi^{\prime}(z_{w})\geq 0$. Since $w>0$, we therefore must in fact have $\phi^{\prime}(z_{w})>0$ and

[TABLE]

Eq. (5) now implies that $z_{w}$ is non-decreasing as a function of $w$.

Let $w^{\star}$ be such that $z_{w^{\star}}=z^{\star}$ (this can be obtained by Eq. 5). The remainder of the proof proceeds in two steps. The first step establishes that our initial setting $w_{\text{H}}=\beta/\alpha$ is large enough to guarantee that the initial interval $[z_{w_{\text{L}}},z_{w_{\text{H}}}+\alpha]=\left[z_{0},z_{\beta/\alpha}+\alpha\right]$ contains the solution $\min\{z^{\star},1\}$. The execution of the algorithm then continues to maintain this condition, i.e., $\min\{z^{\star},1\}\in[z_{w_{\text{L}}},z_{w_{\text{H}}}+\alpha]$, which we refer to as the invariant, while halving $\lvert w_{\text{H}}-w_{\text{L}}\rvert$. That the invariant holds can be seen as follows: First, if $z_{0}\leq{}z^{\star}\leq{}z_{\beta/\alpha}$, then the update rule guarantees that $z_{w_{\text{L}}}\leq{}z^{\star}\leq{}z_{w_{\text{H}}}$ for every iteration. On the other hand, if $z^{\star}>z_{\beta/\alpha}$, then $z_{w_{\text{H}}}=z_{\beta/\alpha}$ for every iteration, and so Step 1 below guarantees that $z^{\star}\in\left[z_{\beta/\alpha},z_{\beta/\alpha}+\alpha\right]\supseteq\left[z_{w_{\text{L}}},z_{w_{\text{H}}}+\alpha\right]$.

The algorithm terminates after at most

[TABLE]

iterations. If the reason for termination is that $\lvert z_{\text{H}}-z_{\text{L}}\rvert\leq\alpha$ then the lemma follows, thanks to the invariant. Otherwise, we must have $\lvert w_{\text{H}}-w_{\text{L}}\rvert\leq\Delta$, so our invariant together with the monotonicity of $z_{w}$ in $w$ implies that $w_{\text{H}}\leq w^{\star}+\Delta$. Our second step below establishes that in this case we must also have $z_{\text{H}}\leq z^{\star}+\alpha$. Our invariant separately also implies that $\min\{z^{\star},1\}\leq z_{\text{H}}+\alpha$, so altogether we have $\min\{z_{\text{H}},1\}-\alpha\leq\min\{z^{\star},1\}\leq\min\{z_{\text{H}},1\}+\alpha$, proving the lemma. It remains to prove the two steps.

Step 1: $z_{0}\leq\min\{z^{\star},1\}\leq z_{\beta/\alpha}+\alpha$.

The first inequality is immediate from the definition of $z^{\star}$ and the fact that $z_{0}<1$. The second inequality holds if $z_{\beta/\alpha}\geq 1$, so it remains to consider $z_{\beta/\alpha}\leq 1$. Let $w=\beta/\alpha$. Then by Eq. (5),

[TABLE]

where the last step follows because $z_{w}\leq 1$. Now by convexity of $\phi$, for any $\tilde{\alpha}>\alpha$

[TABLE]

where the last step follows because $\phi(z_{w})\geq\phi(z_{0})$ and $\tilde{\alpha}>\alpha$. This shows that $z^{\star}\leq z_{w}+\alpha$ and completes Step 1.

Step 2: $z_{w^{\star}+\Delta}\leq z^{\star}+\alpha$.

Let $w=w^{\star}+\Delta$. Then by convexity

[TABLE]

and since $z^{\star}>z_{0}$, we can rearrange this inequality to give

[TABLE]

where the last inequality follows by Eq. (6). By Eq. (5), we also have

[TABLE]

because $z^{\star}>z_{0}$. Combining the two bounds yields

[TABLE]

Applying now Eq. (5) twice, and also using the monotonicity of $\phi^{\prime}$, we obtain

[TABLE]

Therefore,

[TABLE]

Rearranging,

[TABLE]

where the final inequality uses the fact that $w\geq w^{\star}$ and $z^{\star}\geq z_{0}$. Finally, applying the bound (7) and the definition of $\Delta$, we complete Step 2:

[TABLE]

A.2 Proof of Proposition 1

Proof of Proposition 1.

Consider the following contextual bandit instance:

• •

Two actions $a_{g}$ and $a_{b}$, so $K=2$.

• •

$r_{t}(a_{g})=1-\epsilon$ and $r_{t}(a_{b})=0$, regardless of context (there is no noise).

• •

$N$ contexts $x^{1},\ldots,x^{N}$. The context distribution $D_{\mathcal{X}}$ is uniform over these $N$ contexts.

• •

Regressor class $\mathcal{F}$ contains the following $N+1$ predictors:

• –

Ground truth regressor $f^{\star}$ defined by $f^{\star}(x,a_{g})=1-\epsilon,\;\forall{}x$ and $f^{\star}(x,a_{b})=0,\;\forall{}x$.

• –

For each $i\in[N]$, $f_{i}$ satisfying $f_{i}(x^{i},a_{g})=0$, $f_{i}(x^{i},a_{b})=1$, and $f_{i}(x^{j},a_{g})=1-\epsilon$, $f_{i}(x^{j},a_{b})=0$ for all $j\neq{}i$.

We can see that $\pi_{f^{\star}}$ has population reward $1-\epsilon$ and each $\pi_{f_{i}}$ has population reward $(1-1/N)(1-\epsilon)$. Thus, each $f_{i}$ has expected regret of $(1-\epsilon)/N$.

Suppose $S$ is the set of contexts that have been observed by our algorithms at time $t$, and further assume $\beta_{m}=0$ (as it will be clear that larger $\beta_{m}$ can only make things worse), so that only regressors with zero square loss are considered. Observe that $f^{\star}\in\mathcal{F}_{m}$ and $f_{i}\in\mathcal{F}_{m}$ only if $x^{i}\notin S$.

Let $x^{i}$ be the context observed at time $t$. If $x^{i}\in S$, then all regressors in $\mathcal{F}_{m}$ agree on it, so $a_{g}$ will be played. Now, suppose $x^{i}\notin S$. Then we have $\text{High}_{\mathcal{F}_{m}}(x^{i},a_{g})=1-\epsilon$ (obtained by $f^{\star}$), and $\text{Low}_{\mathcal{F}_{m}}(x^{i},a_{g})=0$ (obtained by $f_{i}$). Likewise, $\text{High}_{\mathcal{F}_{m}}(x^{i},a_{b})=1$ (from $f_{i}$) and $\text{Low}_{\mathcal{F}_{m}}(x^{i},a_{b})=0$ (from $f^{\star}$).

We thus see that our algorithms will make a mistake and incur instantaneous regret of $(1-\epsilon)$ precisely at the time steps for which one of the $N$ contexts is encountered for the first time. The regret of the algorithm after $t$ steps can therefore be lower bounded as $\min\left\{N,\widetilde{\Omega}(t)\right\}$. ∎

A.3 Proofs from Section 4.1

We recall our earlier definition of the disagreement coefficient for the reader’s convenience.

Definition** (Disagreement Coefficient).**

The disagreement coefficient for $\mathcal{F}$ (with respect to $D_{\mathcal{X}}$) is defined as

[TABLE]

In addition, the following condition on $f^{\star}$ is important to obtain fast rates, but it is not stated as an assumption because it is not strictly necessary for any of our algorithms.

Definition 8** (Massart noise condition).**

The distribution $D$ satisfies the Massart noise condition if there exists $\gamma>0$, called a margin, such that

[TABLE]

For all subsequent analyses we will use the following filtration:

[TABLE]

Let $\operatorname{\mathbb{E}}_{t}\left[\cdot\right]\vcentcolon=\operatorname{\mathbb{E}}\left[\cdot\mid{}\mathcal{J}_{t}\right]$ and $\mathrm{Var}_{t}\left[\cdot\right]\vcentcolon=\mathrm{Var}\left[\cdot\mid{}\mathcal{J}_{t}\right]$.

Lemma 2** (Freedman-type inequality e.g. (Agarwal et al., 2014)).**

For any real-valued martingale difference sequence $(Z_{t})_{t\leq{}T}$ with $\lvert Z_{t}\rvert\leq{}R$ almost surely, it holds that with probability at least $1-\delta$,

[TABLE]

for all $\eta\in\left[0,1/R\right]$.

Recall that epoch schedule used by Algorithm 1 and Algorithm 2 is $\tau_{m}=2^{m-1}$. Denote the length of epoch $m$ by $T_{m}=\tau_{m+1}-\tau_{m}=2^{m-1}$. In addition, we will use the notation $g^{\star}_{a}(x)\vcentcolon=f^{\star}(x,a)$ where $f^{\star}$ as in the main text is the predictor that realizes the mean reward function, and also

[TABLE]

for any $g:\mathcal{X}\to\left[0,1\right]$, and action $a\in\mathcal{A}$. When $f\in\mathcal{F}=\mathcal{G}^{\mathcal{A}}$ we will overload this notation by writing $M_{t}(f,a)\vcentcolon={}M_{t}(f(\cdot,a),a)$. Also define the class

[TABLE]

To prove the theorem, we make use of following lemmas.

Lemma 3**.**

For any $g:\mathcal{X}\to\left[0,1\right]$ and $a\in\mathcal{A}$ we have

[TABLE]

Proof.

Note that $a_{t}$ and $r_{t}$ are conditionally independent given $x_{t}$ and also $\operatorname{\mathbb{E}}_{r_{t}}[r_{t}(a)\mid{}x_{t}]=g_{a}^{\star}(x_{t})$. We thus have

[TABLE]

Similarly, since $((g(x_{t})+g_{a}^{\star}(x_{t})-2r_{t}(a))^{2}\leq 4$ we have

[TABLE]

∎

Definition 9** (Covering number).**

For a class $\mathcal{G}^{\prime}\subseteq{}\left\{g:\mathcal{X}\to\left[0,1\right]\right\}$, an empirical $L_{p}$-cover on a sequence $x_{1},\ldots,x_{T}$ at scale $\varepsilon$ is a set $V\subseteq{}\mathbb{R}^{T}$ such that

[TABLE]

We define the covering number $\mathcal{N}_{p}(\mathcal{G}^{\prime},\varepsilon,x_{1:T})$ to be the size of the smallest such cover.

Lemma 4**.**

For any fixed class $\mathcal{G}^{\prime}\subseteq{}\left\{g:\mathcal{X}\to\left[0,1\right]\right\}$ and fixed $a\in\mathcal{A}$, with probability at least $1-\delta$, it holds that

[TABLE]

for all $\tau_{1}\leq{}\tau_{2}$ and $g\in\mathcal{G}^{\prime}$ when $\mathcal{G}^{\prime}$ is finite and

[TABLE]

for all $\tau_{1}\leq{}\tau_{2}$ and $g\in\mathcal{G}^{\prime}$ in the general case.

Remark 2**.**

Equation (10) implies (9), but with weaker constants.

Corollary 2**.**

Define

[TABLE]

With probability at least $1-\delta/2$, it holds that

[TABLE]

for all $g\in\mathcal{G}$, $a\in\mathcal{A}$, and $\tau_{1},\tau_{2}\in\left[T\right]$.

Proof of Lemma 4.

We first prove the inequality in the finite class case.

For any fixed $g\in\mathcal{G}^{\prime}$, $a\in\mathcal{A}$, and $\tau_{1},\tau_{2}\in\left[T\right]$, since $Z_{t}=\operatorname{\mathbb{E}}_{t}\left[M_{t}(g,a)\right]-M_{t}(g,a)$ forms a martingale different sequence with $|Z_{t}|\leq 1$, applying Lemma 2 and Lemma 3 we have with probability $1-\delta$,

[TABLE]

This implies

[TABLE]

after setting $\eta=1/8$ and rearranging. Finally, we apply a union bound over all $g\in\mathcal{G}^{\prime}$ and $\tau\leq{}\tau_{2}\in\left[T\right]$ to get the result.

For the infinite class case, we appeal to Theorem 9 of (Krishnamurthy et al., 2017) (see page 36 specifically — we do not use the final theorem statement but rather an intermediate result that is the consequence of their Lemmas 7, 8, 9, and 10).

Let $\tau_{1}$ and $\tau_{2}$ be fixed. Then the result of (Krishnamurthy et al., 2017) implies that for any class $\mathcal{G}$, any fixed $\varepsilon>0$, $\nu>0$ and $a\in\mathcal{A}$, letting $c=1/8$,

[TABLE]

Rearranging, this implies that with probability at least $1-\delta$,

[TABLE]

Now consider a grid $\varepsilon_{i}\vcentcolon={}e^{i}/T$ for $i\in[\log(T)]$. By union bound, (12) implies that with probability at least $1-\delta$,

[TABLE]

This implies that with probability at least $1-\delta$,

[TABLE]

To see that this inequality is implied by the preceeding inequality, first observe that the infimum over $\varepsilon$ above may be restricted to $\left[1/T,1\right]$ without loss of generality. This holds because $M_{t}$ lies in $\left[-1,1\right]$ and $\mathcal{N}_{1}(\mathcal{G},1,x_{1:T})\leq{}1$, which both follow from the fact that the range of $\mathcal{G}$ lies in $\left[0,1\right]$. Now let $\varepsilon^{\star}$ obtain the infimum and let $i^{\star}=\min\left\{i\mid{}\varepsilon_{i}\geq{}\varepsilon^{\star}\right\}$. Then $\mathcal{N}_{1}(\mathcal{G},\varepsilon_{i^{\star}},x_{1:T})\leq{}\mathcal{N}_{1}(\mathcal{G},\varepsilon^{\star},x_{1:T})$ and $18\varepsilon_{i^{\star}}T\leq{}18e\varepsilon^{\star}T\leq{}50\varepsilon^{\star}T$.

To conclude, we take a union bound over all $\tau_{1}<\tau_{2}\in\left[T\right]$. ∎

Lemma 5**.**

Conditioned on the event of Corollary 2, it holds that

$g^{\star}_{a}\in\widehat{\mathcal{G}}_{m}\left(\frac{{C_{\delta}}}{2(\tau_{m}-1)},a\right)$ for all $m\in[M]$ and $a\in\mathcal{A}$. 2. 2.

For all $\beta\geq{}0$, $m\in[M]$, and $a\in\mathcal{A}$,

[TABLE] 3. 3.

For all $\beta\geq{}0$, $m\in[M]$, $k\in[m]$, and $a\in\mathcal{A}$,

[TABLE] 4. 4.

With $\beta_{m}=\frac{(M-m+1){C_{\delta}}}{\tau_{m}-1}$, we have for any $m\in[M]$, $f^{\star}\in\mathcal{F}_{m}$ and also $\mathcal{F}_{m}\subseteq{}\mathcal{F}_{m-1}\subseteq{}\cdots\subseteq{}\mathcal{F}_{1}$.

Proof.

Each claim in the lemma statement will be handled separately. 

First claim. From (11) and nonnegativity of $\operatorname{\mathbb{E}}_{t}\left[M_{t}(g,a)\right]$, we have that

[TABLE]

Expanding out $M_{t}(g,a)$ and rearranging, this gives $\widehat{\mathcal{R}}_{m}(g^{\star}_{a},a)-\min_{g\in\mathcal{G}}\widehat{\mathcal{R}}_{m}(g,a)\leq\frac{{C_{\delta}}}{2(\tau_{m}-1)}$, which implies $g^{\star}_{a}\in\widehat{\mathcal{G}}_{m}\left(\frac{{C_{\delta}}}{2(\tau_{m}-1)},a\right)$.

Second claim. For any $g\in\widehat{\mathcal{G}}_{m}(\beta,a)$, we have by definition

[TABLE]

Therefore applying (11) leads to

[TABLE]

which implies $g\in\widetilde{\mathcal{G}}_{m}\left(2\beta+\frac{{C_{\delta}}}{\tau_{m}-1},a\right)$.

Third claim. For any $g\in\widehat{\mathcal{G}}_{m}(\beta,a)$, we have for any $k\in[m]$,

[TABLE]

which implies $g\in\widehat{\mathcal{G}}_{k}\left(\frac{\tau_{m}-1}{\tau_{k}-1}\beta+\frac{{C_{\delta}}}{\tau_{k}-1},a\right)$.

Fourth claim. The value of $\beta_{m}$ ensures that $\frac{{C_{\delta}}}{2(\tau_{m}-1)}\leq\beta_{m}$ for any $m\in[M]$, and also for any $k<m$,

[TABLE]

Therefore by the first and the third statement we have the claimed conclusions. ∎

Proposition 3**.**

For any two classes $\mathcal{F},\mathcal{F}^{\prime}$ and any context $x$, $A_{\mathcal{F}}(x)\subseteq{}A_{\mathcal{F}^{\prime}}(x)$.

Lemma 6**.**

Algorithm 1 with Option I ensures that for any $m\in[M]$ and $t\in\{\tau_{m},\ldots,\tau_{m+1}-1\}$,

[TABLE]

Proof.

For any $f\in\mathcal{F}_{m}$ and any $a\in\operatorname*{arg\,max}_{a^{\prime}\in\mathcal{A}}f(x_{t},a^{\prime})$, we have by definitions

[TABLE]

which implies $a\in A_{t}$. On the other hand, for each $a\in A_{t}$, there exists $g_{a}\in\widehat{\mathcal{G}}(\beta_{m},a)$ such that $g_{a}(x_{t})\geq\max_{a^{\prime}}\min_{g\in\widehat{\mathcal{G}}(\beta_{m},a^{\prime})}g(x_{t})$, which further implies that for any $a^{\prime}\neq a$, there exists $g_{a^{\prime}}\in\widehat{\mathcal{G}}(\beta_{m},a^{\prime})$ such that $g_{a}(x_{t})\geq g_{a^{\prime}}(x_{t})$. Therefore, we can construct an $f$ so that $f(\cdot,a)=g_{a}(\cdot)$ and $f(\cdot,a^{\prime})=g_{a^{\prime}}(\cdot)$ for all $a^{\prime}\neq a$, so that clearly $f\in\mathcal{F}_{m}$ and $a\in\operatorname*{arg\,max}_{a^{\prime}\in\mathcal{A}}f(x_{t},a^{\prime})$. This proves the lemma. ∎

Lemma 7**.**

Conditioned on the event of Corollary 2, Algorithm 1 with Option I and $\beta_{m}=\frac{(M-m+1){C_{\delta}}}{\tau_{m}-1}$ ensures that for any $m\in[M]$, we have $\mathcal{F}_{m}\subseteq{}\mathcal{F}(\varepsilon_{m})$ with

[TABLE]

where $P_{\eta}=\operatorname{Pr}_{x}\left(f^{\star}(x,\pi^{\star}(x))-\max_{a\neq{}\pi^{\star}(x)}f^{\star}(x,a)<\eta\right)$.

Proof.

We first prove that for any $t<\tau_{m}$ and $f\in\mathcal{F}_{m}$, the following holds

[TABLE]

Indeed, note that for any $\eta>0$, with realizability we have

[TABLE]

By the definition of $\pi_{f}$ we also have for any $x$, $f(x,\pi_{f}(x))-f(x,\pi^{\star}(x))\geq{}0$ and thus

[TABLE]

Now suppose round $t$ is in epoch $k$. Since both $f\in\mathcal{F}_{m}\subseteq\mathcal{F}_{k}$ and $f^{\star}\in\mathcal{F}_{k}$, we have $\pi_{f}(x_{t}),\pi^{\star}(x_{t})\in{}A_{t}$ by Lemma 6. Therefore, the fact that $a_{t}$ is drawn uniformly from $A_{t}$ implies

[TABLE]

and likewise

[TABLE]

Lastly, plugging the equality

[TABLE]

proves Eq. (14). Averaging over $t=1,\ldots,\tau_{m}-1$ then gives

[TABLE]

Using the second statement of Lemma 5 we have $\sum_{t=1}^{\tau_{m}-1}\operatorname{\mathbb{E}}_{t}\left[M_{t}(f,a)\right]\leq 2(\tau_{m}-1)\beta_{m}+{C_{\delta}}=(2M-2m+3){C_{\delta}}$ and thus

[TABLE]

completing the proof by the definition of $\mathcal{F}(\varepsilon_{m})$. ∎

We are now ready to prove Theorem 2, which is restated below with an extra result under the Massart condition.

Theorem 5** (Full version of Theorem 2).**

With $\beta_{m}=\frac{(M-m+1){C_{\delta}}}{\tau_{m}-1}$ and ${C_{\delta}}$ as in Corollary 2, Algorithm 1 with Option I ensures that with probability at least $1-\delta$,

[TABLE]

*In particular, for finite classes regret is bounded as $\widetilde{O}\left(T^{\frac{3}{4}}\left(\log\left\lvert\mathcal{G}\right\rvert\right)^{\frac{1}{4}}\sqrt{\theta_{0}K}\right)$.

Furthermore, if the Massart noise condition (Definition 8) is satisfied with parameter $\gamma$, then Algorithm 1 configured as above with $\delta=1/T$ enjoys an in-expectation regret bound of*

[TABLE]

which for finite classes is upper bounded by $\widetilde{O}\left(\frac{\theta_{0}K^{2}\log\left(\left\lvert\mathcal{G}\right\rvert T\right)}{\gamma^{2}}\right)$.

Remark 3**.**

This theorem and the subsequent regret bounds based on moment conditions (Theorem 6 and Theorem 7) give a high-probability empirical regret bound in the general case, but only give an in-expectation regret bound under the Massart condition. This is because one incurs an extra $O(\sqrt{T})$ factor in going from a (conditional) expected regret bound to an empirical regret bound, which is a low order term in the general case but may dominate in the Massart case.

Proof.

We will first provide a bound on

[TABLE]

then relate this quantity to the left-hand-side of (15) and (16) at the end.

This proof conditions on the above event and the events of Corollary 2, which happen with probability at least $1-\delta/2$, and bounds the conditional expected regret terms $\operatorname{\mathbb{E}}_{t}\left[f^{\star}(x_{t},\pi^{\star}(x_{t}))-f^{\star}(x_{t},a_{t})\right]$ individually.

For any $\eta^{\prime}>0$, we recall the definition used in the proof of Lemma 7: $P_{\eta^{\prime}}=\operatorname{Pr}_{x}\left(f^{\star}(x,\pi^{\star}(x))-\max_{a\neq{}\pi^{\star}(x)}f^{\star}(x,a)<\eta^{\prime}\right)$. Further define two events:

[TABLE]

We then have

[TABLE]

Next we argue two facts (suppose round $t$ is in epoch $m$): $E_{1}$ implies $x_{t}\in\mathrm{Dis}(\mathcal{F}_{m})$, and $E_{2}$ implies that there exists $a^{\prime}\in A_{t}$ such that $W_{\mathcal{F}_{m}}(x,a^{\prime})>\eta^{\prime}/2$. Indeed, with $a$ being the action stated in event $E_{1}$, we know that by Lemma 6 there exists $f\in\mathcal{F}_{m}$ such that $a\in\operatorname*{arg\,max}_{a^{\prime}}f(x_{t},a)$. However, clearly $a$ is not in $\operatorname*{arg\,max}_{a^{\prime}}f^{\star}(x_{t},a)$, and thus by $f^{\star}\in\mathcal{F}_{m}$ and the definition of disagreement region we have $x_{t}\in\mathrm{Dis}(\mathcal{F}_{m})$. On the other hand, with $a$ being the action stated in event $E_{2}$, we have

[TABLE]

where the last inequality is by the fact $a\in A_{t}$ and the definition of $A_{t}$. The last inequality thus implies that there exists $a^{\prime}\in A_{t}$ such that $W_{\mathcal{F}_{m}}(x,a^{\prime})>\eta^{\prime}/2$. We therefore continue with

[TABLE]

Combining everything we arrive at for any $\eta,\eta^{\prime}>0$,

[TABLE]

In the general case we simply bound $P_{\eta}$ and $P_{\eta^{\prime}}$ by $1$ and choose the optimal $\eta$ and $\eta^{\prime}$ to arrive at a regret bound of order $O\left(T^{\frac{3}{4}}{C_{\delta}}^{\frac{1}{4}}\sqrt{\theta_{0}K\log T}+\log(1/\delta)\right)$. On the other hand, under the Massart condition (Definition 8) one can pick $\eta=\eta^{\prime}=\gamma$ so that $P_{\eta}=P_{\eta^{\prime}}=0$ and obtain a regret bound of order $O\left(\frac{\theta_{0}K^{2}{C_{\delta}}\log^{2}T}{\gamma^{2}}\right)$.

Lastly, we relate the sum of conditional expected instantaneous regrets to the left-hand side of (15) and (16). In the general case, since instantaneous regret lies in $\left[-1,1\right]$, Azuma-Hoeffding implies that

[TABLE]

with probability at least $1-\delta/2$. By union bound, the theorem statement holds with probability at least $1-\delta$.

In the Massart case, the law of total expectation implies

[TABLE]

where the second term uses boundedness of regret along with the fact that the events of Corollary 2 hold with probability at least $1-1/T$. ∎

A.4 Proofs from Section 4.2

Similarly to the notation $M_{t}(g,a)$ for the case $\mathcal{F}=\mathcal{G}^{\mathcal{A}}$, for a general predictor class $\mathcal{F}$ we define for any $f\in\mathcal{F}$

[TABLE]

and also the class

[TABLE]

Finally, for any $a\in\mathcal{A}$ we define a class

[TABLE]

We establish several lemmas similar to those in Appendix A.3.

Lemma 8**.**

For any $f\in\mathcal{F}$ we have

[TABLE]

Lemma 9**.**

Define

[TABLE]

With probability at least $1-\delta/2$, it holds that

[TABLE]

for all $f\in\mathcal{F}$ and $\tau_{1},\tau_{2}\in\left[T\right]$.

Proof of Lemma 9.

We first prove the inequality in the finite class case. For any fixed $f\in\mathcal{F}$, and $\tau_{1}\leq\tau_{2}\in\left[T\right]$, $Z_{t}\vcentcolon=\operatorname{\mathbb{E}}_{t}\left[M_{t}(f)\right]-M_{t}(f)$ forms a martingale different sequence with $|Z_{t}|\leq 1$. Applying Lemma 2 and Lemma 8 we have with probability $1-\delta$,

[TABLE]

which implies after setting $\eta=1/8$ and rearranging.

[TABLE]

We apply a union bound over all $f\in\mathcal{F}$ and $\tau_{1}\leq{}\tau_{2}\in\left[T\right]$ to get the result.

To handle the infinite class case we invoke Lemma 4. In particular, for any fixed $a$, the lemma with $\mathcal{G}^{\prime}=\mathcal{F}|_{a}$ implies that with probability at least $1-\delta$,

[TABLE]

for all $f\in\mathcal{F}$ and $\tau_{1}\leq{}\tau_{2}$. Observe that $M_{t}(f)=\sum_{a\in\mathcal{A}}M_{t}(f(\cdot,a),a)$. Taking a union bound and then summing over all actions, the preceding statement therefore implies that with probability at least $1-\delta$,

[TABLE]

for all $f\in\mathcal{F}$ and $\tau_{1}\leq{}\tau_{2}$. The final result follows from superadditivity of the infimum. ∎

Lemma 10**.**

Conditioned on the event of Lemma 9, it holds that

$f^{\star}\in\widehat{\mathcal{F}}_{m}\left(\frac{{C_{\delta}^{\prime}}}{2(\tau_{m}-1)}\right)$ for all $m\in[M]$. 2. 2.

For all $\beta\geq{}0$ and $m\in[M]$,

[TABLE]

Consequently, we have $\operatorname{\mathbb{E}}_{\tau_{m-1}}\left[M_{\tau_{m-1}}(f)\right]\leq\frac{2\beta(\tau_{m}-1)+{C_{\delta}^{\prime}}}{T_{m-1}}$ for any $f\in\widehat{\mathcal{F}}_{m}(\beta)$. 3. 3.

For all $\beta\geq{}0$, $m\in[M]$, and $k\in[m]$,

[TABLE] 4. 4.

With $\beta_{m}=\frac{(M-m+1){C_{\delta}}}{\tau_{m}-1}$, we have for any $m\in[M]$, $f^{\star}\in\mathcal{F}_{m}$ and also $\mathcal{F}_{m}\subseteq{}\mathcal{F}_{m-1}\subseteq{}\cdots\subseteq{}\mathcal{F}_{1}$.

Proof.

The proof of this lemma is essentially the same as that of Lemma 5 in Appendix A.3. The only new statement is the second statement of the second claim in Lemma 10. This holds because for any $f\in\widehat{\mathcal{F}}_{m}(\beta)\subseteq{}\widetilde{\mathcal{F}}_{m}\left(2\beta+\frac{{C_{\delta}^{\prime}}}{\tau_{m}-1}\right)$, we have

[TABLE]

and also by the epoch schedule of the algorithm $\operatorname{\mathbb{E}}_{t}\left[M_{t}(f)\right]$ remains the same for all $t\in\{\tau_{m-1},\ldots,\tau_{m}-1\}$ and thus $T_{m-1}\operatorname{\mathbb{E}}_{\tau_{m-1}}\left[M_{\tau_{m-1}}(f)\right]=\sum_{t=\tau_{m-1}}^{\tau_{m}-1}\operatorname{\mathbb{E}}_{t}\left[M_{t}(f)\right]\leq 2\beta(\tau_{m}-1)+{C_{\delta}^{\prime}}$, proving the statement. ∎

We are now ready to prove the main theorems, which are again restated with extra results under the Massart condition.

Theorem 6** (Full version of Theorem 3).**

With $\beta_{m}=\frac{(M-m+1){C_{\delta}^{\prime}}}{\tau_{m}-1}$ and ${C_{\delta}^{\prime}}$ as in Lemma 9, Algorithm 1 with Option II ensures that with probability at least $1-\delta$,

[TABLE]

In particular, for finite classes regret is bounded as $\widetilde{O}\left(\sqrt{TL_{2,0}\log\left\lvert\mathcal{F}\right\rvert}\right)$.

Furthermore, if the Massart noise condition Definition 8 is satisfied with parameter $\gamma$, then Algorithm 1 configured as above with $\delta=1/T$ enjoys an in-expectation regret bound of

[TABLE]

which for finite classes is bounded as $\widetilde{O}\left(\frac{L_{2,0}\log\left\lvert\mathcal{F}\right\rvert}{\gamma}\right)$.

Proof.

Similar to the proof of Theorem 2, we condition on the events of Lemma 9, which happen with probability at least $1-\delta/2$.

With $m$ denoting the epoch to which round $t$ belongs and $P_{\eta}=\operatorname{Pr}_{x}\left(f^{\star}(x,\pi^{\star}(x))-\max_{a\neq{}\pi^{\star}(x)}f^{\star}(x,a)<\eta\right)$ for any $\eta>0$, we have

[TABLE]

where the last step holds because $x\in U_{0}(a)$ along with the fact $f^{\star}\in\mathcal{F}_{m-1}$ implies

[TABLE]

and thus by definition $a\in A_{\tau_{m-1}}$. We continue with

[TABLE]

Summing over $t=1,\ldots,T$, we arrive at

[TABLE]

Finally in the general case we bound $P_{\eta}$ by $1$ and pick the optimal $\eta$ to arrive at a conditional expected regret bound of order $O(\sqrt{TL_{2,0}{C_{\delta}^{\prime}}}\log T+\log(1/\delta))$, while under the Massart condition (Definition 8) one can pick $\eta=\gamma$ so that $P_{\eta}=0$ and obtain a conditional expected regret bound of order $O\left(\frac{L_{2,0}{C_{\delta}^{\prime}}\log^{2}T}{\gamma}\right)$.

As in the proof of Theorem 5, we relate the sum of conditional expected instantaneous regrets back to the quantities in the theorem statement differently in the general case and the Massart case. In the general case we have

[TABLE]

with probability at least $1-\delta/2$.

In the Massart case, the law of total expectation implies

[TABLE]

∎

Theorem 7** (Full version of Theorem 4).**

With $\beta_{m}=\frac{(M-m+1){C_{\delta}^{\prime}}}{\tau_{m}-1}$, where ${C_{\delta}^{\prime}}$ is as in Lemma 9, and $M_{0}=2+\left\lfloor\log_{2}\left(1+\frac{(2M+3)L_{1}{C_{\delta}^{\prime}}}{\lambda^{2}}\right)\right\rfloor$ for any $\lambda\in(0,1)$, Algorithm 2 ensures that with probability at least $1-\delta$,

[TABLE]

*which for finite classes is bounded by $\widetilde{O}\left(\frac{L_{1}\log\left\lvert\mathcal{F}\right\rvert}{\lambda^{2}}+\sqrt{TL_{2,\lambda}\log\left\lvert\mathcal{F}\right\rvert}\right)$.

Furthermore, if the Massart noise condition (Definition 8) is satisfied with parameter $\gamma$, then Algorithm 2 configured as above with $\delta=1/T$ enjoys an expected regret bound of*

[TABLE]

which for finite classes is bounded by $\widetilde{O}\left(\frac{L_{1}\log\left\lvert\mathcal{F}\right\rvert}{\lambda^{2}}+\frac{L_{2,\lambda}\log\left\lvert\mathcal{F}\right\rvert}{\gamma}\mathbf{1}\{\lambda>\gamma\}\right)$.

Proof.

We condition on the same events of Lemma 9, which hold with probability at least $1-\delta/2$. By the second claim of Lemma 10, we have for any $f\in\mathcal{F}_{M_{0}}$,

[TABLE]

Since Algorithm 2 performs pure exploration for any $t$ before epoch $M_{0}$, we conclude that

[TABLE]

and therefore together with Definition 6, we have for any $f\in\mathcal{F}_{M_{0}}$, $x\in\mathcal{X}$, and $a\in\mathcal{A}$,

[TABLE]

where the last step holds by the choice of $M_{0}$. Next we claim that for any $t\geq\tau_{M_{0}}$, if $x_{t}\in U_{\lambda}(a)$ for some $a$, then it must be the case $a_{t}=a=\pi^{\star}(x_{t})$. To begin, we have that $a=\pi^{\star}(x_{t})$, which is by the definition of $U_{\lambda}(a)$. Moreover, with $m$ being the epoch to which $t$ belongs and $a^{\prime}=\operatorname*{arg\,max}_{a\neq\pi^{\star}(x_{t})}\text{High}_{\mathcal{F}_{m}}(x_{t},a)$, we have

[TABLE]

where the inequality is by $x_{t}\in U_{\lambda}(a)$, $f^{\star}\in\text{High}_{\mathcal{F}_{m}}$, and Eq. (18). By the optimistic strategy of Algorithm 2, this implies $a_{t}=a$.

Finally we proceed exactly the same as the proof of Theorem 3 to arrive at for any $\eta>0$, $\lambda\in(0,1)$, $m>M_{0}$, and $t$ in epoch $m$,

[TABLE]

With the fact established above, since $\tau_{m-1}\geq\tau_{M_{0}}$ we continue with

[TABLE]

Therefore, the regret bound is

[TABLE]

Again in general we bound $P_{\eta}$ by $1$ and pick the optimal $\eta$ to arrive at

[TABLE]

while under the Massart condition we pick $\eta=\gamma$ so that $P_{\eta}=0$ and

[TABLE]

Specifically, if we choose $\lambda\leq\gamma$, then every $x_{t}$ is in $U_{\lambda}(\pi^{\star}(x_{t}))$ and thus $a_{t}=\pi^{\star}(x_{t})$ for all $t\geq\tau_{M_{0}}$ and the algorithm suffers no regret at all after the warm start, that is, ${\text{\rm Reg}}_{T}=O\left(\frac{L_{1}{C_{\delta}^{\prime}}\log T}{\lambda^{2}}\right)$.

To conclude we proceed as in the proof of Theorem 6: In the general case we have

[TABLE]

with probability at least $1-\delta/2$ by Azuma-Hoeffding.

In the Massart case, the law of total expectation implies

[TABLE]

∎

Proof of Proposition 2.

For this proof we will adopt the shorthand $\mathbf{1}\left\{U_{\lambda}(a)\right\}\vcentcolon=\mathbf{1}\left\{x\in{}U_{\lambda}(a)\right\}$.

We first consider the $\ell_{2}$ case. In this case (using $w\in{}\mathbb{R}^{d}$ as a stand-in for $f-f^{\star}$ and $\mathcal{W}^{\star}\vcentcolon={}\mathcal{W}-w^{\star}\setminus{}\left\{0\right\}$) it is sufficient to take

[TABLE]

In the sparse high-dimensional setting we have

[TABLE]

As remarked in the main body, in general it holds that $L_{1}\leq{}L_{2,0}$. Nonetheless, it is also possible to directly bound $L_{1}$ using similar reasoning to the proof above:

[TABLE]

for the $\ell_{2}$ example and

[TABLE]

for the sparsity example. ∎

Appendix B Experimental Details

B.1 Datasets

We evaluated on datasets for learning-to-rank and for multiclass classification.

The learning-to-rank datasets, which were previously used for evaluating contextual semibandits in (Krishnamurthy et al., 2016), are as follows:

• •

Microsoft Learning to Rank (Qin & Liu, 2010). We use the MSLR-WEB30K variant available at https://www.microsoft.com/en-us/research/project/mslr/. This dataset has $T=31278$, $d=136$. We limit the choices to $K=10$ documents (actions) per query. The MSLR repository comes partitioned into five segments, each with $T=31278$ queries and a varying number of documents. We use the first three segments for the documents in our training dataset and use documents from the fourth segment for validation.

• •

Yahoo! Learning to Rank Challenge V2.0 (Chapelle & Chang, 2011) (variant C14B at https://webscope.sandbox.yahoo.com/catalog.php?datatype=c). The dataset has $T=33850$, $d=415$, and $K=6$. We hold out 7000 examples for test.

Each learning-to-rank dataset contains over $30,000$ queries, with the number of documents varying. In both datasets feedback each document is labeled with relevance score in $\left\{0,\ldots,4\right\}$. We transform this to a contextual bandit problem by presenting $K$ documents as actions and their relevance scores as the rewards, so that the goal of the learner is to choose the document with the highest relevance each time it is presented with a query.

The multiclass classification datasets are taken from the UCI repository (Lichman, 2013) summarized in Table 1. This collection was previously used for evaluating contextual bandit learning in (Dudík et al., 2011). Each context is labeled with the index in $[K]$ of the class to which the context belongs, and the reward for selecting a class is $1$ if correct, [math] otherwise.

Validation

Validation is performed by simulating the algorithm’s predictions on examples from a holdout set without allowing the algorithm to incorporate these examples. The validation error at round $t$ therefore approaches the instantaneous expected reward $\operatorname{\mathbb{E}}_{x_{t},a_{t}}\left[f^{\star}(x_{t},a_{t})\right]$ at a rate determined by uniform convergence for the class $\mathcal{F}$. We also plot the validation reward of a “supervised” baseline obtained by training the oracle (either Linear or GB5) on the entire training set at once (including rewards for all actions).

Noisy dataset variants

For all of the multiclass datasets we also create an alternate version with real-valued costs by constructing a reward matrix $R_{t}\in\mathbb{R}^{K\times{}K}$ and returning $R_{t}(a,a^{\star})$ as the reward for selecting action $a$ when $a^{\star}$ is the correct label at time $t$. $R_{t}$ is constructed as a (possibly asymmetric) matrix with all ones on the diagonal ($R_{t}(a,a)=1$) and random values in the range $\left[0,1\right]$ for each off-diagonal entry. The off diagonal elements are generated through the following process: 1) For each off-diagonal pair $(a,a^{\prime})$ draw a “mean” $\mu(a,a^{\prime})\in\left[0,1\right]$ uniformly at random. This value of $\mu$ is held constant across all timesteps and all repetitions. 2) At time $t$, sample $R_{t}(a,a^{\prime})$ as a Bernoulli random variable with bias $\mu(a,a^{\prime})$. The reward matrices that were sampled are included in Section B.6 for reference.

B.2 Benchmark algorithms

We compared with the following benchmark algorithms:

• •

$\epsilon$-Greedy (Langford & Zhang, 2008). Policy is updated on a doubling schedule: Every $2^{i/2}$ rounds. We use an exploration probability of $\max\left\{1/\sqrt{t},\epsilon\right\}$ at time $t$, then tune $\epsilon$ as described in the main paper.

• •

ILOVETOCONBANDITS (Agarwal et al., 2014): Updated every $2^{i/2}$ rounds. We tune the constant in front of the parameter $\mu_{m}$ described in Algorithm 1 in (Agarwal et al., 2014).

• •

Bootstrap (Dimakopoulou et al., 2017): At each epoch, the algorithm draws $N$ bootstrap replicates of the dataset so far, then fits a predictor in $\mathcal{F}$ to each replicate, giving a collection of predictors $(f_{i})_{i\in\left[N\right]}$. To predict on a new context $x$ we compute the mean and variance of the predictions $f_{i}(x,\cdot)$, then pick the action that maximizes the upper confidence bound for a Normal distribution with this mean and variance. We tune the confidence $\beta$ on these predictions, so that the algorithm picks the action maximizing $\textrm{Mean}(a)+\sqrt{\beta\textrm{Var}(a)}$.

• •

As discussed in the main body, we tune the parameter $\beta=\beta_{m}$ for both RegCB variants.

For each algorithm we tried 8 different values of the relevant parameter coming from a logarithmically spaced grid ranging from $10^{2}$ to $10^{-8}$ for the confidence interval-based algorithms (RegCB and Bootstrap) and range $10^{-1}$ to $10^{-8}$ for $\epsilon$-Greedy and ILTCB.

Each algorithm must be supplied with a model class $\mathcal{F}$ and an optimization oracle for this class. Both the model class $\mathcal{F}$ and the oracle implementation are hyperparameters. How to choose the oracle once the class $\mathcal{F}$ is been fixed is discussed below.

B.3 Oracle implementation

All of the oracle-based algorithms require optimization oracles, for either predictor classes or policy classes. We consider the following three types of basic oracles.

Weighted regression onto single action

[TABLE] 2. 2.

Weighted regression onto all actions

[TABLE] 3. 3.

Weighted multiclass classification

[TABLE]

Oracles for importance-weighted observations

One of the most common datasets one needs to optimize over to implement oracle-based contextual bandit algorithms is an importance weighted history of interactions. That is, $H_{T}=\left\{(x_{t},a_{t},r_{t}(a_{t}),p_{t}(a_{t}))\right\}_{t=1}^{T}$, where $x_{t}$ and $r_{t}(a_{t})$ are the unmodified context and reward provided by nature, $a_{t}$ is the action selected by a randomized contextual bandit algorithm, and $p_{t}(a_{t})$ is the (positive) probability that $a_{t}$ was selected. The core optimization problem that must be solved for such a dataset (e.g., in $\epsilon$-Greedy) is

[TABLE]

This problem most naturally reduces to weighted multiclass classification, but under the realizability assumption in Assumption 1 it can also be reduced to regression in a number of principled ways. The full list of possible reductions we consider is as follows:

• •

Unweighted regression

[TABLE]

• •

Importance-weighted regression

[TABLE]

• •

Regression with importance weighted targets

[TABLE]

• •

Importance-weighted multiclass

[TABLE]

Note that in this case the policy class $\Pi$ is not necessarily induced by a predictor class $\mathcal{F}$, though when it is it may be possible to further reduce this optimization problem to one of the first three problems.

The minimizer of (D) corresponds to the maximizer of (19). Reductions (A), (B), and (C) all have the property that if the conditional expectation version of the loss (e.g. $\sum_{t=1}^{T}\operatorname{\mathbb{E}}_{(x_{t},r_{t})\sim{}\mathcal{D}}\operatorname{\mathbb{E}}_{a_{t}\mid{}x_{t},H_{t-1}}\left[(f(x_{t},a_{t})-r_{t}(a_{t}))^{2}\right]$ for (A)) is used, then the Bayes predictor $f^{\star}(x,a)=\operatorname{\mathbb{E}}\left[r(a)\mid{}x\right]$ is the minimizer when $f^{\star}\in\mathcal{F}$, which (via uniform convergence) justifies the use of the empirical versions.

Oracle choices for benchmark algorithms

Depending on the needs of each benchmark algorithm, (A), (B), (C), or (D) as well as other oracles may be possible to use or required. We discuss the choices for each benchmark

• •

$\epsilon$-Greedy: This strategy only needs to solve an importance weighted argmax of the form (19), so all of (A), (B), (C), and (D) can be used under realizability. Note that since actions are sampled uniformly in the non-greedy rounds, (A) and (B) are equivalent under this strategy. In experiments we use (B).

• •

Bootstrap: Like RegCB, this strategy is tailored to the realizable regression setting, so (A) suffices. While (B) and (C) could also be used, we expect them to have higher variance.

• •

ILOVETOCONBANDITS: This algorithm requires two different oracles. First, it requires the optimization problem (19) to be solved on the unmodified reward/context sequence. Second, it requires a bonafide cost-sensitive classification optimization oracle of the form

[TABLE]

for an artificial sequence of rewards which may not be realizable even when the rewards given by nature are. As in $\epsilon$-Greedy, the first oracle can use (A), (B), (C), and (D). The second oracle is more complicated. Cost-sensitive classification is typically not implemented directly and instead is reduced to either weighted multiclass (D) or multi-output regression, for which (C) is a special case. Note that (D) can further be reduced to (A), (B), (C), but because we do not expect realizability to hold it is more natural to use the direct reduction to (C) in this case. In experiments we used (B) for empirical regret minimizer and (C) for the cost-sensitive classifier to solve the optimization problem OP in Agarwal et al. (2014).

Label-dependent features

For different datasets we consider different instantiations of the general predictor class setup described in the main paper. We assume there is a base context space $\mathcal{Z}$ and predictor class $\mathcal{G}:\mathcal{Z}\to\mathbb{R}$. Give such a class there are two natural ways to build a class of predictors over the joint context-action space depending on how the dataset is featurized.

• •

Label-dependent features For the MSLR and Yahoo datasets, each context comes with a distinct set of features for each action. This is captured by our abstraction by defining a fixed feature map $\phi:\mathcal{X}\times{}\mathcal{A}\to\mathcal{Z}$, then defining the class $\mathcal{F}$ via $\mathcal{F}=\left\{(x,a)\mapsto{}g(\phi(x,a))\mid{}g\in\mathcal{G}\right\}$.

• •

Label-independent features When the contexts do not have label-dependent features, we use one instance of the base real-valued predictor class $\mathcal{G}$ for each action, i.e. we set $\mathcal{Z}=\mathcal{X}$ and take $\mathcal{F}=\left\{(x,a)\mapsto{}g_{a}(x)\mid{}g=(g_{a})_{a\in\mathcal{A}}\in\mathcal{G}^{\mathcal{A}}\right\}$.

Predictor class and base oracle implementation

We use real-valued predictors from the scikit-learn library (Pedregosa et al., 2011). The two predictor classes used were

• •

sklearn.linear_model.Ridge(alpha=1)

• •

sklearn.tree_model.GradientBoostingRegressor(max_depth=5, n_estimators=100).

Each of the scikit-learn predictor classes handles this real-valued output case directly, via the fit() function for each class. In the label-dependent feature case we use a single oracle for $\mathcal{G}$, and in the label-independent feature case we use the oracle for $\mathcal{G}$, then take $\mathcal{F}=\mathcal{G}^{\mathcal{A}}$, so that there are actually $\left\lvert\mathcal{A}\right\rvert$ oracle instances.

Incremental implementation for RegCB

As mentioned in the main body, we restrict the optimization for the gradient boosting oracle when used with RegCB. At the beginning of each epoch $m$, we find best regression tree ensemble on the dataset so far (with respect to $\widehat{R}_{m}$). For each round within the epoch, we keep this tree structure fixed for the call to $\textsc{Oracle}(H)$, so that only the ensemble and leaf weights need to be re-optimized.

B.4 Holdouts and multiple trials

Each dataset shuffled via random permutation, then presented to the learner in order.

Each (algorithm, parameter configuration, dataset) tuple was run for 5 repetitions. For a given trial we distinguish between two sources of randomness: Randomness from the dataset, which may come from the random ordering or from randomness in the labels as described in the dataset section, and randomness in the contextual bandit algorithm’s decisions. We control for randomness in the dataset across different (algorithm, parameter) configurations by giving each repetition an index $k$ and using the same random seed to select the dataset randomness across all configurations. This means that when $k$ is fixed, all variance is induced by the algorithm’s action distribution.

Validation reward was computed every $T/15$ steps.

B.5 Full collection of plots

B.6 UCI Reward Matrices