Practical Contextual Bandits with Regression Oracles

TL;DR

This paper introduces a new practical and theoretically sound approach for contextual bandits that uses regression oracles, generalizes existing methods, and performs well empirically.

Contribution

It proposes a novel technique combining realizability and agnostic methods using regression oracles, extending UCB and LinUCB to more expressive models.

Findings

Achieves low regret under certain assumptions

Performs comparably or better than existing baselines

Generalizes UCB and LinUCB to complex models

Abstract

A major challenge in contextual bandits is to design general-purpose algorithms that are both practically useful and theoretically well-founded. We present a new technique that has the empirical and computational advantages of realizability-based approaches combined with the flexibility of agnostic methods. Our algorithms leverage the availability of a regression oracle for the value-function class, a more realistic and reasonable oracle than the classification oracles over policies typically assumed by agnostic methods. Our approach generalizes both UCB and LinUCB to far more expressive possible model classes and achieves low regret under certain distributional assumptions. In an extensive empirical evaluation, compared to both realizability-based and agnostic baselines, we find that our approach typically gives comparable or superior results.

Figures

Tables

Table 1. Table 1: UCI datasets (before validation split).

Dataset	$n$	$d$	$K$
letter	20000	17	26
optdigits	5620	65	10
adult	45222	105	2
page-blocks	5473	11	5
pendigits	10992	17	10
satimage	6430	37	6
vehicle	846	19	4
yeast	1479	9	9

Table 2. Table 2: Mean reward matrix: yeast

1.00	0.02	0.26	0.89	0.20	0.31	0.97	0.34	0.39
0.25	1.00	0.65	0.29	0.03	0.52	0.30	0.10	0.09
0.49	0.92	1.00	0.61	0.33	0.84	0.22	0.37	0.62
0.21	0.67	0.13	1.00	0.27	0.69	0.39	0.97	0.15
0.26	0.63	0.11	0.39	1.00	0.25	0.34	0.17	0.69
0.78	0.11	0.22	0.16	0.22	1.00	0.68	0.39	0.55
0.40	0.71	0.40	0.45	0.23	0.48	1.00	0.99	0.43
0.41	0.48	0.85	0.87	0.22	0.46	0.33	1.00	0.96
0.07	0.76	0.49	0.74	0.44	0.85	0.09	0.23	1.00

Table 3. Table 3: Mean reward matrix: letter

1.00	0.33	0.82	0.04	0.11	0.60	0.53	0.42	0.34	0.62	0.44	0.74	0.52	0.58	0.65	0.99	0.82	0.41	0.88	0.82	0.05	0.72	0.80	0.74	0.71	0.54
0.12	1.00	0.40	0.22	0.72	0.99	0.26	0.67	0.60	0.72	0.94	0.35	0.25	0.40	0.75	0.72	0.41	0.99	0.45	0.37	0.71	0.08	0.40	0.77	0.76	0.28
0.19	0.47	1.00	0.73	0.19	0.33	0.84	0.62	0.89	0.98	0.84	0.18	0.62	0.48	0.40	0.74	0.83	0.68	0.14	0.70	0.06	0.19	0.92	0.41	0.15	0.68
0.16	0.65	0.25	1.00	0.96	0.07	0.51	0.34	0.66	0.84	0.60	0.59	0.12	0.71	0.20	0.49	0.04	0.32	0.86	0.56	0.55	0.37	0.83	0.28	0.13	0.56
0.27	0.78	0.18	0.78	1.00	0.04	0.56	0.67	0.94	0.79	0.75	0.50	0.04	0.82	0.01	0.55	0.57	0.11	0.06	0.57	0.49	0.30	0.04	0.63	0.12	0.01
0.28	0.30	0.18	0.07	0.78	1.00	0.25	0.52	0.25	0.85	0.48	0.62	0.97	0.35	0.22	0.98	0.59	0.98	0.97	0.71	0.02	0.61	0.25	0.13	0.37	0.20
0.77	0.93	0.03	0.26	0.27	0.14	1.00	0.25	0.36	0.05	0.24	0.88	0.96	0.66	0.30	0.06	0.86	0.16	0.27	0.55	0.25	0.84	0.50	0.48	0.91	0.92
0.24	0.02	0.67	0.27	0.01	0.10	0.42	1.00	0.21	0.75	0.46	0.11	0.22	0.93	0.01	0.64	0.64	0.68	0.58	0.78	0.82	0.65	0.18	0.73	0.28	0.84
0.57	0.09	0.91	0.46	0.94	0.04	0.11	0.76	1.00	0.45	0.82	0.42	0.19	0.84	0.11	0.29	0.22	0.46	0.32	0.91	0.79	0.71	0.14	0.61	0.85	0.92
0.66	0.26	0.28	0.64	0.72	0.31	0.68	0.51	0.83	1.00	0.91	0.12	0.84	0.95	0.57	0.00	0.03	0.41	0.46	0.48	0.68	0.75	0.82	0.35	0.61	0.39
0.73	0.56	0.59	0.39	0.63	0.87	0.65	0.13	0.09	0.68	1.00	0.31	0.89	0.86	0.81	0.36	0.64	0.60	0.24	0.59	1.00	0.05	0.24	0.33	0.80	0.44
0.06	0.32	0.83	0.74	0.28	0.73	0.32	0.15	0.98	0.26	0.61	1.00	0.64	0.43	0.40	0.05	0.08	0.45	0.92	0.23	0.87	0.81	0.17	0.31	0.43	0.86
0.63	0.82	0.50	0.58	0.45	0.26	0.62	0.58	0.87	0.92	0.57	0.69	1.00	0.68	1.00	0.94	0.14	0.94	0.04	0.03	0.18	0.31	0.98	0.94	0.76	0.62
0.97	0.57	0.21	0.13	0.76	0.53	0.82	0.79	0.67	0.78	0.69	0.43	0.83	1.00	0.78	0.09	0.95	0.48	0.89	0.08	0.94	0.31	0.42	0.69	0.09	0.21
0.58	0.39	0.11	0.01	0.90	0.67	0.32	0.89	0.97	0.08	0.26	0.53	0.92	0.23	1.00	0.90	0.34	0.23	0.18	0.05	0.96	0.15	0.96	0.34	0.06	0.82
0.80	0.46	0.77	0.75	0.45	0.28	0.14	0.91	0.08	0.73	0.08	0.67	0.06	0.11	0.48	1.00	0.03	0.64	0.90	0.48	0.84	0.71	0.93	0.97	0.59	0.95
0.71	0.46	0.92	0.58	0.24	0.39	0.42	0.16	0.02	0.05	0.68	0.25	0.15	0.20	0.82	0.89	1.00	0.74	0.58	0.49	0.64	0.95	0.80	0.41	0.25	0.00
0.29	0.98	0.42	0.54	0.06	0.14	0.99	0.54	0.22	0.64	0.73	0.50	0.33	0.72	0.13	0.72	0.45	1.00	0.63	0.86	0.32	0.70	0.12	0.44	0.72	0.89
0.56	0.63	0.53	0.35	0.85	0.57	0.26	0.80	0.83	0.45	0.68	0.09	0.72	0.34	0.02	0.71	0.55	0.83	1.00	0.99	0.33	0.13	0.04	0.32	0.21	0.57
0.96	0.22	0.33	0.27	0.27	0.69	0.89	0.58	0.40	0.43	0.55	0.31	0.26	0.91	0.51	0.12	0.57	0.25	0.01	1.00	0.36	0.68	0.61	0.17	0.30	0.72
0.43	0.13	0.17	0.73	0.62	0.56	0.06	0.39	0.45	0.58	0.70	0.72	0.59	0.27	0.41	0.78	0.47	0.40	0.85	1.00	1.00	0.63	0.91	0.15	0.29	0.65
0.18	0.28	0.94	0.31	0.10	0.50	0.08	0.25	0.96	0.84	0.15	0.25	0.05	0.20	0.81	0.91	0.62	0.09	0.50	0.67	0.11	1.00	0.76	0.39	0.83	0.17
0.26	0.80	0.68	0.78	0.18	0.95	0.18	0.70	0.31	0.51	0.91	0.78	0.75	0.11	0.91	0.90	0.98	0.11	0.38	0.27	0.85	0.90	1.00	0.22	0.05	0.88
0.95	0.75	0.82	0.31	0.13	0.10	0.67	0.14	0.92	0.24	0.75	0.61	0.34	0.63	0.02	0.76	0.17	0.61	0.12	0.57	0.73	0.80	0.14	1.00	0.41	0.40
0.83	0.19	0.76	0.74	0.42	0.14	0.70	0.88	0.18	0.12	0.21	0.44	0.46	0.76	0.16	0.90	0.52	0.28	0.02	0.59	0.20	0.44	0.96	0.20	1.00	0.84
0.03	0.67	0.47	0.34	0.50	0.43	0.56	0.11	0.36	0.93	0.50	0.64	0.47	0.97	0.12	0.35	0.68	0.79	0.40	0.74	0.37	0.10	0.02	0.14	0.99	1.00

Table 4. Table 4: Mean reward matrix: optdigits

1.00	0.56	0.12	0.40	0.78	0.51	0.18	0.85	0.96	0.98
0.19	1.00	0.23	0.03	0.95	0.92	0.29	0.17	0.40	0.51
0.31	0.43	1.00	0.56	0.83	1.00	0.33	0.09	0.77	0.15
0.73	0.96	0.07	1.00	0.84	0.15	0.77	0.78	0.68	0.13
0.04	0.66	0.25	0.99	1.00	0.06	0.70	0.63	0.90	0.16
0.61	0.32	0.76	0.16	0.93	1.00	0.83	0.23	0.11	0.67
0.58	0.88	1.00	0.28	0.74	0.28	1.00	0.49	0.87	0.16
0.97	0.05	0.70	0.65	0.05	0.20	0.33	1.00	0.37	0.53
0.35	0.51	0.26	0.85	0.62	0.30	0.78	0.90	1.00	0.86
0.82	0.87	0.38	0.61	0.42	0.24	0.06	0.82	0.38	1.00

Table 5. Table 5: Mean reward matrix: page-blocks

1.00	0.38	0.66	0.16	0.96
0.35	1.00	0.24	0.59	0.41
0.14	0.54	1.00	0.77	0.93
0.09	0.20	0.99	1.00	0.24
0.63	0.73	0.69	0.03	1.00

Table 6. Table 6: Mean reward matrix: pendigits

1.00	0.37	0.56	0.96	0.74	0.82	0.10	0.93	0.61	0.60
0.09	1.00	0.66	0.44	0.55	0.70	0.59	0.05	0.56	0.77
0.91	0.09	1.00	0.46	0.45	1.00	0.16	0.71	0.16	0.81
0.04	0.53	0.17	1.00	0.05	0.24	0.67	0.78	0.70	0.33
0.49	0.52	0.30	0.46	1.00	0.50	0.40	0.73	0.86	0.03
0.29	0.79	0.46	0.01	0.42	1.00	0.60	0.32	0.98	0.59
0.13	0.52	0.36	0.01	0.10	0.78	1.00	0.20	0.62	0.64
0.27	0.13	0.47	0.39	0.41	0.38	0.29	1.00	0.43	0.78
0.70	0.78	0.29	0.21	0.50	0.13	0.17	0.25	1.00	0.23
0.63	0.63	0.53	0.74	0.82	0.37	0.80	0.88	0.59	1.00

Table 7. Table 7: Mean reward matrix: satimage

1.00	0.06	0.12	0.79	0.98	0.27
0.87	1.00	0.64	0.78	0.63	0.13
1.00	0.63	1.00	0.62	0.34	0.76
0.11	0.52	0.63	1.00	0.11	0.29
0.07	0.67	0.23	0.52	1.00	0.45
0.73	0.97	0.20	0.72	0.79	1.00

Table 8. Table 8: Mean reward matrix: vehicle

1.00	0.36	0.18	0.52
0.01	1.00	0.80	0.76
0.67	0.03	1.00	0.40
0.19	0.77	0.62	1.00

Table 9. Table 9: Mean reward matrix: adult

1.00	0.61
0.66	1.00

Equations

$$E\left[r(a)\mid x,a\right]=f^{\star }(x,a)\text{for all x∈X and a∈A.}$$

$${\text{Reg}}_T=\sum _{t=1}^T{r}_t({\pi }^{\star }(x_t))-\sum _{t=1}^T{r}_t(a_t),$$

$$\text{textsc}Oracle(H)=\underset{f\in \mathcal{F}}{\mathrm{argmin}}\sum _{(w,x,a,y)\in H}w(f(x,a)-y{)}^2.$$

$${\text{High}}_{{\mathcal{F}}^{\prime }}(x,a)$$

$${\text{Low}}_{{\mathcal{F}}^{\prime }}(x,a)$$

$$a_{t}\sim\text{Unif}\left(\Bigl{\{}a\mathbin{\Bigm{|}}\text{High}_{\mathcal{F}_{m}}(x_{t},a)\geq{}\max_{a^{\prime}\in\mathcal{A}}\text{Low}_{\mathcal{F}_{m}}(x_{t},a^{\prime})\Bigr{\}}\right),$$

$$a_t=\underset{a\in \mathcal{A}}{\mathrm{argmax}}{\text{High}}_{{\mathcal{F}}_m}(x_t,a)$$

$$z_{\text{High}}$$

$$z_{\text{Low}}$$

$$\left|{\text{High}}_{{\hat{\mathcal{F}}}_m(\beta )}(x,a)-z_{\text{High}}\right|$$

$$\left|{\text{Low}}_{{\hat{\mathcal{F}}}_m(\beta )}(x,a)-z_{\text{Low}}\right|$$

$$\displaystyle\mathcal{F}(\varepsilon)=\Bigl{\{}f\in\mathcal{F}:\>\operatorname{\mathbb{E}}[r(\pi_{f}(x))]\geq\operatorname{\mathbb{E}}[r(\pi^{\star}(x))]$$

$$W_{\mathcal{F}}(x,a)={\text{High}}_{\mathcal{F}}(x,a)-{\text{Low}}_{\mathcal{F}}(x,a).$$

$$\mathrm{Dis}(\mathcal{F})$$

$$\displaystyle=\Bigl{\{}x\;\mathbin{\Bigm{|}}\;\exists{}f,f^{\prime}\in\mathcal{F}:\operatorname*{arg\,max}_{a\in\mathcal{A}}f(x,a)\neq{}\operatorname*{arg\,max}_{a\in\mathcal{A}}f^{\prime}(x,a)\Bigr{\}}.$$

$$A_{\mathcal{F}}(x)=\bigcup _{f\in \mathcal{F}}\underset{a\in \mathcal{A}}{\mathrm{argmax}}f(x,a).$$

$${\theta }_0:=\underset{\delta >0,\epsilon >0}{\sup }\frac{\delta }{\epsilon }{\mathrm{Pr}}_{D_{\mathcal{X}}}$$

$$\displaystyle\quad\exists{}a\in{}A_{\mathcal{F}(\varepsilon)}(x):W_{\mathcal{F}(\varepsilon)}(x,a)>\delta\Bigr{]}.$$

$${\text{Reg}}_T=\tilde{O}\left(T^{\frac{3}{4}}{\left(\log \left|\mathcal{G}\right|\right)}^{\frac{1}{4}}\sqrt{{\theta }_{0}K}\right).$$

$$\displaystyle\bigl{(}f(x,a)-f^{\star}(x,a)\bigr{)}^{2}$$

$$\displaystyle\quad{}\leq L_{1}\operatorname{\mathbb{E}}_{x^{\prime}\sim D_{\mathcal{X}}}\operatorname{\mathbb{E}}_{a^{\prime}\sim\text{Unif}\left(\mathcal{A}\right)}\Bigl{[}\bigl{(}f(x^{\prime},a^{\prime})-f^{\star}(x^{\prime},a^{\prime})\bigr{)}^{2}\Bigr{]}\enspace.$$

$${\Vert h\Vert }_{L_{\infty }({\mathcal{D}}^{\prime })}\le L_1\cdot {\Vert h\Vert }_{L_2({\mathcal{D}}^{\prime })}.$$

$$U_{\lambda}(a)\vcentcolon=\Bigl{\{}x\;\mathbin{\Bigm{|}}\;f^{\star}(x,a)\geq f^{\star}(x,a^{\prime})+\lambda\text{ for all $a^{\prime}\neq a$}\Bigr{\}}.$$

$$\displaystyle\bigl{(}f(x,a)-f^{\star}(x,a)\bigr{)}^{2}$$

$$\displaystyle\leq L_{2,\lambda}\operatorname{\mathbb{E}}_{x^{\prime}\sim D_{\mathcal{X}}}\operatorname{\mathbb{E}}_{a^{\prime}\sim{}\text{Unif}\left(\mathcal{A}\right)}\Bigl{[}\mathbf{1}\bigl{\{}x^{\prime}\in U_{\lambda}(a^{\prime})\bigr{\}}$$

$$\displaystyle\hskip 115.63243pt{}\cdot\bigl{(}f(x^{\prime},a^{\prime})-f^{\star}(x^{\prime},a^{\prime})\bigr{)}^{2}\Bigr{]}.$$

$$\frac{L_{2,0}}{K}{E}_{x\sim D_{\mathcal{X}}}[(f(x,{\pi }^{\star }(x))-f^{\star }(x,{\pi }^{\star }(x)){)}^2].$$

$${\text{Reg}}_T=\tilde{O}\left(\sqrt{T{L}_{2,0}\log \left|\mathcal{F}\right|}\right).$$

$${\text{Reg}}_T=\tilde{O}\left(\frac{L_1\log \left|\mathcal{F}\right|}{{\lambda }^2}+\sqrt{T{L}_{2,\lambda }\log \left|\mathcal{F}\right|}\right).$$

$$\frac{K}{\lambda_{\min}\left(\sum_{a\in\mathcal{A}}\operatorname{\mathbb{E}}_{x}\bigl{[}\mathbf{1}\left\{x\in{}U_{\lambda}(a)\right\}\,\phi(x,a)\phi(x,a)^{\top}\bigr{]}\right)},$$

$$\frac{K}{{\lambda }_{\min }\left({\sum }_{a\in \mathcal{A}}{E}_x\left[\varphi (x,a)\varphi (x,a{)}^{\top }\right]\right)}.$$

$$\frac{2Ks}{\psi_{\min}\left(\sum_{a\in\mathcal{A}}\operatorname{\mathbb{E}}_{x}\bigl{[}\mathbf{1}\left\{x\in{}U_{\lambda}(a)\right\}\,\phi(x,a)\phi(x,a)^{\top}\bigr{]}\right)},$$

$$\frac{2Ks}{\psi_{\min}\left(\sum_{a\in\mathcal{A}}\operatorname{\mathbb{E}}_{x}\bigl{[}\phi(x,a)\phi(x,a)^{\top}\bigr{]}\right)}.$$

$$\widetilde{O}\left(s\sqrt{KT\log{}d\,\bigm{/}\,\psi_{\min}\left(\operatorname{\mathbb{E}}_{x}\bigl{[}\phi(x,\pi^{\star}(x))\phi(x,\pi^{\star}(x))^{\top}\bigr{]}\right)}\right).$$

Full text

\xpatchcmd

1 Introduction

We study the design of practically useful, theoretically well-founded, general-purpose algorithms for the contextual bandits problem. In this setting, the learner repeatedly receives context, then selects an action, resulting in a received reward. The aim is to learn a policy, a rule for choosing actions based on context, so as to maximize the long-term cumulative reward. For instance, a news portal must repeatedly choose articles to present to each user to maximize clicks. Here, the context is information about the user, the actions represent the choice of articles, and the reward indicates if there was a click. We refer the reader to a recent ICML 2017 tutorial (http://hunch.net/~rwil/) for further examples and motivation.

Approaches to contextual bandit learning can broadly be put into two groups. Some methods (Langford & Zhang, 2008; Agarwal et al., 2014) are agnostic in the sense that they are provably effective for any given policy class and data distribution. In contrast, realizability-based approaches such as LinUCB and variants (Chu et al., 2011; Li et al., 2017; Filippi et al., 2010) or Thompson Sampling (Thompson, 1933) assume the data is generated from a particular parametrized family of models. Computationally tractable realizability-based algorithms are only known for specific model families, such as when the conditional reward distributions come from a generalized linear model.

The two groups of approaches seem to have different advantages and disadvantages. Empirically, in the contextual semibandit setting, Krishnamurthy et al. (2016) found that the realizability-based LinUCB approach outperforms all agnostic baselines using a linear policy class. However, the agnostic approaches were able to overcome this shortcoming by using a more powerful policy class. Computationally, previous realizability-based approaches have been limited by their reliance on either closed-form confidence bounds (as in LinUCB variants), or the ability to efficiently sample from and frequently update the posterior (as in Thompson sampling). Agnostic approaches, on the other hand, typically assume an oracle for cost-sensitive classification, which is in general computationally intractable, though often practically feasible for many natural policy classes.

In this paper, we aim to develop techniques that combine what is best about both of these approaches. To this end, in Section 3, we propose computationally efficient and practical realizability-based algorithms for arbitrary model classes. As is often done in agnostic approaches, we assume the availability of an oracle which reduces to a standard learning setting and knows how to efficiently leverage the structure of the model class. Specifically, we require access to a squared regression oracle over the model class that we use for predicting rewards, given contexts. Since regression can often be solved efficiently, the availability of such an oracle is a rather mild assumption, far more reasonable than the kind of cost-sensitive classification oracle more commonly assumed, which typically must solve NP-hard problems. In fact, for this reason, even the classification oracles are typically approximated by regression oracles in practice (see, e.g., Beygelzimer & Langford, 2009). Our main algorithmic components here are motivated by and adapted from a recent work of Krishnamurthy et al. (2017) on cost-sensitive active learning.

In Section 4, we prove that our algorithms are effective in the sense of achieving low regret under certain favorable distributional assumptions. Specifically, we show that our methods enjoy low regret so long as certain quantities like the disagreement coefficient (Hanneke et al., 2014; Krishnamurthy et al., 2017) are bounded. We also present a second set of bounds in terms of certain distributional coefficients which generalize the exploration parameters introduced by Bastani & Bayati (2015) from linear to general function classes. As a special consequence, we obtain nearly dimension-free results for sparse linear bandits in high dimensions.

Finally, in Section 5, we conduct a very extensive empirical evaluation of our algorithms on a number of datasets and against both realizability-based and agnostic baselines. In this test of practical effectiveness, we find that our approach gives comparable or superior results in nearly all cases, and we also validate the distributional assumptions required for low-regret guarantees on these datasets.

2 Preliminaries

We consider the following contextual bandit protocol. Contexts are drawn from an arbitrary space, $x\in\mathcal{X}$, actions are from a finite set, $a\in\mathcal{A}\vcentcolon=\{1,\ldots,K\}$, for some fixed $K$, and reward vectors are from a bounded set, $r\in[0,1]^{K}$, with component $r(a)$ denoting the reward for action $a\in\mathcal{A}$.

We consider a stochastic setting where there is a fixed and unknown distribution $D$ over the context-reward pairs $(x,r)$. Its marginal distribution over $\mathcal{X}$ is denoted by $D_{\mathcal{X}}$. The learning protocol proceeds in rounds $t=1,\ldots,T$. In each round $t$, nature samples $(x_{t},r_{t})$ according to $D$ and reveals $x_{t}$ to the learner. The learner chooses an action $a_{t}\in\mathcal{A}$ and observes the reward $r_{t}(a_{t})$. The goal of the learner is to maximize the reward and do well compared with any strategy that models the expected reward $\operatorname{\mathbb{E}}[r(a)\mathbin{|}x,a]$ via a function $f:\mathcal{X}\times\mathcal{A}\to[0,1]$. These mappings $f$ are drawn from a given class of predictors $\mathcal{F}$, such as the class of linear predictors or regression trees.

The main assumption this paper follows is that the class $\mathcal{F}$ is rich enough to contain a predictor that perfectly predicts the expected reward of any action under any context, that is:

Assumption 1** (Realizability).**

There is a predictor $f^{\star}\in\mathcal{F}$ such that

[TABLE]

Given a predictor $f\in\mathcal{F}$, the associated optimal strategy $\pi_{f}:\mathcal{X}\to\mathcal{A}$, called a policy, always picks the action with the highest predicted reward, i.e., $\pi_{f}(x)\vcentcolon=\operatorname*{arg\,max}_{a\in\mathcal{A}}f(x,a)$ (breaking ties arbitrarily). We use the abbreviation $\pi^{\star}\vcentcolon=\pi_{f^{\star}}$ for the underlying optimal policy. The formal goal of the learner is then to minimize the regret

[TABLE]

which compares the accumulated rewards between the optimal policy and the learner’s strategy. The classic Exp4 algorithm (Auer et al., 2002) achieves an optimal regret bound of order $O(\sqrt{TK\ln|\mathcal{F}|})$ (for any finite $\mathcal{F}$), but the computational complexity is unfortunately linear in $|\mathcal{F}|$.

Regression Oracle

To overcome the computational obstacle, our algorithms reduce the contextual bandit problem to weighted least-squares regression. Abstracting the computational complexity, we assume access to a weighted least-squares regression oracle over the predictor class $\mathcal{F}$, which takes any set $H$ of weighted examples $(w,x,a,y)\in{\mathbb{R}_{+}\times\mathcal{X}\times\mathcal{A}\times[0,1]}$ as input, and outputs the predictor with the smallest weighted squared loss:

[TABLE]

As mentioned, such regression tasks are very common in machine learning practice and the availability of such oracle is thus a very mild assumption.

3 Algorithms

The high-level idea of our algorithms is the following. As data is collected, we maintain a subset of $\mathcal{F}$, which we refer to as the version space, that only contains predictors with small squared loss on observed data. When a new example arrives, we construct upper and lower confidence bounds on the reward of each action based on the predictors in the version space. Finally, with these confidence bounds, we either optimistically pick the action with the highest upper bound, similar to UCB and LinUCB, or randomize among all actions that are potentially the best.

The challenge here is to maintain such version spaces and confidence bounds efficiently, and we show that this can be done using a simple binary search together with a small number of regression oracle calls.

We now describe our algorithms more formally. First, we define the upper and lower reward bounds with respect to a a subset $\mathcal{F}^{\prime}\subseteq\mathcal{F}$ as

[TABLE]

Our algorithms will induce the confidence bounds by instantiating these quantities using the version space as $\mathcal{F}^{\prime}$. To reduce computational costs, our algorithms update according to a doubling epoch schedule. Epoch $m$ will begin at time $\tau_{m}=2^{m-1}$, and $M=O(\log T)$ is the total number of epochs. At each epoch $m$ our algorithms (implicitly) construct a version space $\mathcal{F}_{m}\subseteq{}\mathcal{F}$, and then select an action based on the reward ranges defined by $\text{High}_{\mathcal{F}_{m}}\!(x,a)$ and $\text{Low}_{\mathcal{F}_{m}}\!(x,a)$ for each time $t$ that falls into epoch $m$. Specifically, we consider two algorithm variants: the first one uniformly at random picks from actions that are plausible to be the best, that is,

[TABLE]

where $\text{Unif}\left(S\right)$ denotes the uniform distribution over a set $S$; the second one simply behaves optimistically and picks the action with the highest upper reward bound, that is,

[TABLE]

(ties are broken arbitrarily). For technical reasons, the optimistic variant also spends the first few epochs doing pure exploration as a warm start for the algorithm.

To construct these version spaces, we further introduce the following least-squares notation for any $m\geq 2$:

• •

\widehat{R}_{m}(f)=\frac{1}{\tau_{m}-1}\sum_{s<\tau_{m}}\bigl{(}f(x_{s},a_{s})-r_{s}(a_{s})\bigr{)}^{2},

• •

\widehat{\mathcal{F}}_{m}(\beta)=\bigl{\{}f\in\mathcal{F}\mathbin{\bigm{|}}\widehat{R}_{m}(f)-\min_{f\in\mathcal{F}}\widehat{R}_{m}(f)\leq\beta\bigr{\}},

and also let $\widehat{\mathcal{F}}_{1}(\beta)=\mathcal{F}$ for any $\beta$. With this notation $\mathcal{F}_{m}$ is simply set to $\widehat{\mathcal{F}}(\beta_{m})$ for some tolerance parameter $\beta_{m}$.

Product Classes

Sometimes it is desirable to have a product predictor class, that is, $\mathcal{F}=\mathcal{G}^{\mathcal{A}}$, where $\mathcal{G}:\mathcal{X}\to\left[0,1\right]$ is a “base class” and each $f\in\mathcal{F}$, described by a $K$-tuple $(g_{a})_{a\in\mathcal{A}}$ where $g_{a}\in\mathcal{G}$, predicts according to $f(x,a)=g_{a}(x)$. Similar to the general case, we introduce the following notation for $m\geq 2$:

• •

$\widehat{\mathcal{R}}_{m}(g,a)=\frac{1}{\tau_{m}-1}\sum_{s<\tau_{m}}(g(x_{s})-r_{s}(a_{s}))^{2}\mathbf{1}\left\{a_{s}=a\right\}$