TL;DR
This paper shows that simple greedy algorithms can be rate optimal in contextual bandits under certain conditions, and introduces Greedy-First, a new adaptive algorithm that reduces unnecessary exploration while maintaining optimal regret.
Contribution
It proves greedy algorithms can be rate optimal under covariate diversity and introduces Greedy-First, an adaptive method that achieves optimality without distribution assumptions.
Findings
Greedy algorithms can be rate optimal with sufficient context randomness.
Greedy-First adaptively chooses between greedy and exploratory strategies.
Greedy-First outperforms traditional exploration-based algorithms in simulations.
Abstract
The contextual bandit literature has traditionally focused on algorithms that address the exploration-exploitation tradeoff. In particular, greedy algorithms that exploit current estimates without any exploration may be sub-optimal in general. However, exploration-free greedy algorithms are desirable in practical settings where exploration may be costly or unethical (e.g., clinical trials). Surprisingly, we find that a simple greedy algorithm can be rate optimal (achieves asymptotically optimal regret) if there is sufficient randomness in the observed contexts (covariates). We prove that this is always the case for a two-armed bandit under a general class of context distributions that satisfy a condition we term covariate diversity. Furthermore, even absent this condition, we show that a greedy algorithm can be rate optimal with positive probability. Thus, standard bandit algorithms may…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
\OneAndAHalfSpacedXI\TheoremsNumberedThrough\ECRepeatTheorems\EquationsNumberedThrough
\RUNTITLE
Exploration-Free Contextual Bandits \TITLEMostly Exploration-Free Algorithms for
Contextual Bandits \ARTICLEAUTHORS\AUTHORHamsa Bastani \AFFWharton School, \[email protected] \AUTHORMohsen Bayati \AFFStanford Graduate School of Business, \[email protected] \AUTHORKhashayar Khosravi \AFFStanford University Electrical Engineering, \[email protected]
\ABSTRACT
The contextual bandit literature has traditionally focused on algorithms that address the exploration-exploitation tradeoff. In particular, greedy algorithms that exploit current estimates without any exploration may be sub-optimal in general. However, exploration-free greedy algorithms are desirable in practical settings where exploration may be costly or unethical (e.g., clinical trials). Surprisingly, we find that a simple greedy algorithm can be rate optimal (achieves asymptotically optimal regret) if there is sufficient randomness in the observed contexts (covariates). We prove that this is always the case for a two-armed bandit under a general class of context distributions that satisfy a condition we term covariate diversity. Furthermore, even absent this condition, we show that a greedy algorithm can be rate optimal with positive probability. Thus, standard bandit algorithms may unnecessarily explore. Motivated by these results, we introduce Greedy-First, a new algorithm that uses only observed contexts and rewards to determine whether to follow a greedy algorithm or to explore. We prove that this algorithm is rate optimal without any additional assumptions on the context distribution or the number of arms. Extensive simulations demonstrate that Greedy-First successfully reduces exploration and outperforms existing (exploration-based) contextual bandit algorithms such as Thompson sampling or upper confidence bound (UCB).
\KEYWORDSsequential decision-making, contextual bandit, greedy algorithm, exploration-exploitation
1 Introduction
Service providers across a variety of domains are increasingly interested in personalizing decisions based on customer characteristics. For instance, a website may wish to tailor content based on an Internet user’s web history (Li et al. 2010), or a medical decision-maker may wish to choose treatments for patients based on their medical records (Kim et al. 2011). In these examples, the costs and benefits of each decision depend on the individual customer or patient, as well as their specific context (web history or medical records respectively). Thus, in order to make optimal decisions, the decision-maker must learn a model predicting individual-specific rewards for each decision based on the individual’s observed contextual information. This problem is often formulated as a contextual bandit (Auer 2002, Langford and Zhang 2007, Li et al. 2010), which generalizes the classical multi-armed bandit problem (Thompson 1933, Lai and Robbins 1985).
In this setting, the decision-maker has access to possible decisions (arms) with uncertain rewards. Each arm is associated with an unknown parameter that is predictive of its individual-specific rewards. At each time , the decision-maker observes an individual with an associated context vector . Upon choosing arm , she realizes a (linear) reward of
[TABLE]
where are idiosyncratic shocks. One can also consider nonlinear rewards given by generalized linear models (e.g., logistic, probit, and Poisson regression); in this case, (1) is replaced with
[TABLE]
where is a suitable inverse link function (Filippi et al. 2010, Li et al. 2017). The decision-maker’s goal is to maximize the cumulative reward over different individuals by gradually learning the arm parameters. Devising an optimal policy for this setting is often computationally intractable, and thus, the literature has focused on effective heuristics that are asymptotically optimal, including UCB (Dani et al. 2008, Abbasi-Yadkori et al. 2011), Thompson sampling (Agrawal and Goyal 2013, Russo and Van Roy 2014), information-directed sampling (Russo and Van Roy 2018), and algorithms inspired by -greedy methods (Goldenshluger and Zeevi 2013, Bastani and Bayati 2020).
The key ingredient in designing these algorithms is addressing the exploration-exploitation tradeoff. On one hand, the decision-maker must explore or sample each decision for random individuals to improve her estimate of the unknown arm parameters ; this information can be used to improve decisions for future individuals. Yet, on the other hand, the decision-maker also wishes to exploit her current estimates to make the estimated best decision for the current individual in order to maximize cumulative reward. The decision-maker must therefore carefully balance both exploration and exploitation to achieve good performance. In general, algorithms that fail to explore sufficiently may fail to learn the true arm parameters, yielding poor performance.
However, exploration may be prohibitively costly or infeasible in a variety of practical environments (Bird et al. 2016). In medical decision-making, choosing a treatment that is not the estimated-best choice for a specific patient may be unethical; in marketing applications, testing out an inappropriate ad on a potential customer may result in the costly, permanent loss of the customer. Such concerns may deter decision-makers from deploying bandit algorithms in practice.
In this paper, we analyze the performance of exploration-free greedy algorithms. Surprisingly, we find that a simple greedy algorithm can achieve the same state-of-the-art asymptotic performance guarantees as standard bandit algorithms if there is sufficient randomness in the observed contexts (thereby creating natural exploration). In particular, we prove that the greedy algorithm is near-optimal for a two-armed bandit when the context distribution satisfies a condition we term covariate diversity; this property requires that the covariance matrix of the observed contexts conditioned on any half space is positive definite. We show that covariate diversity is satisfied by a natural class of continuous and discrete context distributions. Furthermore, even absent covariate diversity, we show that a greedy approach provably converges to the optimal policy with some probability that depends on the problem parameters. Our results hold for arm rewards given by both linear and generalized linear models. Thus, exploration may not be necessary at all in a general class of problem instances, and is only sometimes be necessary in other problem instances.
Unfortunately, one may not know a priori when a greedy algorithm will converge, since its convergence depends on unknown problem parameters. For instance, the decision-maker may not know if the context distribution satisfies covariate diversity; if covariate diversity is not satisfied, the greedy algorithm may be undesirable since it may achieve linear regret some fraction of the time (i.e., it fails to converge to the optimal policy with positive probability). To address this concern, we present Greedy-First, a new algorithm that seeks to reduce exploration when possible by starting with a greedy approach, and incorporating exploration only when it is confident that the greedy algorithm is failing with high probability. In particular, we formulate a simple hypothesis test using observed contexts and rewards to verify (with high probability) if the greedy arm parameter estimates are converging at the asymptotically optimal rate. If not, our algorithm transitions to a standard exploration-based contextual bandit algorithm.
Greedy-First satisfies the same asymptotic guarantees as standard contextual bandit algorithms without our additional assumptions on covariate diversity or any restriction on the number of arms. More importantly, Greedy-First does not perform any exploration (i.e., remains greedy) with high probability if the covariate diversity condition is met. Furthermore, even when covariate diversity is not met, Greedy-First provably reduces the expected amount of forced exploration compared to standard bandit algorithms. This occurs because the vanilla greedy algorithm provably converges to the optimal policy with some probability even for problem instances without covariate diversity; however, it achieves linear regret on average since it may fail a positive fraction of the time. Greedy-First leverages this observation by following a purely greedy algorithm until it detects that this approach has failed. Thus, in any bandit problem, the Greedy-First policy explores less on average than standard algorithms that always explore. Simulations confirm our theoretical results, and demonstrate that Greedy-First outperforms existing contextual bandit algorithms even when covariate diversity is not met.
Finally, Greedy-First provides decision-makers with a natural interpretation for exploration. The hypothesis test for adopting exploration only triggers when an arm has not received sufficiently diverse samples; at this point, the decision-maker can choose to explore that arm by assigning it random individuals, or to discard it based on current estimates and continue with a greedy approach. In this way, Greedy-First reduces the opaque nature of experimentation, which we believe can be valuable for aiding the adoption of bandit algorithms in practice.
1.1 Related Literature
We study sequential decision-making algorithms under the classic linear contextual bandit framework, which has been extensively studied in the computer science, operations, and statistics literature (see Chapter 4 of Bubeck and Cesa-Bianchi (2012) for an informative review). A key feature of this setting is the presence of bandit feedback, i.e., the decision-maker only observes feedback for her chosen decision and does not observe counterfactual feedback from other decisions she could have made; this obstacle inspires the exploration-exploitation tradeoff in bandit problems.
The contextual bandit setting was first introduced by Auer (2002) through the LinRel algorithm and was subsequently improved through the OFUL algorithm by Dani et al. (2008) and the LinUCB algorithm by Chu et al. (2011). More recently, Abbasi-Yadkori et al. (2011) proved an upper bound of regret after time periods when contexts are -dimensional. While this literature often allows for arbitrary (adversarial) context sequences, we consider the more restricted setting where contexts are generated i.i.d. from some unknown distribution. This additional structure is well-suited to certain applications (e.g., clinical trials on treatments for a non-infectious disease) and allows for improved regret bounds in (see Goldenshluger and Zeevi 2013, who prove an upper bound of regret), and more importantly, allows us to delve into the performance of exploration-free policies which have not been analyzed previously.
Recent work has applied contextual bandit techniques for personalization in a variety of applications such as healthcare (Bastani and Bayati 2020, Tewari and Murphy 2017, Mintz et al. 2017, Kallus and Zhou 2018, Chick et al. 2018, Zhou et al. 2019), recommendation systems (Chu et al. 2011, Kallus and Udell 2016, Agrawal et al. 2019, Bastani et al. 2018), and dynamic pricing (Cohen et al. 2016, Qiang and Bayati 2016, Javanmard and Nazerzadeh 2019, Ban and Keskin 2020, Bastani et al. 2019). However, this substantial literature requires exploration. Exploration-free greedy policies are desirable in practical settings where exploration may be costly or unethical.
Greedy Algorithms.
A related literature studies greedy (but not exploration-free) algorithms in discounted Bayesian multi-armed bandit problems. The seminal paper by Gittins (1979) showed that greedily applying an index policy is optimal for a classical multi-armed bandit in Bayesian regret (with a known prior over the unknown parameters). Woodroofe (1979) and Sarkar (1991) extend this result to a Bayesian one armed bandit with a single i.i.d. covariate when the discount factor approaches 1, and Wang et al. (2005a, b) generalize this result with a single covariate and two arms. Mersereau et al. (2009) further model known structure between arm rewards. However, these policies are not exploration-free; in particular, the Gittins index of an arm is not simply the arm parameter estimate, but includes an additional factor that implicitly captures the value of exploration for under-sampled arms. Recent work has shown a sharp equivalence between the UCB policy (which incorporates exploration) and the Gittins index policy as the discount factor approaches one (Russo 2019). In contrast, we consider a greedy policy with respect to unbiased arm parameter estimates, i.e., without incorporating any exploration. It is surprising that such a policy can be effective; in fact, we show that it is not rate optimal in general, but is rate optimal for the linear contextual bandit if there is sufficient randomness in the context distribution.
It is also worth noting that, unlike the literature above, we consider undiscounted minimax regret with unknown and deterministic arm parameters. Gutin and Farias (2016) show that the Gittins analysis does not succeed in minimizing Bayesian regret over all sufficiently large horizons, and propose “optimistic” Gittins indices (which incorporate additional exploration) to solve the undiscounted Bayesian multi-armed bandit.
There are also technical parallels between our work and the analysis of greedy policies in the dynamic pricing literature (Lattimore and Munos 2014, Broder and Rusmevichientong 2012). When there is no context, the greedy algorithm provably converges to a suboptimal price with nonzero probability (den Boer and Zwart 2013, Keskin and Zeevi 2014, 2018). However, in the presence of contexts, Qiang and Bayati (2016) show that changes in the demand environment can induce natural exploration for an exploration-free greedy algorithm, thereby ensuring asymptotically optimal performance. Our work significantly differs from this line of analysis since we need to learn multiple reward functions (for each arm) simultaneously. Specifically, in dynamic pricing, the decision-maker always receives feedback from the true demand function; in contrast, in the contextual bandit, we only receive feedback from a decision if we choose it, thereby complicating the analysis. As a result, the greedy policy is always rate optimal in the setting of Qiang and Bayati (2016), but only rate optimal in the presence of covariate diversity in our setting.
Covariate Diversity.
The adaptive control theory literature has studied “persistent excitation”: for linear models, if the sample path of the system satisfies this condition, then the minimum eigenvalue of the covariance matrix grows at a suitable rate, implying that the parameter estimates converge over time (Narendra and Annaswamy 1987, Nguyen 2018). Thus, if persistent excitation holds for each arm, we will eventually recover the true arm rewards. However, the problem remains to derive policies that ensure that such a condition holds for each (optimal) arm; classical bandit algorithms achieve this goal with high probability by incorporating exploration for under-sampled arms. Importantly, a greedy policy that does not incorporate exploration may not satisfy this condition, e.g., the greedy policy may “drop” an arm. The covariate diversity assumption ensures that there is sufficient randomness in the observed contexts, thereby exogenously ensuring that persistent excitation holds for each arm regardless of the sample path taken by the bandit algorithm.
Conservative Bandits.
Our approach is also related to recent literature on designing conservative bandit algorithms (Wu et al. 2016, Kazerouni et al. 2017) that operate within a safety margin, i.e., the regret is constrained to stay below a certain threshold that is determined by a baseline policy. This literature proposes algorithms that restrict the amount of exploration (similar to the present work) in order to satisfy a safety constraint. Wu et al. (2016) studies the classical multi-armed bandit, and Kazerouni et al. (2017) generalizes these results to the contextual linear bandit.
Additional Related Work.
Since the first draft of this paper appeared online, there have been two follow-up papers that cite our work and provide additional theoretical and empirical validation for our results. Kannan et al. (2018) consider the case where an adversary selects the observed contexts, but these contexts are then perturbed by white noise; they find that the greedy algorithm can be rate optimal in this setting even for small perturbations. Bietti et al. (2018) perform an extensive empirical study of contextual bandit algorithms on datasets that are publicly available on the OpenML platform. These datasets arise from a variety of applications including medicine, natural language, and sensors. Bietti et al. (2018) find that the greedy algorithm outperforms a wide range of bandit algorithms in cumulative regret on more that datasets. This study provides strong empirical validation of our theoretical findings.
1.2 Main Contributions and Organization of the Paper
We begin by studying conditions under which the greedy algorithm performs well. In §2, we introduce the covariate diversity condition (Assumption 2.3), and show that it holds for a general class of continuous and discrete context distributions. In §3, we show that when covariate diversity holds, the greedy policy is asymptotically optimal for a two-armed contextual bandit with linear rewards (Theorem 3.3); this result is extended to rewards given by generalized linear models in Proposition 3.12. For problem instances with more than two arms or where covariate diversity does not hold, we prove that the greedy algorithm is asymptotically optimal with some probability, and we provide a lower bound on this probability (Theorem 3.14).
Building on these results, in §4, we introduce the Greedy-First algorithm that uses observed contexts and rewards to determine whether the greedy algorithm is failing or not via a hypothesis test. If the test detects that the greedy steps are not receiving sufficient exploration, the algorithm switches to a standard exploration-based algorithm. We show that Greedy-First achieves rate optimal regret bounds without our additional assumptions on covariate diversity or number of arms. More importantly, we prove that Greedy-First remains purely greedy (while achieving asymptotically optimal regret) for almost all problem instances for which a pure greedy algorithm is sufficient (Theorem 4.2). Finally, for problem instances with more than two arms or where covariate diversity does not hold, we prove that Greedy-First remains exploration-free and rate optimal with some probability, and we provide a lower bound on this probability (Theorem 4.4). This result implies that Greedy-First reduces exploration on average compared to standard bandit algorithms.
Finally, in §5, we run simulations on synthetic and real datasets to verify our theoretical results. We find that the greedy algorithm outperforms standard bandit algorithms when covariate diversity holds, but can perform poorly when this assumption does not hold. However, Greedy-First outperforms standard bandit algorithms even in the absence of covariate diversity, while remaining competitive with the greedy algorithm in the presence of covariate diversity. Thus, Greedy-First provides a desirable compromise between avoiding exploration and learning the true policy.
2 Problem Formulation
We consider a -armed contextual bandit for time steps, where is unknown. Each arm is associated with an unknown parameter . For any integer , let denote the set . At each time , we observe a new individual with context vector . We assume that is a sequence of i.i.d. samples from some unknown distribution that admits probability density with respect to the Lebesgue measure. If we pull arm , we observe a stochastic linear reward (in §3.4, we discuss how our results can be extended to generalized linear models)
[TABLE]
where are independent -subgaussian random variables (see Definition 2.1 below).
Definition 2.1
A random variable is -subgaussian if for all we have .
We seek to construct a sequential decision-making policy that learns the arm parameters over time in order to maximize expected reward for each individual.
We measure the performance of by its cumulative expected regret, which is the standard metric in the analysis of bandit algorithms (Lai and Robbins 1985, Auer 2002). In particular, we compare ourselves to an oracle policy , which knows the arm parameters in advance. Upon observing context , the oracle will always choose the best expected arm . Thus, if we choose an arm at time , we incur instantaneous expected regret
[TABLE]
which is simply the expected difference in reward between the oracle’s choice and our choice. We seek to minimize the cumulative expected regret . In other words, we seek to mimic the oracle’s performance by gradually learning the arm parameters.
Additional Notation:
Let be the closed ball of radius around the origin in defined as , and let the volume of a set be .
2.1 Assumptions
We now describe the assumptions required for our regret analysis. Some assumptions will be relaxed in later sections of the paper as noted below.
Our first assumption is that the contexts as well as the arm parameters are bounded. This ensures that the maximum regret at any time step is bounded. This is a standard assumption made in the bandit literature (see e.g., Dani et al. 2008). {assumption}[Parameter Set]
There exists a positive constant such that the context probability density has no support outside the ball of radius , i.e., for all . There also exists a constant such that for all .
Second, we make an assumption on the margin condition (defined below) satisfied by the context probability density (Tsybakov et al. 2004).
Definition 2.2** **(-Margin Condition)
For , we say that the context probability density satisfies the -margin condition, if there exists a constant such that for each :
[TABLE]
Note that any context probability density satisfies the margin condition for by taking ; higher values of impose stronger assumptions on . As shown by Goldenshluger and Zeevi (2009), the convergence rate of bandit algorithms depends on , i.e., when , they prove matching upper and lower bounds of regret, but when , the regret can be as high as . This is because rules out unusual context distributions that become unbounded near the decision boundary (which has zero measure), thereby making learning difficult.
Our second assumption is that satisfies . We impose this assumption for simplicity of the proofs; however, all our results carry through straightforwardly for general values of . To illustrate, we prove convergence of the greedy algorithm for any (see Corollary 3.4 to Theorem 3.3). {assumption}[Margin Condition]
There exists a constant such that for each :
[TABLE]
Remark 2.3
The bandit literature distinguishes between problem-dependent and independent bounds (see, e.g., Abbasi-Yadkori et al. 2011). Specifically, in the problem-dependent case, they assume that there exists some gap between the rewards of the optimal arm and all other arms. Generally, the regret scales as in the problem-dependent case and in the problem-independent case. The problem-independent case corresponds to in the worst case; the problem-dependent case corresponds to when since satisfies \mathbb{P}_{X}\Big{[}0<|X^{\top}(\beta_{1}-\beta_{2})|\leq\Delta\Big{]}=0. As noted earlier, we prove convergence of the greedy algorithm under covariate diversity in both settings (see Corollary 3.4).
Thus far, we have made generic assumptions that are standard in the bandit literature. Our third assumption introduces the covariate diversity condition, which is essential for proving that the greedy algorithm always converges to the optimal policy. This condition guarantees that no matter what our arm parameter estimates are at time , there is a diverse set of possible contexts (supported by the context probability density ) under which each arm may be chosen.
{assumption}
[Covariate Diversity]
There exists a positive constant such that for each vector the minimum eigenvalue of is at least , i.e.,
[TABLE]
Assumption 2.3 holds for a general class of distributions. For instance, if the context probability density is bounded below by a nonzero constant in an open set around the origin, then it would satisfy covariate diversity. This includes common distributions such as the uniform or truncated gaussian distributions. Furthermore, discrete distributions such as the classic Rademacher distribution on binary random variables also satisfy covariate diversity.
Remark 2.4
As discussed in the related literature, the adaptive control theory literature has studied “persistent excitation,” which is reminiscent of the covariate diversity condition without the indicator function . If persistent excitation holds for each arm in a given sample path, then the minimum eigenvalue of the corresponding covariance matrix grows at a suitable rate, and the arm parameter estimate converges over time. However, a greedy policy that does not incorporate exploration may not satisfy this condition, e.g., the greedy policy may “drop” an arm. Assumption 2.3 ensures that there is sufficient randomness in the observed contexts, thereby exogenously ensuring that persistent excitation holds for each arm (see Lemma 3.7), regardless of the sample path taken by the bandit algorithm.
2.2 Examples of Distributions Satisfying Assumptions 2.1-2.3
While Assumptions 2.1-2.2 are generic, it is not straightforward to verify Assumption 2.3. The following lemma provides sufficient conditions (that are easier to check) that guarantee Assumption 2.3.
Lemma 2.5
If there exists a set that satisfies conditions (a), (b), and (c) given below, then satisfies Assumption 2.3.
- (a)
* is symmetric around the origin; i.e., if then .*
- (b)
There exist positive constants such that for all , .
- (c)
There exists a positive constant such that . For discrete distributions, the integral is replaced with a sum.
We now use Lemma 2.5 to demonstrate that covariate diversity holds for a wide range of continuous and discrete context distributions, and we explicitly provide the corresponding constants. It is straightforward to verify that these examples (and any product of their distributions) also satisfy Assumptions 2.1 and 2.2.
Uniform Distribution. Consider the uniform distribution over an arbitrary bounded set that contains the origin. Then, there exists some such that . Taking , we note that conditions (a) and (b) of Lemma 2.5 follow immediately. We now check condition (c) by first stating the following lemma (see Appendix 7 for proof):
Lemma 2.6
* for any .*
By definition, for all , and . Applying Lemma 2.6, we see that condition (c) of Lemma 2.5 holds with constant . 2. 2.
Truncated Multivariate Gaussian Distribution. Let be a multivariate Gaussian distribution , truncated to [math] for all . The density after renormalization is
[TABLE]
Taking , conditions (a) and (b) of Lemma 2.5 follow immediately. Condition (c) of Lemma 2.5 holds with constant
[TABLE]
as shown in Lemma 7.3 in Appendix 7. 3. 3.
Gibbs Distributions with Positive Covariance. Consider the set equipped with a discrete probability density , which satisfies
[TABLE]
for any . Here, are (deterministic) parameters, and is a normalization term known as the partition function in the statistical physics literature. We define , satisfying conditions (a) and (b) of Lemma 2.5. Furthermore, condition (c) follows by definition since the covariance of the distribution is positive-definite. This class of distributions includes the well-known Rademacher distribution (by setting all ).
A special case under which the conditions in Lemma 2.5 hold is when is the entire support of the density ; this is the case in the Gaussian and Gibbs distributions, where and respectively. Now, let be a random vector that satisfies this special case and has mean [math]. Let be another vector that is independent of and satisfies the general form of Lemma 2.5. Then it is easy to see that also satisfies the conditions in Lemma 2.5: parts (a) and (b) clearly hold; to see why (c) holds, note that the cross diagonal entries in are zero since has mean [math]. This construction illustrates how covariate diversity works for distributions that contain a mixture of discrete and continuous components.
3 Greedy Bandit
Notation. Let the design matrix X be the matrix whose rows are . Similarly, for , let be the length vector of potential outcomes . Since we only obtain feedback when arm is played, entries of may be missing. For any let be the set of times when arm was played within the first time steps. We use the notation and to refer to the design matrix, the outcome vector, and vector of idiosyncratic shocks respectively, for observations restricted to time periods in . We estimate at time based on and , using ordinary least squares (OLS) regression that is defined below. We denote this estimator , or for short.
Definition 3.1** **(OLS Estimator)
For any and , the OLS estimator is , which is equal to when is invertible.
We now describe the greedy algorithm and its performance guarantees under covariate diversity.
3.1 Algorithm
At each time step, we observe a new context and use the current arm estimates to play the arm with the highest estimated reward, i.e., . Upon playing arm , a reward is observed. We then update our estimate for arm but we need not update the arm parameter estimates for other arms as for . The update formula is given by
[TABLE]
We do not update the parameter of arm if is not invertible (see Remark 3.2 below for alternative choices). The pseudo-code for the algorithm is given in Algorithm 1.
Remark 3.2
In Algorithm 1, we only update the arm parameter from its (arbitrary) initial value of [math] when the covariance matrix is invertible. However, one can alternatively update the parameter using ridge regression or a pseudo inverse to improve empirical performance. Our theoretical analysis is unaffected by this choice — as we will show in Lemma 3.7, no matter what estimator we use, covariate diversity ensures that the probability that these covariance matrices are singular is upper bounded by , thereby contributing at most an additive constant factor to the cumulative regret (the second term in Lemma 3.11).
3.2 Performance of Greedy Bandit with Covariate Diversity
We now establish a finite-sample upper bound on the cumulative expected regret of the Greedy Bandit for the two-armed contextual bandit when covariate diversity is satisfied.
Theorem 3.3
If and Assumptions 2.1-2.3 are satisfied, the cumulative expected regret of the Greedy Bandit at time is at most
[TABLE]
where the constant is defined in Assumption 2.2 and
[TABLE]
We prove an analogous result for the greedy algorithm in the case where arm rewards are given by generalized linear models (see §3.4 and Proposition 3.12 for details).
Goldenshluger and Zeevi (2013) established a lower bound of for any algorithm in a two-armed contextual bandit. While they do not make Assumption 2.3, the distribution used in their proof satisfies Assumption 2.3; thus their result applies to our setting. Combined with our upper bound (Theorem 3.3), we conclude that the Greedy Bandit is rate optimal111Our upper bound in Theorem 3.3 scales as in the context dimension . This is because the term scales as for standard distributions satisfying covariate diversity (e.g., truncated multivariate gaussian or uniform distribution). Thus, our upper bound for the Greedy Bandit is slightly worse (by a factor of ) than the upper bound of established in Bastani and Bayati (2020) for the OLS Bandit..
We can easily remove Assumption 2.2 and extend Theorem 3.3 to general margin conditions (i.e., in Definition 2.2) in order to cover problem-independent settings as well:
Corollary 3.4
Let denote the general margin condition satisfied by (Definition 2.2). If and only Assumptions 2.1 and 2.3 are satisifed, the cumulative expected regret of the Greedy Bandit is at most
[TABLE]
The proof of this result is given in Appendix 10.1. In other words, the Greedy Bandit continues to be rate optimal under general margin conditions for the two-armed contextual bandit as long as covariate diversity is satisfied.
3.3 Proof of Theorem 3.3
Notation. Let denote the true set of contexts where arm is optimal. Then, let denote the estimated set of contexts at time where arm appears optimal; in other words, if the context , then the greedy policy will choose arm at time (since we assume without loss of generality that ties are broken randomly as selected by and thus, and partition the context space ).
For any , let denote the -algebra containing all observed information up to time before taking an action; thus, our policy is -measurable. Furthermore, let which is the -algebra containing all observed information before time .
Define as the sample covariance matrix for observations from arm up to time . We may compare this to the expected covariance matrix for arm under the greedy policy, defined as .
Proof Strategy. Intuitively, covariate diversity (Assumption 2.3) guarantees that there is sufficient randomness in the observed contexts, which creates natural “exploration.” In particular, no matter what our current arm parameter estimates are at time , each arm will be chosen by the greedy policy with at least some constant probability (with respect to ) depending on the observed context. We formalize this intuition in the following lemma.
Lemma 3.5
Given Assumptions 2.1 and 2.3, the following holds for any :
[TABLE]
Proof 3.6
Proof of Lemma 3.5. For any observed context , note that by Assumption 2.1. Re-stating Assumption 2.3 for each , we can write
[TABLE]
since the indicator function and are both nonnegative. \Halmos
Taking , Lemma 3.5 implies that arm 1 will be pulled with probability at least at each time ; the claim holds analogously for arm 2. Thus, each arm will be played at least times in expectation. However, this is not sufficient to guarantee that each arm parameter estimate converges to the true parameter . In Lemma 3.7, we establish a sufficient condition for convergence.
First, we show that covariate diversity guarantees that the minimum eigenvalue of each arm’s expected covariance matrix under the greedy policy grows linearly with . This result implies that not only does each arm receive a sufficient number of observations under the greedy policy, but also that these observations are sufficiently diverse (in expectation). Next, we apply a standard matrix concentration inequality (see Lemma 8.3 in Appendix 8) to show that the minimum eigenvalue of each arm’s sample covariance matrix also grows linearly with . This will guarantee the convergence of our regression estimates for each arm parameter.
Lemma 3.7
Take . Given Assumptions 2.1 and 2.3, the following holds for the minimum eigenvalue of the empirical covariance matrix of each arm :
[TABLE]
Proof 3.8
Proof of Lemma 3.7. Without loss of generality, let . For any , let ; by the greedy policy, we pull arm 1 if and arm 2 if (ties are broken randomly using a fair coin flip ). Thus, the estimated set of optimal contexts for arm 1 is
[TABLE]
First, we seek to bound the minimum eigenvalue of the expected covariance matrix . Expanding one term in the sum, we can write
[TABLE]
where the last line follows from Assumption 2.3. Since the minimum eigenvalue function is concave over positive semi-definite matrices, we can write
[TABLE]
Next, we seek to use matrix concentration inequalities (Lemma 8.3 in Appendix 8) to bound the minimum eigenvalue of the sample covariance matrix . To apply the concentration inequality, we also need to show an upper bound on the maximum eigenvalue of ; this follows trivially from Assumption 2.1 using the Cauchy-Schwarz inequality:
[TABLE]
We can now apply Lemma 8.3, taking the finite adapted sequence to be , so that and . We also take and . Thus, we have
[TABLE]
using the fact . As we showed earlier, . This proves the result. \Halmos
Next, Lemma 3.9 guarantees with high probability that each arm’s parameter estimate has small error with respect to the true parameter if the minimum eigenvalue of the sample covariance matrix has a positive lower bound. Note that we cannot directly use results on the convergence of the OLS estimator since the set of samples from arm at time are not i.i.d. (we use the arm estimate to decide whether to play arm at time ; thus, the samples in are correlated.). Instead, we use a Bernstein concentration inequality to guarantee convergence with adaptive observations. In the following lemma, note that is any deterministic upper bound on the total number of times that arm is pulled until time . In the proof of Lemma 3.11, we will take ; however, we state the lemma for general for later use in our probabilistic guarantees.
Lemma 3.9
Taking and , we have for all ,
[TABLE]
Proof 3.10
Proof of Lemma 3.9.
We begin by noting that if the event holds, then
[TABLE]
As a result, we can write
[TABLE]
where denotes the column of . We can expand
[TABLE]
For simplicity, define . First, note that is -subgaussian, since is -subgaussian and . Next, note that and are both measurable; taking the expectation gives . Thus, the sequence is a martingale difference sequence adapted to the filtration . Applying a standard Bernstein concentration inequality (see Lemma 8.1 in Appendix 8), we can write
[TABLE]
where is an upper bound on the number of nonzero terms in above sum, i.e., an upper bound on . This yields the desired result. \Halmos
To summarize, Lemma 3.7 provides a lower bound (with high probability) on the minimum eigenvalue of the sample covariance matrix. Lemma 3.9 states that if such a bound holds on the minimum eigenvalue of the sample covariance matrix, then the estimated parameter is close to the true (with high probability). Having established convergence of the arm parameters under the Greedy Bandit, one can use a standard peeling argument to bound the instantaneous expected regret of the Greedy Bandit algorithm (the remaining proof is given in Appendix 9).
Lemma 3.11
Define . Then, the instantaneous expected regret of the Greedy Bandit at time satisfies
[TABLE]
where , is defined in Assumption 2.2, and is defined in Theorem 3.3.
Note that can be upper bounded using Lemma 3.7. Substituting this in the upper bound derived on in Lemma 3.11, and using finishes the proof of Theorem 3.3.
3.4 Generalized Linear Rewards
In this section, we discuss how our results generalize when the arm rewards are given by a generalized linear model (GLM). Now, upon playing arm after observing context , the decision-maker realizes a reward with expectation , where is the inverse link function. For instance, in logistic regression, this would correspond to a binary reward with ; in Poisson regression, this would correspond to an integer-valued reward with ; in linear regression, this would correspond to .
In order to describe the greedy policy in this setting, we give a brief overview of the exponential family, generalized linear model, and maximum likelihood estimation.
Exponential family.
A univariate probability distribution belongs to the canonical exponential family if its density with respect to a reference measure (e.g., Lebesgue measure) is given by
[TABLE]
where is the underlying real-valued parameter, and are real-valued functions, and is assumed to be twice continuously differentiable. For simplicity, we assume the reference measure is the Lebesgue measure. It is well known that if is distributed according to the above canonical exponential family, then it satisfies and , where and denote the first and second derivatives of the function with respect to , and is strictly convex (see e.g., Lehmann and Casella 1998).
Generalized linear model (GLM).
The natural connection between exponential families and GLMs is provided by assuming that the density of for the context and arm is given by . where is defined in . In other words, the reward upon playing arm for context is with density
[TABLE]
Using the aforementioned properties of the exponential family, , i.e., the link function . This implies that is continuously differentiable and its derivative is . Thus, is strictly increasing since is strictly convex.
Maximum likelihood estimation.
Suppose that we have samples from a distribution with density . The maximum likelihood estimator of based on this sample is given by
[TABLE]
Since is strictly convex (so is strictly concave), the solution to (7) can be obtained efficiently (see e.g., McCullagh and Nelder 1989). It is not hard to see that whenever is positive definite, this solution is unique (see Appendix 10.2 for a proof). We denote this unique solution by .
Now we are ready to generalize the Greedy Bandit algorithm when the arm rewards are given by a GLM. Using similar notation as in the linear reward case, given the estimates at time , the greedy policy plays the arm that maximizes expected estimated reward, i.e.,
[TABLE]
Since is a strictly increasing function, this translates to .
Next, we state the following result (proved in Appendix 10.2) that Algorithm 2 achieves logarithmic regret when and the covariate diversity assumption holds.
Proposition 3.12
Consider arm rewards given by a GLM with -subgaussian noise . Define . If and Assumptions 2.1-2.3 are satisfied, the cumulative expected regret of Algorithm 2 at time is at most
[TABLE]
where the constant is defined in Assumption 2.2, is the Lipschitz constant of the function on the interval , and is defined as .
3.5 Performance of Greedy Bandit without Covariate Diversity
Thus far, we have shown that the greedy algorithm is rate optimal when there are only two arms and in the presence of covariate diversity in the observed context distribution. However, when these additional assumptions do not hold, the greedy algorithm may fail to converge to the true arm parameters and achieve linear regret. We now show that a greedy approach achieves rate optimal performance with some probability even when these assumptions do not hold. This result will motivate the design of the Greedy-First algorithm in §4.
Assumptions.* For the rest of the paper, we allow the number of arms , and remove Assumption 2.3 on covariate diversity. Instead, we will make the following weaker Assumption 2, which is typically made in the contextual bandit literature222This assumption is slightly different as stated than the assumptions made in prior literature; however, the assumptions are equivalent for bounded (Assumption 2.1). (see e.g., Goldenshluger and Zeevi 2013, Bastani and Bayati 2020), which allows for multiple arms, and relaxes the assumption on observed contexts (e.g., allowing for intercept terms in the arm parameters). {assumption}[Positive-Definiteness] Let and be mutually exclusive sets that include all arms. Sub-optimal arms satisfy for some and every . On the other hand, each optimal arm , has a corresponding set . Define for all . Then, there exists such that for all , .*
Algorithm.* We consider a small modification of the Greedy Bandit (Algorithm 1), by initializing each arm parameter estimate with random samples. Note that OLS requires at least samples for an arm parameter estimate to be well-defined, and Algorithm 1 does not update the arm parameter estimates from the initial ad-hoc value of [math] until this stage is reached (i.e., the covariance matrix for a given arm becomes invertible); thus, all actions up to that point are essentially random. Consequently, we argue that initializing each arm parameter with samples at the beginning is qualitatively no different than Algorithm 1. We consider general values of to study how the probabilistic guarantees of the greedy algorithm vary with the number of initial samples.*
Remark 3.13
We note that there is a class of explore-then-exploit bandit algorithms that follow a similar strategy of randomly sampling each arm for a length of time and using those estimates for the remaining horizon (Bubeck and Cesa-Bianchi 2012). However, (i) is a function of the horizon length in these algorithms (typically ) while we consider to be a (small) constant with respect to , and (ii) these algorithms do not follow a greedy strategy since they do not update the parameter estimates after the initialization phase.
Result.* The following theorem shows that the Greedy Bandit converges to the correct policy and achieves rate optimal performance with at least some problem-specific probability.*
Theorem 3.14
Under Assumptions 2.1, 2.2, and 2, Greedy Bandit achieves logarithmic cumulative regret with probability at least
[TABLE]
where the function is defined as
[TABLE]
Here denotes the matrix obtained by drawing random samples from distribution , and
[TABLE]
Proof Strategy.* The proof of Theorem 3.14 is provided in Appendix 12. We observe that if all arm parameter estimates remain within a Euclidean distance of from their true values for all time periods , then the Greedy Bandit converges to the correct policy and is rate optimal. We derive lower bounds on the probability that this event occurs using Lemma 3.9, after proving suitable lower bounds on the minimum eigenvalue of the covariance matrices. The key steps are as follows:*
Assuming that the minimum eigenvalue of the sample covariance matrix for each arm is above some threshold value , we derive a lower bound on the probability that after initialization, each arm parameter estimates lie within a ball of radius centered around the true arm parameter. 2. 2.
Next, we derive a lower bound on the probability that these estimates remain within this ball after rounds for some choice of . 3. 3.
We use the concentration result in Lemma 8.3 to derive a lower bound on the probability that the minimum eigenvalue of the sample covariance matrix of each arm in is above for any . 4. 4.
We derive a lower bound on the probability that the estimates ultimately remain inside the ball with radius . This ensures that no sub-optimal arm is played for any . 5. 5.
Summing up these probability terms implies Theorem 3.14. The parameters and can be chosen arbitrarily and we optimize over their choice.
The following Proposition 3.15 illustrates some of the properties of the function in Theorem 3.14 with respect to problem-specific parameters. The proof is provided in Appendix 12.
Proposition 3.15
The function defined in Equation is non-increasing with respect to and ; it is non-decreasing with respect to , and . Furthermore, the limit of this function when goes to zero is
[TABLE]
In other words, the greedy algorithm is more likely to succeed when there is less noise and when there are fewer arms; it is also more likely to succeed with additional initialization samples, when the optimal arms each have a larger probability of being the best arm under , and when the sub-optimal arms are worse than the optimal arms by a larger margin. Intuitively, these conditions make it easier for the Greedy Bandit to avoid “dropping a good arm” early on, which would result in its convergence to the wrong policy. As the noise goes to zero, the greedy algorithm always succeeds as long as the sample covariance matrix for each of the arms is positive definite after the initialization periods.
In Corollary 3.16, we simplify the expression in Theorem 3.14 for better readability. However, the simplified expression leads to poor tail bounds when is close to , while the general expression in Theorem 3.14 works when as demonstrated later in §4.3 (see Figure 1).
Corollary 3.16
Under the assumptions of Theorem 3.14, Greedy Bandit achieves logarithmic cumulative regret with probability at least
[TABLE]
where function is defined as .
To summarize, these probabilistic guarantees on the success of Greedy Bandit suggest that a greedy approach can be effective and rate optimal in general with at least some probability. Therefore, in the next section, we introduce the Greedy-First algorithm which executes a greedy strategy and only resorts to forced exploration when the observed data suggests that the greedy updates are not converging. This helps eliminate unnecessary exploration with high probability.
4 Greedy-First Algorithm
As noted in Theorem 3.3, the optimality of the Greedy Bandit requires that there are only two arms and that the context distribution satisfies covariate diversity. The latter condition rules out some standard settings, e.g., the arm rewards cannot have an intercept term (since the addition of a one to every context vector would violate Assumption 2.3). While there are many examples that satisfy these conditions (see §2.2), the decision-maker may not know a priori whether a greedy algorithm is appropriate for her particular setting. Thus, we introduce the Greedy-First algorithm (Algorithm 3), which is rate optimal without these additional assumptions, but seeks to use the greedy algorithm without forced exploration when possible.
4.1 Algorithm
The Greedy-First algorithm has two inputs and . It starts by following the greedy algorithm up to time , after which it iteratively checks whether all the arm parameter estimates are converging to their true values at a suitable rate. A sufficient statistic for checking this is simply the minimum eigenvalue of the sample covariance matrix of each arm; if this value is above the threshold of , then greedy estimates are converging with high probability. On the other hand, if this condition is not met, the algorithm switches to a standard bandit algorithm with forced exploration. We choose the OLS Bandit algorithm (introduced by Goldenshluger and Zeevi (2013)* for two arms and extended to the general setting by Bastani and Bayati (2020)**), provided in Appendix 11.*
Remark 4.1
Greedy-First can switch to any contextual bandit algorithm (e.g., OFUL by Abbasi-Yadkori et al. (2011) or Thompson sampling by Agrawal and Goyal (2013), Russo and Van Roy (2018)) instead of the OLS Bandit. Then, the assumptions used in the theoretical analysis would be replaced with analogous assumptions required by that algorithm. Our proof naturally generalizes to adopt the assumptions and regret guarantees of the new algorithm when Greedy Bandit fails.
In practice, may be an unknown constant. Thus, we suggest the following heuristic routine to estimate this parameter:
Execute Greedy Bandit for time steps. 2. 2.
Estimate using the observed data via . 3. 3.
If , this suggests that one of the arms is not receiving sufficient samples, and thus, Greedy-First will switch to OLS Bandit immediately. Otherwise, execute Greedy-First for with .
The pseudo-code for this heuristic is given in Appendix 11. The regret guarantees of Greedy-First (given in the next section) are always valid, but the choice of the input parameters may affect the empirical performance of Greedy-First and the probability with which it remains exploration-free. For example, if is too small, then Greedy-First may incorrectly switch to OLS Bandit even when a greedy algorithm will converge; thus, choosing is advisable.
4.2 Regret Analysis of Greedy-First
As noted in §3.5, we replace the more restrictive assumption on covariate diversity (Assumption 2.3) with a more standard assumption made in the bandit literature (Assumption 2.2). Theorem 4.2 establishes an upper bound of on the expected cumulative regret of Greedy-First. Furthermore, we establish that Greedy-First remains purely greedy with high probability when there are only two arms and covariate diversity is satisfied.
Theorem 4.2
The cumulative expected regret of Greedy-First at time is at most
[TABLE]
where , is the constant defined in Theorem 3.3, and is the coefficient of in the upper bound of the regret of the OLS Bandit algorithm.
Furthermore, if Assumption 2.3 is satisfied (with the specified parameter ) and , then the Greedy-First algorithm will purely execute the greedy policy (and will not switch to the OLS Bandit algorithm) with probability at least , where , and . Note that can be made arbitrarily small since is an input parameter to the algorithm.
The key insight to this result is that the proof of Theorem 3.3 only requires Assumption 2.3 in the proof of Lemma 3.7. The remaining steps of the proof hold without the assumption. Thus, if the conclusion of Lemma 3.7, holds at every , then we are guaranteed at most regret by Theorem 3.3, regardless of whether Assumption 2.3 holds.
Proof 4.3
Proof of Theorem 4.2. First, we will show that Greedy-First achieves asymptotically optimal regret. Note that the expected regret during the first rounds is upper bounded by . For the period we consider two cases: (1) the algorithm pursues a purely greedy strategy, i.e., , or (2) the algorithm switches to the OLS Bandit algorithm, i.e., .
Case 1: By construction, we know that , for all . This is because Greedy-First only switches when the minimum eigenvalue of the sample covariance matrix for some arm is less than . Therefore, if the algorithm does not switch, it implies that the minimum eigenvalue of each arm’s sample covariance matrix is greater that or equal to for all values of . Then, the conclusion of Lemma 3.7 holds in this time range ( holds for all ). Consequently, even if Assumption 2.3 does not hold and , Lemma 3.11 holds and provides an upper bound on the expected regret . This implies that the regret bound of Theorem 3.3, after multiplying by , holds for Greedy-First. Therefore, Greedy-First is guaranteed to achieve regret in the period for some constant that depends only on and . Hence, the regret in this case is upper bounded by .
Case 2: Once again, by construction, we know that for all before the switch. Then, using the same argument as in Case 1, Theorem 3.3 guarantees that we achieve at most regret for some constant over the interval . Next, Theorem 2 of Bastani and Bayati (2020) guarantees that, under Assumptions 2.1, 2.2 and 2, the OLS Bandit’s cumulative regret in the interval is upper bounded by for some constant . Thus, the total regret is at most . Note that although the switching time is a random variable, the upper bound on the cumulative regret holds uniformly regardless of the value of .
Thus, the Greedy-First algorithm always achieves cumulative regret. Next, we prove that when Assumption 2.3 holds and , the Greedy-First algorithm maintains a purely greedy policy with high probability. In particular, Lemma 3.7 states that if the specified satisfies for each vector , then at each time ,
[TABLE]
where . Thus, by using a union bound over all arms, the probability that the algorithm switches to the OLS Bandit algorithm is at most
[TABLE]
This concludes the proof. \Halmos
4.3 Probabilistic Guarantees for Greedy-First Algorithm
The key value proposition of Greedy-First is to reduce forced exploration when possible. Theorem 4.2 established that Greedy-First eliminates forced exploration entirely with high probability when there are only two arms and when covariate diversity holds. However, a natural question might be the extent to which Greedy-First reduces forced exploration in general problem instances.
To answer this question, we leverage the probabilistic guarantees we derived for the greedy algorithm in §3.5. Note that unlike the greedy algorithm, Greedy-First always achieves rate optimal regret. We now study the probability with which Greedy-First is purely greedy under an arbitrary number of arms and the less restrictive Assumption 2.2. However, we impose that all arms are optimal for some set of contexts under , i.e., . This is because Greedy-First always switches to the OLS Bandit when an arm is sub-optimal across all contexts. In order for any algorithm to achieve logarithmic cumulative regret, sub-optimal arms must be assigned fewer samples over time and thus, the minimum eigenvalue of the sample covariance matrices of those arms cannot grow sufficiently fast; as a result, the Greedy-First algorithm will switch with probability . This may be practically desirable as the decision-maker can decide whether to “drop” the arm and proceed greedily or to use an exploration-based algorithm when the switch triggers.
Theorem 4.4
Let Assumptions 2.1, 2.2, and 2 hold and suppose that . Then, with probability at least
[TABLE]
Greedy-First remains purely greedy (does not switch to an exploration-based bandit algorithm) and achieves logarithmic cumulative regret. The function is closely related to the function from Theorem 3.14, and is defined as
[TABLE]
The proof of Theorem 4.4 is provided in Appendix 12. The steps followed are similar to that of the proof of Theorem 3.14. In the third step of the proof strategy of Theorem 3.14 (see §3.5), we used concentration results to derive a lower bound on the probability that the minimum eigenvalue of the sample covariance matrix of all arms in are above for any (note that we are assuming in this section). For Greedy Bandit, this result was only required for the played arm; in contrast, for Greedy-First to remain greedy, all arms are required to have the minimum eigenvalues of their sample covariance matrices above . This causes the difference in and since we need a union bound over all arms. The additional constraints on ensure that the Greedy-First algorithm does not switch,
The following Proposition 4.5 illustrates some of the properties of the function in Theorem 4.4 with respect to problem-specific parameters. The proof is provided in Appendix 12.
Proposition 4.5
The function defined in Equation is non-increasing with respect to and ; it is non-decreasing with respect to and . Furthermore, the limit of this function when goes to zero is
[TABLE]
where .
These relationships mirror those in Proposition 3.15, i.e., Greedy-First is more likely to remain exploration-free when Greedy Bandit is more likely to succeed. In particular, Greedy-First is more likely to avoid exploration entirely when there is less noise and when there are fewer arms; it is also more likely to avoid exploration with additional initialization samples and when the optimal arms each have a larger probability of being the best arm under . Intuitively, these conditions make it easier for the greedy algorithm to avoid “dropping” an arm, so the minimum eigenvalue of each arm’s sample covariance matrix grows at a suitable rate over time, allowing Greedy-First to remain greedy.
In Corollary 4.6, we simplify the expression in Theorem 4.4 for better readability. However, the simplified expression leads to poor tail bounds when is close to , while the general expression in Theorem 4.4 works when as demonstrated in Figure 1.
Corollary 4.6
Under the assumptions made in Theorem 4.4, Greedy-First remains purely greedy and achieves logarithmic cumulative regret with probability at least
[TABLE]
where the function is defined in Corollary 3.16.
We now illustrate the probabilistic bounds given in Theorems 3.14 and 4.4 through a simple example.
Example 4.7
Let and . Suppose that arm parameters are given by and . Furthermore, suppose that the distribution of covariates is the uniform distribution on the unit ball , implying . The constants and are chosen to satisfy Assumption 2; here, we choose , and . We then numerically plot our lower bounds on the probability of success of the Greedy Bandit (Theorem 3.14) and on the probability that Greedy-First remains greedy (Theorem 4.4) via Equations and respectively. Figure 1 depicts these probabilities as a function of the noise for several values of initialization samples .
We note that our lower bounds are very conservative, and in practice, both Greedy Bandit and Greedy-First succeed and remain exploration-free respectively with much larger probability. For instance, as observed in Example 4.7, one can optimize over the choice of and . In the next section, we verify via simulations that both Greedy Bandit and Greedy-First are successful with a higher probability than our lower bounds may suggest.
5 Simulations
We now validate our theoretical findings on synthetic and real datasets.
5.1 Synthetic Data
Linear Reward.* We compare Greedy Bandit and Greedy-First with state-of-the-art contextual bandit algorithms. These include:*
OFUL* by Abbasi-Yadkori et al. (2011), which builds on the original upper confidence bound (UCB) approach of Lai and Robbins (1985),* 2. 2.
Prior-dependent TS* by Russo and Van Roy (2014), which builds on the original Thompson sampling approach of Thompson (1933),* 3. 3.
Prior-free TS* by Agrawal and Goyal (2013), which builds on the original Thompson sampling approach of Thompson (1933), and* 4. 4.
OLS Bandit* by Goldenshluger and Zeevi (2013)**, which builds on -greedy methods.*
Prior-dependent TS requires knowledge of the prior distribution of arm parameters , while prior-free TS does not. All algorithms require knowledge of an upper bound on the noise variance . Following the setup of Russo and Van Roy (2014)**, we consider Bayes regret over randomly-generated arm parameters. In particular, for each scenario, we generate problem instances and sample the true arm parameters independently. At each time step within each instance, new context vectors are drawn i.i.d. from a fixed context distribution . We then plot the average Bayes regret across all these instances, along with the confidence interval, as a function of time with a horizon length . We take and (see Appendix 13 for simulations with other values of and ). The noise variance .
We consider four different scenarios, varying (i) whether covariate diversity holds, and (ii) whether algorithms have knowledge of the true prior. The first condition allows us to explore how the performance of Greedy Bandit and Greedy-First compare against benchmark bandit algorithms when conditions are favorable / unfavorable for the greedy approach. The second condition helps us understand how knowledge of the prior distribution and noise variance affects the performance of benchmark algorithms relative to Greedy Bandit and Greedy-First (which do not require this knowledge). When the correct prior is provided, we assume that OFUL and both versions of TS know the noise variance.
Context vectors:
For scenarios where covariate diversity holds, we sample the context vectors from a truncated Gaussian distribution, i.e., truncated to have norm at most . For scenarios where covariate diversity does not hold, we generate the context vectors the same way but we add an intercept term.
Arm parameters and prior:
For scenarios where the algorithms have knowledge of the true prior, we sample the arm parameters independently from , and provide all algorithms with knowledge of , and prior-dependent TS with the additional knowledge of the true prior distribution of arm parameters. For scenarios where the algorithms do not have knowledge of the true prior, we sample the arm parameters independently from a mixture of Gaussians, i.e., they are sampled from the distribution with probability and from the distribution with probability . However, prior-dependent TS is given the following incorrect prior distribution over the arm parameters: . The OLS Bandit parameters are set to , and for Greedy-First. None of the algorithms in this scenario are given knowledge of ; rather, this parameter is sequentially estimated over time using past data within the algorithm.
Results.
Figure 2 shows the cumulative Bayes regret of all the algorithms for the four different scenarios discussed above (with and without covariate diversity, with and without the true prior). When covariate diversity holds (a-b), the Greedy Bandit is the clear frontrunner, and Greedy-First achieves the same performance since it never switches to OLS Bandit. However, when covariate diversity does not hold (c-d), we see that the Greedy Bandit performs very poorly (achieving linear regret), but Greedy-First is the clear frontrunner. This is because the greedy algorithm succeeds a significant fraction of the time (Theorem 3.14), but fails on other instances. Thus, always following the greedy algorithm yields poor performance, but a standard bandit algorithm like the OLS Bandit explores unnecessarily in the instances where a greedy algorithm would have sufficed. Greedy-First leverages this observation by only exploring (switching to OLS Bandit) when the greedy algorithm has likely failed, thereby outperforming both Greedy Bandit and OLS Bandit. Thus, Greedy-First provides a desirable compromise between avoiding exploration and learning the true policy.
Logistic Reward.* We now move beyond linear rewards and explore how the performance of Greedy Bandit (Algorithm 2) compares to other bandit algorithms for GLM rewards when covariate diversity holds. We compare to the state-of-the-art GLM-UCB algorithm (Filippi et al. 2010), which is designed to handle GLM reward functions unlike the bandit algorithms from the previous section. Our reward is logistic, i.e, with probability and is [math] otherwise.*
We again consider Bayes regret over randomly-generated arm parameters. For each scenario, we generate problem instances (due to the computational burden of solving a maximum likelihood estimation step in each iteration) and sample the true arm parameters independently. At each time step within each instance, new context vectors are drawn i.i.d. from a fixed context distribution . We then plot the average Bayes regret across all these instances, along with the confidence interval, as a function of time with a horizon length . Once again, we sample the context vectors from a truncated Gaussian distribution, i.e., truncated to have norm at most . Note that this context distribution satisfies covariate diversity. We take , and we sample the arm parameters independently from . We consider two different scenarios for and . In the first scenario, we take ; in the second scenario, we take .
Results:
Figure 3 shows the cumulative Bayes regret of the Greedy Bandit and GLM-UCB algorithms for the two different scenarios discussed above. As is evident from these results, the Greedy Bandit far outperforms GLM-UCB. We suspect that this is due to the conservative construction of confidence sets in GLM-UCB, particularly for large values of and . In particular, the radius of the confidence set in GLM-UCB is proportional to where . Hence, the radius of the confidence set scales as , which is exponentially large in . This can be seen from the difference in Figure 3 (a) and (b); in (b),* is much larger, causing GLM-UCB’s performance to severely degrade. Although the same quantity appears in the theoretical analysis of Greedy Bandit for GLM (Proposition 3.12), the empirical performance of Greedy Bandit appears much better.*
Additional Simulations.* We explore the performance of Greedy Bandit as a function of and ; we find that the performance of Greedy Bandit improves dramatically as the dimension increases, while it degrades with the number of arms (as predicted by Proposition 3.15). We also study the dependence of the performance of Greedy-First on the input parameters (which determines when to switch) and (which are inputs to OLS Bandit after switching); we find that the performance of Greedy-First is quite robust to the choice of inputs. Note that Greedy Bandit is entirely parameter-free. These simulations can be found in Appendix 13.*
5.2 Simulations on Real Datasets
We now explore the performance of Greedy and Greedy-First with respect to competing algorithms on real datasets. As mentioned earlier, Bietti et al. (2018)* performed an extensive empirical study of contextual bandit algorithms on 524 datasets that are publicly available on the OpenML platform, and found that the greedy algorithm outperforms a wide range of bandit algorithms in cumulative regret on more that 400 datasets. We take a closer look at 3 healthcare-focused datasets ((a) EEG, (b) Eye Movement, and (c) Cardiotocography) among these. We also study the (d) warfarin dosing dataset (Consortium 2009), a publicly available patient dataset that was used by Bastani and Bayati (2020)** for analyzing contextual bandit algorithms.*
Setup:
These datasets all involve classification tasks using patient features. Accordingly, we take the number of decisions to be the number of classes, and consider a binary reward (* if we output the correct class, and [math] otherwise). The dimension of the features for datasets (a)-(d) is 14, 27, 35 and 93 respectively; similarly, the number of arms is 2, 3, 3, and 3 respectively.*
Remark 5.1
Note that we are now evaluating regret rather than Bayes regret. This is because our arm parameters are given by the true data, and are not simulated from a known prior distribution.
We compare to the same algorithms as in the previous section, i.e., OFUL, prior-dependent TS, prior-free TS, and OLS Bandit. As an additional benchmark, we also include an oracle policy, which uses the best linear model trained on all the data in hindsight; thus, one cannot perform better than the oracle policy using linear models on these datasets.
Results:
In Figure 4, we plot the regret (averaged over 100 trials with randomly permuted patients) as a function of the number of patients seen so far, along with the confidence intervals. First, in both datasets (a) and (b), we observe that Greedy Bandit and Greedy-First perform the best; Greedy-First recognizes that the greedy algorithm is converging and does not switch to an exploration-based strategy. In dataset (c), the Greedy Bandit gets “stuck” and does not converge to the optimal policy on average. Here, Greedy-First performs the best, followed closely by the OLS Bandit. This result is similar to our results in Fig 2 (c-d), but in this case, exploration appears to be necessary in nearly all instances, explaining the extremely close performance of Greedy-First and OLS Bandit. Finally, in dataset (d), we see that the Greedy Bandit performs the best, followed by Greedy-First. An interesting feature of this dataset is that one arm (high dose) is optimal for a very small number of patients; thus, dropping this arm entirely leads to better performance over a short horizon than attempting to learn its parameter. In this case, Greedy Bandit is not converging to the optimal policy since it never assigns any patient the high dose. However, Greedy-First recognizes that the high-dose arm is not getting sufficient samples and switches to an exploration-based algorithm. As a result, Greedy-First performs worse than the Greedy Bandit. However, if the horizon were to be extended333Our horizon is limited by the number of patients available in the dataset., Greedy-First and the other bandit algorithms would eventually overtake the Greedy Bandit. Alternatively, for non-binary reward functions (e.g., when cost of a mistake for high-dose patients is larger than for other patients) Greedy Bandit would perform poorly.
Looking at these results as a whole, we see that Greedy-First is a robust frontrunner. When exploration is unnecessary, it matches the performance of the Greedy Bandit; when exploration is necessary, it matches or outperforms competing bandit algorithms.
6 Conclusions and Discussions
We prove that a greedy algorithm can be rate optimal in cumulative regret for a two-armed contextual bandit as long as the contexts satisfy covariate diversity. Greedy algorithms are significantly preferable when exploration is costly (e.g., result in lost customers for online advertising or A/B testing) or unethical (e.g., personalized medicine or clinical trials). Furthermore, the greedy algorithm is entirely parameter-free, which makes it desirable in settings where tuning is difficult or where there is limited knowledge of problem parameters. Despite its simplicity, we provide empirical evidence that the greedy algorithm can outperform standard contextual bandit algorithms when the contexts satisfy covariate diversity. Even when the contexts do not satisfy covariate diversity, we prove that a greedy algorithm is rate optimal with some probability, and provide lower bounds on this probability. However, in many scenarios, the decision-makers may not know whether their problem instance is amenable to a greedy approach, and may still wish to ensure that their algorithm provably converges to the correct policy. In this case, the decision-maker may under-explore by using a greedy algorithm, while a standard bandit algorithm may over-explore (since the greedy algorithm converges to the correct policy with some probability in general). Consequently, we propose the Greedy-First algorithm, which follows a greedy policy in the beginning and only performs exploration when the observed data indicate that exploration is necessary. Greedy-First is rate optimal without the covariate diversity assumption. More importantly, it remains exploration-free when covariate diversity is satisfied, and may provably reduce exploration even when covariate diversity is not satisfied. Our empirical results suggest that Greedy-First outperforms standard bandit algorithms (e.g., UCB, Thompson Sampling, and -greedy methods) by striking a balance between avoiding exploration and converging to the correct policy.
\ACKNOWLEDGMENT
The authors gratefully acknowledge the National Science Foundation CAREER award CMMI: 1554140 and the Stanford Human Centered AI and Data Science Initiatives. This paper has also benefitted from valuable feedback from anonymous referees, and various seminar participants. They have been instrumental in guiding us to improve the paper.
\SingleSpacedXI
7 Properties of Covariate Diversity
{repeattheorem}
[Lemma 2.5] If there exists a set that satisfies conditions (a), (b), and (c) given below, then satisfies Assumption 2.3.
- (a)
* is symmetric around the origin; i.e., if then .*
- (b)
There exist positive constants such that for all , .
- (c)
There exists a positive constant such that . For discrete distributions, the integral is replaced with a sum.
Proof 7.1
Proof of Lemma 2.5. Since for all at least one of or holds, and using conditions (a), (b), and (c) of Lemma 2.5 we have:
[TABLE]
Here, the first inequality follows from the fact that is positive semi-definite, the first equality follows from condition (a) and a change of variable (), the second inequality is by condition (b), the third inequality uses which follows from condition (b), and the last inequality uses condition (c). \Halmos
We now state the proofs of lemmas that were used in §2.2.
{repeattheorem}
[Lemma 2.6] For any we have .
Proof 7.2
Proof of Lemma 2.6.
First note that is symmetric with respect to each axis, therefore the off-diagonal entries in are zero. In particular, the entry of the integral is equal to which is zero when using a change of variable that has the identity as its Jacobian and keeps the domain of integral unchanged but changes the sign of . Also, by symmetry, all diagonal entry terms are equal. In other words,
[TABLE]
Now for computing the right hand side integral, we introduce the spherical coordinate system as
[TABLE]
and the determinant of its Jacobian is given by
[TABLE]
Now, using symmetry, and summing up equation (13) with used instead of for all , we obtain
[TABLE]
Comparing this to
[TABLE]
we obtain that
[TABLE]
\Halmos
Lemma 7.3
The following inequality holds
[TABLE]
where .
Proof 7.4
Proof of Lemma 7.3.
We can lower-bound the density by the uniform density as follows. Note that we have and as a result for any satisfying we have
[TABLE]
Using this we can derive a lower bound on the desired covariance as following
[TABLE]
where we used Lemma 2.6 in the third line. This concludes the proof. \Halmos
8 Useful Concentration Results
Lemma 8.1** **(Bernstein Concentration)
Let be a martingale difference sequence, and let be -subgaussian. Then, for all we have
[TABLE]
Proof 8.2
Proof of Lemma 8.1. See Theorem 2.3 of Wainwright (2019) and let and for all . \Halmos
Lemma 8.3** **(Theorem 3.1 of Tropp (2011))
Let be a filtration and consider a finite sequence of positive semi-definite matrices with dimension adapted to this filtration. Suppose that almost surely. Define the series and . Then for all we have:
[TABLE]
9 Proof of Theorem 3.3
We first prove a lemma on the instantaneous regret of the Greedy Bandit using a standard peeling argument. The proof is adapted from Bastani and Bayati (2020)* with a few modifications.*
Notation.* We define the following events to simplify notation. For any , let*
[TABLE]
{repeattheorem}
[Lemma 3.11] The instantaneous expected regret of the Greedy Bandit at time satisfies
[TABLE]
where , is defined in Assumption 2.2, and is defined in Theorem 3.3.
Proof 9.1
Proof of Lemma 3.11. We can decompose the regret as . Now we can expand each term as
[TABLE]
For each satisfying , let us define the region where arm is superior over arm
[TABLE]
Note that we may incur a nonzero regret if or if and the tie-breaking random variable indicates an action other than as the action to be taken. It is worth mentioning that in the case we do not incur any regret if indicates arm as the action to be taken. Nevertheless, as regret is a non-negative quantity, we can write
[TABLE]
where in the second line we used a union bound, in the sixth line we used the fact that and are independent of the event which only depends on , and also a Cauchy-Schwarz inequality showing . Therefore, we need to bound the first term in above. Fix and note that when we include events and , we can use Lemma 3.9 which proves sharp concentrations for and . Let us now define the following set
[TABLE]
where . Note that since is bounded above by , the set only needs to be defined for . We can now expand the first term in Equation (9.1) for , by conditioning on as following
[TABLE]
where in the first inequality we used the fact that conditioning on , is bounded above by , in the second inequality we used the fact that the event is a subset of the event , and in the last inequality we used the margin condition given in Assumption 2.2. Now we reach to the final part of the proof, where conditioning on and we want to bound the probability that we pull a wrong arm. Note that conditioning on , the event happens only when at least one of the following two events: i) or ii) happens. This is true according to
[TABLE]
Therefore,
[TABLE]
where in the third line we used , in the fourth line we used Cauchy-Schwarz inequality. Now using the notation described in Equation this can be rewritten as
[TABLE]
in the fifth line we used the fact that both and only depend on which is independent of for all , and in the sixth line we used Lemma 3.9. We can also bound this probability by , which is better than for small values of . Hence, using we can write the regret as
[TABLE]
where we take . Note that functions and are both decreasing for and therefore
[TABLE]
Computing the above terms using integration by parts and using the inequality yields
[TABLE]
where is defined as (4). By replacing this in (9.1) and substituting we get
[TABLE]
as desired. \Halmos
Having this lemma proved, it is now fairly straightforward to prove Theorem 3.3.
Proof 9.2
Proof of Theorem 3.3.
The expected cumulative regret is the sum of expected regret for times up to time . As the regret term at time is upper bounded by and as , by using Lemma 3.7 and Lemma 3.11 we can write
[TABLE]
finishing up the proof. \Halmos
10 General margin condition and nonlinear rewards
10.1 Proof of Corollary 3.4
We now analyze the regret of Greedy Bandit for more general values of the margin condition parameter satisfied by the context probability density (recall Definition 2.2 in §2.1).
Proof 10.1
Proof of Corollary 3.4. This corollary is easily implied from Lemma 3.11 and Theorem 3.3 with a very slight modification. Note that all the arguments in Lemma 3.11 hold and the only difference is where we want to bound the probability in Equation (9.1). In this Equation, if we use the -margin bound as
[TABLE]
we obtain that
[TABLE]
which turns the regret bound in Equation (9.1) into
[TABLE]
Now we claim that the above summation has an upper bound that only depends on and . If we prove this claim, the dependency of the regret bound with respect to can only come from the term and therefore we can prove the desired asymptotic bounds. For proving this claim, consider the summation above and let . Recall from the proof of Lemma 3.11 that for we have . Hence, for each using we have
[TABLE]
Furthermore, all the terms corresponding to are upper bounded by . Therefore, the summation in (21) is bounded above by
[TABLE]
for some function . This is true according to the fact that is the maximum of , that only depends on , and that only depends on . In above we also used the well-known identity that . Now replacing in the Equation (21) and putting together all the constants we reach to
[TABLE]
for some function .
The last part of the proof is summing up the instantaneous regret terms for . Note that , and using Lemma 3.7 for , we can bound the probabilities by and therefore
[TABLE]
Now note that the integral of over the interval satisfies
[TABLE]
which yields the desired result. \Halmos
10.2 Proof of Proposition 3.12
Uniqueness of solution of Equation (7).* We first prove that the solution to maximum likelihood equation in Equation (7) is unique whenever the design matrix is positive definite. The first order optimality condition in Equation (7) implies that*
[TABLE]
Now suppose that there are two solutions to the above equation, namely and . Then, we can write
[TABLE]
Using the mean-value theorem, for each we have
[TABLE]
where belongs to the line connecting . Replacing this in above equation implies that
[TABLE]
Note that is strictly increasing meaning that is always positive. Therefore, letting , we have that
[TABLE]
Therefore, if the design matrix is positive definite, so is . Hence, Equation (23) implies that .
Proof of Proposition 3.12.* We first state and prove the following lemma:*
Lemma 10.2
Consider the generalized linear model with the inverse link function . Suppose that we have samples , where , where and . Furthermore, assume that the design matrix is positive definite. Let be the (unique) solution to the Equation and let be an arbitrary positive number. Recall that and suppose , then
[TABLE]
The proof of Lemma 10.2 is adapted from Chen et al. (1999)**. We use the following lemma from their paper:
Lemma 10.3** **(Chen et al. 1999)
Let be a smooth injection from to with . Define and . Then, implies that
- (i)
, 2. (ii)
**
Proof 10.4
Proof of Lemma 10.2. Note that is the solution to the Equation (22) and therefore
[TABLE]
Using the mean-value theorem for any and we have
[TABLE]
where is a point that lies on the line segment between and . Define
[TABLE]
As , is an injection from to satisfying . Consider the sets and . If , for each lies on the line segment between and and therefore we have according to the Cauchy-Schwarz inequality. Then for each
[TABLE]
or in other words . In particular, for any we have . Therefore, letting , Lemma 10.3 implies that . Note that if we let , then by the assumption of lemma and hence there exists satisfying , i.e., . Now we claim that . The is not very difficult to prove. In particular, according to Equation (24) we know that
[TABLE]
Since the function is injective, it implies that . As a result, and . The desired inequality follows according to Equation (10.4). \Halmos
Now, we can prove the following corollary to Lemma 3.9 for generalized linear models.
Corollary 10.5
Consider rewards given by a generalized linear model with link function . Suppose that the noise terms are -subgaussian for some . Let be the estimated parameter of arm . Taking and , we have for all ,
[TABLE]
Proof 10.6
Proof of Corollary 10.5. Note that if the design matrix is positive definite, then the event is the subset of the event
[TABLE]
To show this, suppose the contrary is true, i.e., we have while . Then, applying Lemma 10.3 with implies that
[TABLE]
which is a contradiction. Therefore,
[TABLE]
where the last inequality follows from Lemma 3.9. \Halmos
Now, we prove a lemma following the same lines of idea as Lemma 3.11 but for generalized linear models.
Lemma 10.7
Recall that . Suppose that Assumptions 2.1 and 2.2 hold. Then, the instantaneous expected regret of the Greedy Bandit for GLMs (Algorithm 2) at time satisfies
[TABLE]
where , is defined in Assumption 2.2, is the Lipschitz constant of the function on the interval , and is defined in Proposition 3.12.
Proof 10.8
Proof of Lemma 10.7. The proof is very similar to the proof of Lemma 3.11. We can decompose the regret as . Now we can expand each term as
[TABLE]
as is Lipschitz over the interval and for all . Now one can follow all the arguments in Lemma 3.11 up to the point that we use concentration results for . In particular, Equation (9.1) reads as
[TABLE]
Using the concentration result on Corollary 10.5, and noting that is independent of for all , the right hand side of above equation turns into
[TABLE]
Now note that is at most equal to (since is upper bounded by ). As , therefore if , then . Hence, for all values of .
[TABLE]
We can simply use whenever this number is larger than one as this describes a probability term. Therefore,
[TABLE]
where we take . Note that functions and are both decreasing for and therefore
[TABLE]
Using the change of variable , integration by parts, and the inequality , we obtain that
[TABLE]
By replacing this in the regret equation above and substituting we get
[TABLE]
as desired. \Halmos
The only other result that we need is an upper bound on the probability terms . We can re-use Lemma 3.7 for this purpose, since the greedy decision does not change, i.e., . Thus, the minimum eigenvalue of each of the covariance matrices is bounded below by with high probability as before. We can now finally prove Proposition 3.12 by summing up the regret terms up to time .
Proof 10.9
Proof of Proposition 3.12. The regret term at time is upper bounded by . Noting that , we can apply Lemma 3.7 and Lemma 10.7 to write
[TABLE]
\Halmos
11 Additional Details on Greedy-First
We first present the pseudo-code for OLS Bandit and the heuristic for Greedy-First. The OLS Bandit algorithm was introduced by Goldenshluger and Zeevi (2013)* and generalized by Bastani and Bayati (2020)**. Here, we describe the more general version that applies to more than two arms where some arms may be uniformly sub-optimal. As mentioned earlier, in addition to Assumptions 2.1 and 2.2, the regret analysis of the OLS Bandit requires Assumption 2. The algorithm defines forced-sample sets, which prescribe a set of times when we forced-sample arm (regardless of the observed covariates ):*
[TABLE]
Thus, the set of forced samples from arm up to time is .
We also define all-sample sets \mathcal{S}_{i,t}=\left\{t^{\prime}\bigm{|}\pi_{t^{\prime}}=i\mbox{ and }1\leq t^{\prime}\leq t\right\}, where we have played arm up to time . By definition, . The algorithm proceeds as follows. During any forced sampling time , the corresponding arm (arm ) is played regardless of the observed covariates . At all other times, the algorithm uses two different arm parametere estimates to make decisions. First, it estimates arm parameters via OLS applied only to the forced sample set, and discards each arm that is sub-optimal by a margin of at least . Then, it applies OLS to the all-sample set, and picks the arm with the highest estimated reward among the remaining arms. Algorithm 4 provides the pseudo-code for OLS Bandit.
The pseudo-code for the Heuristic Greedy-First bandit is as follows.
12 Missing Proofs of §3.5 and §4.3
Proof 12.1
Proof of Proposition 3.15. We first start by proving monotonicity results:
- •
Let . Note that only the second, the third, and the last term of , defined in Equation (3.14), depend on . As for any positive number , the function is increasing with respect to , second and third terms are increasing with respect to . Furthermore, the last term can be expressed as
[TABLE]
Each term in above sum is increasing with respect to . Therefore, the function is increasing with respect to . As is one minus the infimum of taken over the possible parameter space of and , it is non-increasing with respect to , yielding the desired result.
- •
Let and suppose that we use the superscript for the function when . We claim that for all , and conditioning on we have . Note that the region for which does not matter as it leads to a negative probability of success in the formula , and we can only restrict our attention to the region for which . To prove the claim, let and define for the constant . Note that captures the first two terms of in Equation (3.14). As we later going to replace we only restrict our attention to . The derivative of is equal to which is negative when . Note that if and if we drop the third, fourth, and fifth term in (see Equation (3.14)) that are all positive, we obtain , leaving us in the undesired regime. Therefore, on the desired regime of study, the derivative is negative and is decreasing. It is not very difficult to see that . Returning to our original claim, if we calculate it is easy to observe that the third term cancels out and we end up with
[TABLE]
where we used the inequality . This proves our claim. Note that whenever when varies in the range , the quantity covers the range . Therefore, we can write that
[TABLE]
as desired.
- •
Let . In this case it is very easy to check that the first, fourth and fifth terms in (see Equation (3.14)) do not depend on . Dependency of second and third terms are in the form for some constant , which is decreasing with respect . Therefore, if we use the superscript for the function when , we have that which implies
[TABLE]
as desired.
- •
Similar to the previous part, it is easy to observe that the first, second, and third term in , defined in Equation (3.14) do not depend on . The dependency of last two terms with respect to is of the form and which both are decreasing functions of . The rest of argument is similar to the previous part and by replicating it with reach to the conclusion that is non-increasing with respect to .
- •
Suppose that , and . Similar to before, we use superscript to denote the function when . Then it is easy to check that the last three terms in and are the same. Therefore, for comparing and , one only needs to compare the first two terms. Letting and we have
[TABLE]
Similar to the proof of second part, it is not very hard to prove that on the reasonable regime for the parameters the function is decreasing and therefore
[TABLE]
as and . Taking the infimum implies the desired result.
Now we derive the limit of when . For each , define and . Then, by computing the function for these specific choices of parameters and upper bounding the summation in Equation (3.14) with its maximum times the number of terms we get
[TABLE]
for positive constants and that do not depend on . Note that for any ,
[TABLE]
Therefore, by taking limit with respect to we get
[TABLE]
proving one side of the result. For achieving the desired result we need to prove that which is straightforward. To see this, note that the function always satisfies
[TABLE]
As a result, for any we have
[TABLE]
By taking limits we reach to the desired conclusion. \Halmos
Proof 12.2
Proof of Proposition 4.5. We omit proofs regarding to the monotonicity results as they are very similar to those provided in Proposition 3.15.
For deriving the limit when , define and . Then, by computing the function for these specific values we have
[TABLE]
for positive constants and that do not depend on . Note that for ,
[TABLE]
Therefore, by taking limit with respect to we get
[TABLE]
proving one side of the result. For achieving the desired result we need to prove that the other side of this inequality. Note that the function always satisfies
[TABLE]
Note that the function is increasing with respect to . This is easy to verify as the first derivative of with respect to is equal to
[TABLE]
which is increasing for . Therefore, by using and we have
[TABLE]
Substituting this in Equation (27) implies that
[TABLE]
By taking limits we reach to the desired conclusion. \Halmos
12.1 Proofs of Theorems 3.14 and 4.4
Let us first start by introducing two new notations and recalling some others. For each define
[TABLE]
and recall that
[TABLE]
Note that whenever , the sets and coincide. We first start by proving some lemmas that will be used later to prove Theorems 3.14 and 4.4. The first lemma provides an upper bound on the probability that the estimate of one of the arms at time has an error of at least while the minimum eigenvalue of covariance matrices at is at least .
Lemma 12.3
Let be arbitrary. Then
[TABLE]
Remark 12.4
Note that Lemma 3.9 provides an upper bound on the same probability event described above. However, those results are addressing the case that samples are highly correlated due to greedy decisions. In the first rounds that rounds of random sampling are executed for each arm, samples are independent and we can use sharper tail bounds. This would help us to get better probability guarantees for the Greedy Bandit algorithm.
Proof 12.5
Proof of Lemma 12.3. Note that we can write
[TABLE]
Note that if , this means that the covariance matrix is invertible. Therefore, we can write
[TABLE]
To avoid clutter, we drop the term in equations. By letting the probability in Equation (28) turns into
[TABLE]
where in the second inequality we used a union bound. Note that in above means the probability distribution over the matrix , which can also be thought as the multi-dimensional probability distribution of , or alternatively . Now fixing , the matrix only depends on and we can use the well-known Chernoff bound for subgaussian random variables to achieve
[TABLE]
Now note that when we have
[TABLE]
Hence,
[TABLE]
Putting this back in Equation (12.5) gives
[TABLE]
as desired. In above we use the fact that is equal to as they both describe the probability that the minimum eigenvalue of a matrix derived from random samples from is not smaller than . \Halmos
Lemma 12.6
For an arbitrary and we have
[TABLE]
Proof 12.7
Proof of Lemma 12.6. This is an immediate consequence of Lemma 3.9. Replace and note that always holds as rounds of random sampling for arms other than exist in algorithm. \Halmos
The next step is proving that if all arm estimates are within the ball of radius around their true values, the minimum eigenvalue of arms in grow linearly, while sub-optimal arms are not picked by Greedy Bandit algorithm. The proof is a simple generalization of Lemma 3.7.
Lemma 12.8
For each
[TABLE]
Furthermore, for each and conditioning on the event , arm would not be played at time under greedy policy.
Proof 12.9
Proof of Lemma 12.8. We again use the concentration inequality in Lemma 8.3. Let and recall that
[TABLE]
denote the expected and sample covariance matrices of arm at time respectively. The aim is deriving an upper bound on the probability that minimum eigenvalue of is less than the threshold . Note that consists of two different types of terms: 1) random sampling rounds and 2) greedy action rounds . We analyze these two types separately as following:
- •
. Note that during the first periods, each arm receives random samples from the distribution and therefore using concavity of the function we have
[TABLE]
where is a random sample from distribution .
- •
. If holds for all , then
[TABLE]
The reason is very simple; basically having means that and therefore for each satisfying , using two Cauchy-Schwarz inequalities we can write
[TABLE]
for each . Therefore, by taking a maximum over we obtain . Hence,
[TABLE]
using Assumption 2, which holds for all optimal arms, i.e, .
Putting these two results together and using concavity of over positive semi-definite matrices we have
[TABLE]
Now the rest of the argument is similar to Lemma 3.7. Note that in the proof of Lemma 3.7, we simply put . Here if we use an arbitrary together with derived via Cauchy-Schwarz inequality, then Lemma 8.3 implies that
[TABLE]
The second event inside the probability event can be removed, as it always holds under . The first event also can be translated to and therefore for all we have
[TABLE]
as desired.
For a sub-optimal arm using Assumption 2, for each there exist such that and as a result conditioning on by using a Cauchy-Schwarz inequality we have
[TABLE]
This implies that and therefore arm is not played for at time (Note that once rounds of random sampling are finished the algorithm executes greedy algorithm). As this result holds for all choices of , arm becomes sub-optimal at time , as desired. \Halmos
Here, we state the final lemma, which bounds the probability that the event occurs whenever holds for any . In other words, this lemma shows that if the minimum eigenvalue of covariance matrix of arm at time is large, then the estimate of arm at time will be close to the true , with a high probability.
Lemma 12.10
For each
[TABLE]
Proof 12.11
Proof of Lemma 12.10. This is again obvious using Lemma 3.9. \Halmos
Now we are ready to prove Theorems 3.14 and 4.4. As the proofs of these two theorems are very similar we state and prove a lemma that implies both theorems.
Lemma 12.12
Let Assumption and 2 hold. Suppose that Greedy Bandit algorithm with -rounds of forced sampling in the beginning is executed. Let . Suppose that is an event which can be decomposed as , then event
[TABLE]
holds with probability at least
[TABLE]
In above, denotes the minimum eigenvalue of a matrix obtained from random samples from the distribution and constants are defined in Equations (14) and (15).
Proof 12.13
Proof of Lemma 12.12. One important property to note is the following result on the events:
[TABLE]
The reason is that the estimates for arms other than arm do not change at time , meaning that for each . Therefore, the above equality is obvious. This observation comes handy when we want to avoid using a union bound over different arms for the probability of undesired event. For deriving a lower bound on the probability of desired event we have
[TABLE]
Therefore, we can write
[TABLE]
The first term is equal to . The reason is simple; probability of each is given by and these events are all independent due to the random sampling. Therefore, the probability that at least one of them does not happen is given by the mentioned expression. In addition, the probability of the second event can be upper bounded by
[TABLE]
where we used Lemma 12.3 together with a union bound. For finding an upper bound on the the second probability, we treat terms and differently. Basically, for the first interval we have guarantees when holds (Lemma 12.6) and for the second interval the guarantee comes from having the event (Lemma 12.8). Hence, we can write
[TABLE]
using Equation (30) and carefully dividing the event into some smaller events. Note that by using the second part of Lemma 12.8, if the event holds, then is equal to one of the elements in and sub-optimal arms in will not be pulled. Therefore, the first term is upper bounded by
[TABLE]
using uniform upper bound provided in Lemma 12.6 and . This concludes the proof. \Halmos
Proof 12.14
Proof of Theorem 3.14 The proof consists of using Lemma 12.12. Basically, if we know that the events for and all hold, we have derived a lower bound on the probability that greedy succeeds. The reason is pretty simple here, if the distance of true parameters and is at most for each , we can easily ensure that the minimum eigenvalue of covariance matrices of optimal arms are growing linearly, and sub-optimal arms remain sub-optimal for all using Lemma 12.8. Therefore, we can prove the optimality of Greedy Bandit algorithm and also establish its logarithmic regret. Therefore, in this case we need not use any in Lemma 12.12, we simply put , where is the whole probability space. Then we have
[TABLE]
The upper bound on the last term can be derived as following
[TABLE]
which by using Lemmas 12.8 and 12.10 can be upper bounded by
[TABLE]
Summing up all these term yields the desired upper bound. Now note that this upper bound is algorithm-independent and holds for all values of , and and therefore we can take the supremum over these values for our desired event (or infimum over undesired event). This concludes the proof. \Halmos
For proving Theorem 4.4 the steps are very similar, the only difference is that the desired event happens if all events , hold, and in addition to that, events all need to hold for some . Recall that in Theorem 4.4, and therefore we can use the notations and interchangeably. For Greedy-First, we define for some . This basically, means we need to take for some .
Proof 12.15
Proof of Theorem 4.4 The proof is very similar to proof of Theorem 3.14. For arbitrary we want to derive a bound on the probability of the event
[TABLE]
Note that if and , then having events implies that the events all hold. In other words, Greedy-First does not switch to the exploratory algorithm and is able to achieve logarithmic regret. Let us substitute which implies that . Lemma 12.12 can be used to establish a lower bound on the probability of this event as
[TABLE]
Hence, we only need to derive an upper bound on the last term. By expanding this based on the value of we have
[TABLE]
using a union bound and the fact that the space has already been included in the first term, so its complement can be included in the second term. Now, using Lemmas 12.8 and 12.10 this can be upper bounded by
[TABLE]
As mentioned earlier, we can take supremum on parameters as long as they satisfy , and . They would lead to the same result only with the difference that the infimum over should be replaced by and these two functions satisfy
[TABLE]
which yields the desired result. \Halmos
Proof 12.16
Proof of Corollary 3.16. We want to use the result of Theorem 3.14. In this theorem, let us substitute and . After this substitution, Theorem 3.14 implies that the Greedy Bandit algorithm succeeds with probability at least
[TABLE]
For deriving a lower bound on the first term let us use the concentration inequality in Lemma 8.3. Note that here the samples are drawn i.i.d. from the same distribution . Therefore, by applying this Lemma we have
[TABLE]
Note that the second event, i.e. happens with probability one. This is true according to
[TABLE]
where and the inequality is true according to the Jensen’s inequality for the concave function . Now note that, this expectation can be bounded by
[TABLE]
according to Assumption 2 and another use of Jensen’s inequality for the function . Note that this part of proof was very similar to Lemma 12.8. Thus, with a slight modification we get
[TABLE]
After using this inequality together with the inequality , and after replacing values of and , the lower bound on the probability of success of Greedy Bandit reduces to
[TABLE]
In above we used the expansion . In order to finish the proof note that by a Cauchy-Schwarz inequality . Furthermore, and therefore the above bound is greater than or equal to
[TABLE]
as desired. \Halmos
Proof 12.17
Proof of Corollary 4.6. Proof of this corollary is very similar to the previous corollary. Extra conditions of the corollary ensure that both lie on their accepted region. For avoiding clutter, we skip the proof. \Halmos
13 Additional Simulations
We now explore the performance of Greedy Bandit as a function of and , as well as the dependence of the performance of Greedy-First on the input parameters (which determines when to switch) and (which are inputs to OLS Bandit after switching). Note that Greedy Bandit is entirely parameter-free.
13.1 More than Two Arms ()
We simulate the Greedy Bandit with arms, and vary the dimension . Here, we fix the context distribution to truncated at , and we draw arm parameters from . To ensure a fair comparison, we scale the noise variance by so as to keep the signal-to-noise ratio fixed (i.e., ). The results are shown in Figure 5. We find that the performance of Greedy Bandit improves dramatically as the dimension increases, while it degrades with the number of arms (as predicted by Proposition 3.15). When is small relative to , it is likely that Greedy Bandit will drop an arm due to an early poor arm parameter estimate, which then results in linear regret. However, when is large relative to , Greedy Bandit performs very well. We conjecture that this turning point occurs when .
We also repeat the simulations detailed in §5.1 with the only modification that , ; we employ the true prior for OFUL and TS. In Figure 6, we plot the resulting cumulative regret for all algorithms averaged over 1000 runs. We observe that Greedy-First nearly ties with Greedy Bandit as the winner when . However for , Greedy Bandit performs poorly, while Greedy-First performs nearly as well as the best algorithms. Thus, we again see empirical evidence that higher dimension benefits a greedy approach.
13.2 Sensitivity to parameters
We now perform a sensitivity analysis to see how the input parameters , , and affect the performance of Greedy-First. Note that Greedy Bandit is entirely parameter-free. The sensitivity analysis is performed with the same problem parameters as in Figure 2 for the case that covariate diversity does not hold. As can be observed from Figure 7, we find that the performance of Greedy-First is quite robust to the choice of inputs.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Abbasi-Yadkori et al. (2011) Abbasi-Yadkori, Yasin, Dávid Pál, Csaba Szepesvári. 2011. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems. 2312–2320.
- 2Agrawal et al. (2019) Agrawal, Shipra, Vashist Avadhanula, Vineet Goyal, Assaf Zeevi. 2019. Mnl-bandit: A dynamic learning approach to assortment selection. Operations Research 67 (5) 1453–1485.
- 3Agrawal and Goyal (2013) Agrawal, Shipra, Navin Goyal. 2013. Thompson sampling for contextual bandits with linear payoffs. International Conference on Machine Learning. 127–135.
- 4Auer (2002) Auer, Peter. 2002. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research 3 (Nov) 397–422.
- 5Ban and Keskin (2020) Ban, Gah-Yi, N Bora Keskin. 2020. Personalized dynamic pricing with machine learning: High dimensional features and heterogeneous elasticity. Available at SSRN URL https://papers.ssrn.com/sol 3/papers.cfm?abstract_id=2972985 .
- 6Bastani and Bayati (2020) Bastani, Hamsa, Mohsen Bayati. 2020. Online decision making with high-dimensional covariates. Operations Research 68 (1) 276–294.
- 7Bastani et al. (2018) Bastani, Hamsa, Pavithra Harsha, Georgia Perakis, Divya Singhvi. 2018. Learning personalized product recommendations with customer disengagement. Available at SSRN URL https://ssrn.com/abstract=3240970 .
- 8Bastani et al. (2019) Bastani, Hamsa, David Simchi-Levi, Ruihao Zhu. 2019. Meta dynamic pricing: Learning across experiments. Available at SSRN URL https://ssrn.com/abstract=3334629 .
