Equipping Experts/Bandits with Long-term Memory

TL;DR

This paper introduces a reduction-based method for achieving long-term memory guarantees in online learning, providing new algorithms with improved regret bounds and extending results to sparse bandit settings.

Contribution

It presents the first reduction-based approach for long-term memory guarantees in online learning, achieving optimal regret bounds and extending to sparse multi-armed bandits.

Findings

Developed algorithms with regret of order √T(S ln T + n ln K).

Achieved simultaneous adaptation to stochastic and adversarial environments.

Provided lower bounds showing sparse losses do not improve worst-case regret.

Abstract

We propose the first reduction-based approach to obtaining long-term memory guarantees for online learning in the sense of Bousquet and Warmuth, 2002, by reducing the problem to achieving typical switching regret. Specifically, for the classical expert problem with $K$ actions and $T$ rounds, using our framework we develop various algorithms with a regret bound of order $\mathcal{O}(\sqrt{T(S\ln T + n \ln K)})$ compared to any sequence of experts with $S-1$ switches among $n \leq \min\{S, K\}$ distinct experts. In addition, by plugging specific adaptive algorithms into our framework we also achieve the best of both stochastic and adversarial environments simultaneously. This resolves an open problem of Warmuth and Koolen, 2014. Furthermore, we extend our results to the sparse multi-armed bandit setting and show both negative and positive results for long-term memory guarantees. As a side result, our lower bound also implies that sparse losses do not help improve the worst-case regret for contextual bandits, a sharp contrast with the non-contextual case.

Equations

$$\mathcal{O}(\sqrt{TS\ln (TK)})$$

$$\mathcal{O}(\sqrt{TKS\ln (TK)})$$

$$\mathcal{R}(i_{1:T})=\mathbb{E}\left[\sum _{t=1}^T{\ell }_t(I_t)-\sum _{t=1}^T{\ell }_t(i_t)\right]=\mathbb{E}\left[\sum _{t=1}^T{r}_t(i_t)\right],$$

$$\sum _{t=1}^T{w}_t^{\top }{c}_t-\sum _{t=1}^T{c}_t(i)\le \frac{C\ln K}{\eta }+\eta \sum _{t=1}^T\left|c_t(i)\right|,\forall i\in [K]$$

$$\sum _{t=1}^T{w}_t^{\top }{c}_t-\sum _{t=1}^T{c}_t(i)\le \frac{C\ln K}{\eta }+\eta \sum _{t=1}^T\left|w_t^{\top }{c}_t-c_t(i)\right|,\forall i\in [K].$$

$$\sum _{t=1}^T{q}_t^{\top }{h}_t-\sum _{t=1}^T{h}_t(b_t)\le \frac{CS\ln T}{\eta }+\eta \sum _{t=1}^T\left|h_t(b_t)\right|,$$

$$\sum _{t=1}^T{q}_t^{\top }{h}_t-\sum _{t=1}^T{h}_t(b_t)\le \frac{CS\ln T}{\eta }+\eta \sum _{t=1}^T\left|q_t^{\top }{h}_t-h_t(b_t)\right|,$$

$$\sum _{t=1}^T{q}_t^{\top }{h}_t-\sum _{t=1}^T{h}_t(b_t)\le \frac{CS\ln T}{\eta }+\eta \sum _{t=1}^T\sum _{b\in \left\{0,1\right\}}{q}_t(b)\left|h_t(b)\right|.$$

$$\sum _{t=1}^T{w}_t^{\top }{c}_t-\sum _{t=1}^T{c}_t(i,j)\le \frac{C\ln (KM)}{{\eta }_j}+{\eta }_j\sum _{t=1}^T\left|w_t^{\top }{c}_t-c_t(i,j)\right|.$$

$$\mathcal{R}(i_{1:T})=\mathcal{O}\left({\left(\rho S\right)}^{\frac{1}{3}}(nT{)}^{\frac{2}{3}}+n\sqrt{T\rho \ln K}+n{K}^3\ln T\right)$$

$$\sum _{t=1}^T{w}_t^{\top }{c}_t-c_t(i)\le \mathcal{O}\left(\sqrt{(\ln K)\sum _{t=1}^T\left|w_t^{\top }{c}_t-c_t(i)\right|}+\ln K\right).$$

$${\Phi }_t-{\Phi }_{t-1}$$

$$=\sum _i\exp \left(\eta R_{t-1}(i)-{\eta }^2{G}_{t-1}(i)\right)\left(\exp \left(\eta r_t(i)-{\eta }^2{c}_t^2(i)\right)-1\right)$$

$$=\exp \left(\eta w_t^{\top }{c}_t\right)\sum _i\exp \left(\eta R_{t-1}(i)-{\eta }^2{G}_{t-1}(i)\right)\left(\exp \left(-\eta c_t(i)-{\eta }^2{c}_t^2(i)\right)-\exp \left(-\eta w_t^{\top }{c}_t\right)\right)$$

$$\le \exp \left(\eta w_t^{\top }{c}_t\right)\sum _i\exp \left(\eta R_{t-1}(i)-{\eta }^2{G}_{t-1}(i)\right)\left(1-\eta c_t(i)-\exp \left(-\eta w_t^{\top }{c}_t\right)\right)$$

$$\le \exp \left(\eta w_t^{\top }{c}_t\right)\sum _i\exp \left(\eta R_{t-1}(i)-{\eta }^2{G}_{t-1}(i)\right)\eta r_t(i)$$

$$=0,$$

$$⟨q_t-q,\eta h_t+{\eta }^2{h}_t^2⟩$$

$$=⟨q_t-q,\nabla \psi (q_t)-\nabla \psi ({\bar{q}}_{t+1})⟩$$

$$=D_{\psi }(q,q_t)-D_{\psi }(q,{\bar{q}}_{t+1})+D_{\psi }(q_t,{\bar{q}}_{t+1})$$

$$\le D_{\psi }(q,q_t)-D_{\psi }(q,{\tilde{q}}_{t+1})+D_{\psi }(q_t,{\bar{q}}_{t+1})$$

$$=D_{\psi }(q,q_t)-D_{\psi }(q,{\tilde{q}}_{t+1})+\sum _{b=1}^K{q}_t(b)\left(\eta h_t(b)+{\eta }^2{h}_t^2(b)+\exp (-\eta h_t(b)-{\eta }^2{h}_t^2(b))-1\right)$$

$$\le D_{\psi }(q,q_t)-D_{\psi }(q,{\tilde{q}}_{t+1})+{\eta }^2\sum _{b=1}^K{q}_t(b)h_t^2(b)$$

$$\le D_{\psi }(q,q_t)-D_{\psi }(q,q_{t+1})+2\gamma +{\eta }^2\sum _{b=1}^K{q}_t(b)h_t^2(b),$$

$$⟨q_t-q,h_t⟩\le \frac{D_{\psi }(q,q_t)-D_{\psi }(q,q_{t+1})+2\gamma }{\eta }+\eta \sum _{b=1}^{K}q(b)h_t^2(b).$$

$$\sum _{t=s}^e⟨q_t-q,h_t⟩$$

$$\le \frac{\ln \frac{K}{\gamma }+2(t-s+1)\gamma }{\eta }+\eta \sum _{t=s}^e\sum _{b=1}^{K}q(b)h_t^2(b).$$

$$\sum _{t=1}^T{w}_t^{\top }{c}_t-c_t(i)\le \frac{C\ln K}{{\eta }_i}+{\eta }_i\sum _{t=1}^T\left|w_t^{\top }{c}_t-c_t(i)\right|.$$

$$\ln \left(\sum _{i=1}^K{\Phi }_{t,i}\right)-\ln \left(\sum _{i=1}^K{\Phi }_{t-1,i}\right)$$

$$⩽\ln \frac{{\sum }_i{\Phi }_{t-1,i}(1+{\eta }_i{r}_t(i))}{{\sum }_i{\Phi }_{t-1,i}}$$

$$=\ln \frac{{\sum }_i{\Phi }_{t-1,i}}{{\sum }_i{\Phi }_{t-1,i}}=0,$$

$$\ln K=\ln \sum _i{\Phi }_{0,i}\ge \cdots \ge \ln \sum _i{\Phi }_{T,i}\ge \ln {\Phi }_{T,i}=\sum _{t=1}^T({\eta }_i{r}_t(i)-{\eta }_i^2{r}_t^2(i)).$$

$$\sum _{t=1}^T{z}_t(i)r_t(i)\le \frac{C\ln K}{\eta }+2\eta \sum _{t=1}^T{z}_t(i).$$

$$\sum _{t=1}^T{z}_t(i)(5\eta -r_t(i))\le \sum _{t:i_t=i}(5\eta -r_t(i))+\frac{C{S}_i\ln T}{\eta }+\eta B,$$

$$\begin{cases}\sum_{t:i_{t}=i}|5\eta-r_{t}(i)|&\quad\text{if Eq.\leavevmode\nobreak\ \eqref{eqn:switching1_1} holds,}\\ \sum_{t:i_{t}\neq i}z_{t}(i)|5\eta-r_{t}(i)|+\sum_{t:i_{t}=i}(1-z_{t}(i))|5\eta-r_{t}(i)|&\quad\text{if Eq.\leavevmode\nobreak\ \eqref{eqn:switching1_2} holds,}\\ \sum_{t=1}^{T}z_{t}(i)|5\eta-r_{t}(i)|&\quad\text{if Eq.\leavevmode\nobreak\ \eqref{eqn:switching1_3} holds.}\end{cases}$$

$$B\le 3\sum _{t=1}^T{z}_t(i)+3T_i.$$

$$\sum _{t:i_t=i}{r}_t(i)\le \frac{C{S}_i\ln T}{\eta }+8\eta T_i+\sum _{t=1}^T\left(z_t(i)r_t(i)-2\eta z_t(i)\right).$$

$$\sum _{t:i_t=i}{r}_t(i)\le \frac{C(S_i\ln T+\ln K)}{\eta }+8\eta T_i.$$

$$\sum _{t=1}^T{w}_t^{\top }{c}_t-c_t(i)=\mathcal{O}\left(\sqrt{T\ln K}\right)\text{and}\sum _{t=1}^T{q}_t^{\top }{h}_t-h_t(b_t)=\mathcal{O}\left(\sqrt{TS\ln T}\right)$$

Full text

1 Introduction

In this work, we propose a black-box reduction for obtaining long-term memory guarantees for two fundamental problems in online learning: the expert problem [17] and the multi-armed bandit (MAB) problem [6]. In both problems, a learner interacts with the environment for $T$ rounds, with $K$ fixed available actions. At each round, the environment decides the loss for each action while simultaneously the learner selects one of the actions and suffers the loss of this action. In the expert problem, the learner observes the loss of every action at the end of each round (a.k.a. full-information feedback), while in MAB, the learner only observes the loss of the selected action (a.k.a. bandit feedback).

For both problems, the classical performance measure is the learner’s (static) regret, defined as the difference between the learner’s total loss and the loss of the best fixed action. It is well-known that the minimax optimal regret is $\Theta(\sqrt{T\ln K})$ [17] and $\Theta(\sqrt{TK})$ [6, 4] for the expert problem and MAB respectively. Comparing against a fixed action, however, does not always lead to meaningful guarantees, especially when the environment is non-stationary and no single fixed action performs well. To address this issue, prior work has considered a stronger measure called switching/tracking/shifting regret, which is the difference between the learner’s total loss and the loss of a sequence of actions with at most $S-1$ switches. Various existing algorithms (including some black-box approaches) achieve the following switching regret

[TABLE]

We call these typical switching regret bounds. Such bounds essentially imply that the learner pays the worst-case static regret for each switch in the benchmark sequence. While this makes sense in the worst case, intuitively one would hope to perform better if the benchmark sequence frequently switches back to previous actions, as long as the algorithm remembers which actions have performed well previously.

Indeed, for the expert problem, algorithms with long-term memory were developed that guarantee switching regret of order $\mathcal{O}\left(\sqrt{T(S\ln\frac{nT}{S}+n\ln\frac{K}{n})}\right)$, where $n\leq\min\{S,K\}$ is the number of distinct actions in the benchmark sequence [8, 2, 13].111The setting considered in [8, 2] is in fact slightly different from, yet closely related to, the expert problem. One can easily translate their regret bounds into the bounds we present here. Although there is no known lower bound, this regret bound essentially matches the one achieved by a computationally inefficient approach of running Hedge over all benchmark sequences with $S$ switches among $n$ experts, an approach that usually leads to the information-theoretically optimal regret guarantee. Compared to the typical switching regret bound of form (1) (which can be written as $\mathcal{O}(\sqrt{T(S\ln T+S\ln K)})$), this long-term memory guarantee implies that the learner pays the worst-case static regret only for each distinct action encountered in the benchmark sequence, and pays less for each switch, especially when $n$ is very small. Algorithms with long-term memory guarantees have been found to have better empirical performance [8], and applied to practical applications such as TCP round-trip time estimation [30], intrusion detection system [29], and multi-agent systems [31]. We are not aware of any similar studies for the bandit setting.

Overview of our contributions.

The main contribution of this work is to propose a simple black-box approach to equip expert or MAB algorithms with long-term memory and to achieve switching regret guarantees of similar flavor to those of [8, 2, 13]. The key idea of our approach is to utilize a variant of the confidence-rated expert framework of [7], and to use a sub-routine to learn the confidence/importance of each action for each time. Importantly this sub-routine itself is an expert/bandit algorithm over only two actions and needs to enjoy some typical switching regret guarantee (for example of form (1) for the expert problem). In other words, our approach reduces the problem of obtaining long-term memory to the well-studied problem of achieving typical switching regret. Compared to existing methods [8, 2, 13], the advantages of our approach are the following:

1. While existing methods are all restricted to variants of the classical Hedge algorithm [17], our approach allows one to plug in a variety of existing algorithms and to obtain a range of different algorithms with switching regret $\mathcal{O}(\sqrt{T(S\ln T+n\ln K)})$. (Section 3.1)

2. Due to this flexibility, by plugging in specific adaptive algorithms, we develop a parameter-free algorithm whose switching regret is simultaneously $\mathcal{O}(\sqrt{T(S\ln T+n\ln K)})$ in the worst-case and $\mathcal{O}(S\ln T+n\ln(K\ln T))$ if the losses are piece-wise stochastic (see Section 2 for the formal definition). This is a generalization of previous best-of-both-worlds results for static or switching regret [19, 27], and resolves an open problem of Warmuth and Koolen [35]. The best previous bound for the stochastic case is $\mathcal{O}(S\ln(TK\ln T))$ [27]. (Section 3.2)

3. Our framework allows us to derive the first nontrivial long-term memory guarantees for the bandit setting, while existing approaches fail to do so (more discussion to follow). For example, when $n$ is a constant and the losses are sparse, our algorithm achieves switching regret $\mathcal{O}(S^{1/3}T^{2/3}+K^{3}\ln T)$ for MAB, which is better than the typical bound (2) when $S$ and $K$ are large. For example, when $S=\Theta(T^{\frac{7}{10}})$ and $K=\Theta(T^{\frac{3}{10}})$, our bound is of order $\mathcal{O}(T^{\frac{9}{10}}\ln T)$ while bound (2) becomes vacuous (linear in $T$), demonstrating a strict separation in learnability. (Section 4)

To motivate our results on long-term memory guarantees for MAB, a few remarks are in order. It is not hard to verify that existing approaches achieve switching regret $\mathcal{O}(\sqrt{TK(S\ln T+n\ln K)})$ for MAB. However, the polynomial dependence on the number of actions $K$ makes the improvement of this bound over the typical bound (2) negligible. It is well-known that such polynomial dependence on $K$ is unavoidable in the worst-case due to the bandit feedback. This motivates us to consider situations where the necessary dependence on $K$ is much smaller. In particular, Bubeck et al. [10] recently showed that if the loss vectors are $\rho$-sparse, then a static regret bound of order $\mathcal{O}(\sqrt{T\rho\ln K}+K\ln T)$ is achievable, exhibiting a much more favorable dependence on $K$. We therefore focus on this sparse MAB problem and study what nontrivial switching regret bounds are achievable.

We first show that a bound of order $\mathcal{O}(\sqrt{T\rho S\ln(KT)}+KS\ln T)$, a natural generalization of the typical switching regret bound of (2) to the sparse setting, is impossible. In fact, we show that for any $S$ the worst-case switching regret is at least $\Omega(\sqrt{TKS})$, even when $\rho=2$. Since achieving switching regret for MAB can be seen as a special case of contextual bandits [6, 26], this negative result also implies that, surprisingly, sparse losses do not help improve the worst-case regret for contextual bandits, which is a sharp contrast with the non-contextual case studied in [10] (see Theorem 6 and Corollary 7). Despite this negative result, however, as mentioned we are able to utilize our general framework to still obtain improvements over bound (2) when $n$ is small. Our construction is fairly sophisticated, requiring a special sub-routine that uses a novel one-sided log-barrier regularizer and admits a new kind of “local-norm” guarantee, which may be of independent interest.

2 Preliminaries

Throughout the paper, we use $[m]$ to denote the set $\{1,\ldots,m\}$ for some integer $m$. The learning protocol for the expert problem and MAB with $K$ actions and $T$ rounds is as follows: For each time $t=1,\ldots,T$, (1) the learner first randomly selects an action $I_{t}\in[K]$ according to a distribution $p_{t}\in\Delta_{K}$ (the $(K-1)$-dimensional simplex); (2) simultaneously the environment decides the loss vector $\ell_{t}\in[-1,1]^{K}$; (3) the learner suffers loss $\ell_{t}(I_{t})$ and observes either $\ell_{t}$ in the expert problem (full-information feedback) or only $\ell_{t}(I_{t})$ in MAB (bandit feedback). For any sequence of $T$ actions $i_{1},\ldots,i_{T}\in[K]$, the expected regret of the learner against this sequence is defined as

[TABLE]

where the expectation is with respect to both the learner and the environment and $r_{t}(i)$, the instantaneous regret (against action $i$), is defined as $p_{t}^{\top}\ell_{t}-\ell_{t}(i)$. When $i_{1}=\cdots=i_{T}$, this becomes the traditional static regret against a fixed action. Most existing works on switching regret impose a constraint on the number of switches for the benchmark sequence: $\sum_{t=2}^{T}\boldsymbol{1}\left\{i_{t}\neq i_{t-1}\right\}\leq S-1$. In other words, the sequence can be decomposed into $S$ disjoint intervals, each with a fixed comparator as in static regret. Typical switching regret bounds hold for any sequence with this constraint and are in terms of $T,K$ and $S$, such as Eq. (1) and Eq. (2).

The number of switches, however, does not fully characterize the difficulty of the problem. Intuitively, a sequence that frequently switches back to previous actions should be an easier benchmark for an algorithm with long-term memory that remembers which actions performed well in the past. To encode this intuition, prior works [8, 2, 13] introduced another parameter $n=|\left\{i_{1},\ldots,i_{T}\right\}|$, the number of distinct actions in the sequence, to quantify the difficulty of the problem, and developed switching regret bounds in terms of $T,K,S$ and $n$. Clearly one has $n\leq\min\left\{S,K\right\}$, and we are especially interested in the case when $n\ll\min\left\{S,K\right\}$, which is natural if the data exhibits some periodic pattern. Our goal is to understand what improvements are achievable in this case and how to design algorithms that can leverage this property via a unified framework.

Stochastic setting.

In general, we do not make any assumptions on how the losses are generated by the environment, which is known as the adversarial setting in the literature. We do, however, develop an algorithm (for the expert problem) that enjoys the best of both worlds — it not only enjoys some robust worst-case guarantee in the adversarial setting, but also achieves much smaller logarithmic regret in a stochastic setting. Specifically, in this stochastic setting, without loss of generality, we assume the $n$ distinct actions in $\left\{i_{1},\ldots,i_{T}\right\}$ are $1,\ldots,n$. It is further assumed that for each $i\in[n]$, there exists a constant gap $\alpha_{i}>0$ such that $\mathbb{E}_{t}\left[\ell_{t}(j)-\ell_{t}(i)\right]\geq\alpha_{i}$ for all $j\neq i$ and all $t$ such that $i_{t}=i$, where the expectation is with respect to the randomness of the environment conditioned on the history up to the beginning of round $t$. In other words, for every time step the algorithm is compared to the best action whose expected value is constant away from those of other actions. This is a natural generalization of the stochastic setting studied for static regret or typical switching regret [19, 27].

Confidence-rated actions.

Our approach makes use of the confidence-rated expert setting of Blum and Mansour [7], a generalization of the expert problem (and the sleeping expert problem [18]). The protocol of this setting is the same as the expert problem, except that at the beginning of each round, the learner first receives a confidence score $z_{t}(i)$ for each action $i$. The regret against a fixed action $i$ is also scaled by its confidence and is now defined as $\mathbb{E}\left[\sum_{t=1}^{T}z_{t}(i)r_{t}(i)\right]$. The expert problem is clearly a special case with $z_{t}(i)=1$ for all $t$ and $i$. There are a number of known examples showing why this formulation is useful, and our work will add one more to this list.

To obtain a bound on this new regret measure, one can in fact simply reduce it to the regular expert problem [7, 19, 27]. Specifically, let ${\mathcal{A}}$ be some expert algorithm over the same $K$ actions producing sampling distributions $w_{1},\ldots,w_{T}\in\Delta_{K}$. The reduction works by sampling $I_{t}$ according to $p_{t}$ such that $p_{t}(i)\propto z_{t}(i)w_{t}(i),\leavevmode\nobreak\ \forall i$ and then feeding $c_{t}$ to ${\mathcal{A}}$ where $c_{t}(i)=-z_{t}(i)r_{t}(i),\leavevmode\nobreak\ \forall i$. Note that by the definition of $p_{t}$ one has $w_{t}^{\top}c_{t}=\sum_{i}w_{t}(i)z(i)(\ell_{t}(i)-p_{t}^{\top}\ell_{t})=0$. Therefore, one can directly equalize the confidence-rated regret and the regular static regret of the reduced problem: $\mathbb{E}\left[\sum_{t=1}^{T}z_{t}(i)r_{t}(i)\right]=\mathbb{E}\left[\sum_{t=1}^{T}(w_{t}^{\top}c_{t}-c_{t}(i))\right]$.

3 General Framework for the Expert Problem

In this section, we introduce our general framework to obtain long-term memory regret bounds and demonstrate how it leads to various new algorithms for the expert problem. We start with a simpler version and then move on to a more elaborate construction that is essential to obtain best-of-both-worlds results.

3.1 A simple approach for adversarial losses

A simple version of our approach is described in Algorithm 1. At a high level, it simply makes use of the confidence-rated action framework described in Section 2. The reduction to the standard expert problem is executed in Lines 1 and 1, with a black-box expert algorithm ${\mathcal{A}}$.

It remains to specify how to come up with the confidence score $z_{t}(i)$. We propose to learn these scores via a separate black-box expert algorithm ${\mathcal{A}}_{i}$ for each $i$. More specifically, each ${\mathcal{A}}_{i}$ is learning over two actions 0 and 1, where action 0 corresponds to confidence score 0 and action 1 corresponds to score 1. Therefore, the probability of picking action 1 at time $t$ naturally represents a confidence score between 0 and 1, which we denote by $z_{t}(i)$ overloading the notation (Line 1).

As for the losses fed to ${\mathcal{A}}_{i}$, we fix the loss of action 0 to be 0 (since shifting losses by the same amount has no real effect), and set the loss of action 1 to be $5\eta-r_{t}(i)$ (Line 1). The role of the term $-r_{t}(i)$ is intuitively clear — the larger the loss of action $i$ compared to the algorithm, the less confident we should be about it; the role of the constant bias term $5\eta$ will become clear in the analysis (in fact, it can even be removed at the cost of a worse bound — see Appendix B.2).

Finally we specify what properties we require from the black-box algorithms ${\mathcal{A}},{\mathcal{A}}_{1},\ldots,{\mathcal{A}}_{K}$. In short, ${\mathcal{A}}$ needs to ensure a static regret bound, while ${\mathcal{A}}_{1},\ldots,{\mathcal{A}}_{K}$ need to ensure a switching regret bound. See Figure 1 for an illustration of our reduction. The trick is that since ${\mathcal{A}}_{1},\ldots,{\mathcal{A}}_{K}$ are learning over only two actions, this construction helps us to separate the dependence on $K$ and the number of switches $S$. These (static or switching) regret bounds could be the standard worst-case $T$-dependent bounds mentioned in Section 1, in which case we would obtain looser long-term memory guarantees (specifically, $\sqrt{n}$ times worse — see Appendix B.2). Instead, we require these bounds to be data-dependent and in particular of the form specified below:

Condition 1**.**

There exists a constant $C>0$ such that for any $\eta\in(0,1/5]$ and any loss sequence $c_{1},\ldots,c_{T}\in[-2,2]^{K}$, algorithm ${\mathcal{A}}$ (possibly with knowledge of $\eta$) produces sampling distributions $w_{1},\ldots,w_{T}\in\Delta_{K}$ and ensures one of the following static regret bounds:

[TABLE]

Condition 2**.**

There exists a constant $C>0$ such that for any $\eta\in(0,1/5]$, any loss sequence $h_{1},\ldots,h_{T}\in[-3,3]^{2}$, and any $S\in[T]$, algorithm ${\mathcal{A}}_{i}$ (possibly with knowledge of $\eta$) produces sampling distributions $q_{1},\ldots,q_{T}\in\Delta_{2}$ and ensures one of the following switching regret bounds against any sequence $b_{1},\ldots,b_{T}\in\{0,1\}$ with $\sum_{t=2}^{T}\boldsymbol{1}\left\{b_{t}\neq b_{t-1}\right\}\leq S-1$:222In terms of notation in Algorithm 1, $q_{t}=(1-z_{t}(i),z_{t}(i))$.

[TABLE]

We emphasize that these data-dependent bounds are all standard in the online learning literature,333In fact, most standard bounds replace the absolute value we present here with square, leading to even smaller bounds (up to a constant). We choose to use the looser ones with absolute values since this makes the conditions weaker while still being sufficient for all of our analysis. and provide a few examples below (see Appendix A for brief proofs).

Proposition 1**.**

The following algorithms all satisfy Condition 1: Variants of Hedge [20, 34], Prod [12], Adapt-ML-Prod [19], AdaNormalHedge [27], and iProd/Squint [25].

Proposition 2**.**

The following algorithms all satisfy Condition 2: Fixed-share [23], a variant of Fixed-share (Algorithm 5 in Appendix A), and AdaNormalHedge.TV [27].

We are now ready to state the main result for Algorithm 1 (see Appendix B.1 for the proof).

Theorem 3**.**

Suppose Conditions 1 and 2 both hold. With $\eta=\min\left\{\frac{1}{5},\sqrt{\frac{S\ln T+n\ln K}{T}}\right\}$, Algorithm 1 ensures ${\mathcal{R}}(i_{1:T})=\mathcal{O}\left(\sqrt{T(S\ln T+n\ln K)}\right)$ for any loss sequence $\ell_{1},\ldots,\ell_{T}$ and benchmark sequence $i_{1},\dots,i_{T}$ such that $\sum_{t=2}^{T}\boldsymbol{1}\{i_{t}\neq i_{t-1}\}\leq S-1$ and $|\{i_{1},\dots,i_{T}\}|\leq n$.

Our bound in Theorem 3 is slightly worse than the existing bound of $\mathcal{O}\left(\sqrt{T(S\ln\frac{nT}{S}+n\ln\frac{K}{n})}\right)$ [8, 2],444In fact, using the adaptive guarantees of AdaNormalHedge [27] or iProd/Squint [25] that replaces the $\ln K$ dependence in Eq. (4) by a KL divergence term, one can further improve the term $n\ln K$ in our bound to $n\ln\frac{K}{n}$ matching previous bounds. Since this improvement is small, we omit the details.

but still improves over the typical switching regret $\mathcal{O}(\sqrt{T(S\ln T+S\ln K)})$ (Eq. (1)), especially when $n$ is small and $S$ and $K$ are large. To better understand the implication of our bounds, consider the following thought experiment. If the learner knew about the switch points (that is, $\{t:i_{t}\neq i_{t-1}\}$) that naturally divide the whole game into $S$ intervals, she could simply pick any algorithm with optimal static regret ($\sqrt{\text{``\#rounds''}\ln K}$) and apply $S$ instances of this algorithm, one for each interval, which, via a direct application of the Cauchy-Schwarz inequality, leads to switching regret $\sqrt{TS\ln K}$. Compared to bound (1), this implies that the price of not knowing the switch points is $\sqrt{TS\ln T}$. Similarly, if the learner knew not only the switch points, but also the information on which intervals share the same competitor, then she could naturally apply $n$ instances of the static algorithm, one for each set of intervals with the same competitor. Again by the Cauchy-Schwarz inequality, this leads to switching regret $\sqrt{Tn\ln K}$. Therefore, our bound implies that the price of not having any prior information of the benchmark sequence is still $\sqrt{TS\ln T}$.

Compared to existing methods, our framework is more flexible and allows one to plug in any combination of the algorithms listed in Propositions 1 and 2. This flexibility is crucial and allows us to solve the problems discussed in the following sections. The approach of [2] makes use of a sleeping expert framework, a special case of the confidence-rated expert framework. However, their approach is not a general reduction and does not allow plugging in different algorithms. Finally, we note that our construction also shares some similarity with the black-box approach of [14] for a multi-task learning problem.

3.2 Best of both worlds

To further demonstrate the power of our approach, we now show how to use our framework to construct a parameter-free algorithm that enjoys the best of both adversarial and stochastic environments, resolving the open problem of [35] (see Algorithm 2). The key is to derive an adaptive switching regret bound that replaces the dependence on $T$ by the sum of the magnitudes of the instantaneous regret $\sum_{t}|r_{t}(i)|$, which previous works [19, 27] show is sufficient for adapting to the stochastic setting and achieving logarithmic regret.

To achieve this goal, the first modification we need is to change the bias term for the loss of action “1” for ${\mathcal{A}}_{i}$ from $5\eta$ to $5\eta|r_{t}(i)|$. Following the proof of Theorem 3, one can show that the dependence on $|\{t:i_{t}=i\}|$ now becomes $\sum_{t:i_{t}=i}\left|r_{t}(i)\right|$ for the regret against $i$. If we could tune $\eta$ optimally in terms of this data-dependent quality, then this would imply logarithmic regret in the stochastic setting by the same reasoning as in [19, 27].

However, the difficulty is that the optimal tuning of $\eta$ is unknown beforehand, and more importantly, different actions require tuning $\eta$ differently. To address this issue, at a high level we discretize the learning rate and pick $M=\Theta(\ln T)$ exponentially increasing values (Line 2), then we make $M=\Theta(\ln T)$ copies of each action $i\in[K]$, one for each learning rate $\eta_{j}$. More specifically, this means that the number of actions for ${\mathcal{A}}$ increases from $K$ to $KM$, and so does the number of sub-routines with switching regret, now denoted as ${\mathcal{A}}_{ij}$ for $i\in[K]$ and $j\in[M]$. Different copies of an action $i$ share the same loss $\ell_{t}(i)$ for ${\mathcal{A}}$, while action “1” for ${\mathcal{A}}_{ij}$ now suffers loss $5\eta_{j}|r_{t}(i)|-r_{t}(i)$ (Line 2). The rest of the construction remains the same. Note that selecting a copy of an action is the same as selecting the corresponding action, which explains the update rule of the sampling probability $p_{t}$ in Line 2 that marginalizes over $j$. Also note that for a vector in $\mathbb{R}^{KM}$ (e.g., $w_{t},c_{t},z_{t}$), we use $(i,j)$ to index its coordinates for $i\in[K]$ and $j\in[M]$.

Finally, with this new construction, we need algorithm ${\mathcal{A}}$ to exhibit a more adaptive static regret bound and in some sense be aware of the fact that different actions now correspond to different learning rates. More precisely, we replace Condition 1 with the following condition:

Condition 3**.**

There exists a constant $C>0$ such that for any $\eta_{1},\ldots,\eta_{M}\in(0,1/5]$ and any loss sequence $c_{1},\ldots,c_{T}\in[-2,2]^{KM}$, algorithm ${\mathcal{A}}$ (possibly with knowledge of $\eta_{1},\ldots,\eta_{M}$) produces sampling distributions $w_{1},\ldots,w_{T}\in\Delta_{KM}$ and ensures the following static regret bounds: for all $i\in[K]$ and $j\in[M]$:555In fact an analogue of Eq. (3) with individual learning rates would also suffice, but we are not aware of any algorithms that achieve such guarantee.

[TABLE]

Once again, this requirement is achievable by many existing algorithms and we provide some examples below (see Appendix A for proofs).

Proposition 4**.**

The following algorithms all satisfy Condition 3: A variant of Hedge (Algorithm 6 in Appendix A), Adapt-ML-Prod [19], AdaNormalHedge [27], and iProd/Squint [25].

We now state our main result for Algorithm 2 (see Appendix B.3 for the proof).

Theorem 5**.**

Suppose algorithm ${\mathcal{A}}$ satisfies Condition 3 and $\left\{{\mathcal{A}}_{ij}\right\}_{i\in[K],j\in[M]}$ all satisfy Condition 2. Algorithm 2 ensures that for any benchmark sequence $i_{1},\dots,i_{T}$ such that $\sum_{t=2}^{T}\boldsymbol{1}\{i_{t}\neq i_{t-1}\}\leq S-1$ and $|\{i_{1},\dots,i_{T}\}|\leq n$, the following hold:

• •

In the adversarial setting, we have ${\mathcal{R}}(i_{1:T})=\mathcal{O}\left(\sqrt{T(S\ln T+n\ln(K\ln T))}\right);$

• •

In the stochastic setting (defined in Section 2), we have ${\mathcal{R}}(i_{1:T})=\mathcal{O}\left(\sum_{i=1}^{n}\frac{S_{i}\ln T+\ln(K\ln T)}{\alpha_{i}}\right),$ where $S_{i}=1+\sum_{t=2}^{T}\boldsymbol{1}\left\{(i_{t-1}=i\land i_{t}\neq i)\lor(i_{t-1}\neq i\land i_{t}=i)\right\}$ s.t. $\sum_{i\in[n]}S_{i}\leq 3S$.666This definition of $S_{i}$ is the same as the one in the proof of Theorem 3.

In other words, with a negligible price of $\ln\ln T$ for the adversarial setting, our algorithm achieves logarithmic regret in the stochastic setting with favorable dependence on $S$ and $n$. The best prior result is achieved by AdaNormalHedge.TV [27], with regret $\mathcal{O}\left(\sqrt{T(S\ln(TK\ln T))}\right)$ for the adversarial case and $\mathcal{O}\left(\sum_{i=1}^{n}\frac{S_{i}\ln(TK\ln T)}{\alpha_{i}}\right)$ for the stochastic case. We also remark that a variant of the algorithm of [8] with a doubling trick can achieve a guarantee similar to ours, but weaker in the sense that each $\alpha_{i}$ is replaced by $\min_{i}\alpha_{i}$. To the best of our knowledge this was previously unknown and we provide the details in Appendix B.4 for completeness.

4 Long-term Memory under Bandit Feedback

In this section, we move on to the bandit setting where the learner only observes the loss of the selected action $\ell_{t}(I_{t})$ instead of $\ell_{t}$. As mentioned in Section 1, one could directly generalize the approach of [8, 2, 13] to obtain a bound of order $\mathcal{O}(\sqrt{TK(S\ln T+n\ln K)})$, a natural generalization of the full information guarantee, but such a bound is not a meaningful improvement compared to (2), due to the $\sqrt{K}$ dependence that is unavoidable for MAB in the worst case. Therefore, we consider a special case where the dependence on $K$ is much smaller: the sparse MAB problem [10]. Specifically, in this setting we make the additional assumption that all loss vectors are $\rho$-sparse for some $\rho\in[K]$, that is, $\left\|{\ell_{t}}\right\|_{0}\leq\rho$ for all $t$. It was shown in [10] that for sparse MAB the static regret is of order $\mathcal{O}(\sqrt{T\rho\ln K}+K\ln T)$, exhibiting a much favorable dependence on $K$.

Negative result.

To the best of our knowledge, there are no prior results on switching regret for sparse MAB. In light of bound (2), a natural conjecture would be that it would be possible to achieve switching regret of $\mathcal{O}(\sqrt{T\rho S\ln(KT)}+KS\ln T)$ with $S$ switches. Perhaps surprisingly, we show that this is in fact impossible.

Theorem 6**.**

For any $T,S,K\geq 2$ and any MAB algorithm, there exists a sequence of loss vectors that are $2$-sparse, such that the switching regret of this algorithm is at least $\Omega(\sqrt{TKS})$.

The high level idea of the proof is to force the algorithm to overfocus on one good action and thus miss an even better action later. This is similar to the construction of [15, Lemma 3] and [37, Theorem 4.1], and we defer the proof to Appendix C.1. This negative result implies that sparsity does not help improve the typical switching regret bound (2). In fact, since switching regret for MAB can be seen as a special case of the contextual bandits problem [6, 26], this result also immediately implies the following corollary, a sharp contrast compared to the positive result for the non-contextual case mentioned earlier (see Appendix C.1 for the definition of contextual bandit and related discussions).

Corollary 7**.**

Sparse losses do not help improve the worst-case regret for contextual bandits.

Long-term memory to the rescue.

Despite the above negative results, we next show how long-term memory can still help improve the switching regret for sparse MAB. Specifically, we use our general framework to develop a MAB algorithm whose switching regret is smaller than $\mathcal{O}(\sqrt{TKS})$ whenever $\rho$ and $n$ are small while $S$ and $K$ are large. Note that this is not a contradiction with Theorem 6, since in the construction of its proof, $n$ is as large as $\min\left\{S,K\right\}$.

At a high level, our algorithm (Algorithm 3) works by constructing the standard unbiased importance-weighted loss estimator $\widehat{\ell}_{t}$ (Line 3) and plugging it into our general framework (Algorithm 1). However, we emphasize that it is highly nontrivial to control the variance of these estimators without leading to bad dependence on $K$ in this framework where two types of sub-routines interact with each other. To address this issue, we design specialized sub-algorithms ${\mathcal{A}}$ and ${\mathcal{A}}_{i}$ to learn $w_{t}$ and $z_{t}(i)$ respectively. For learning $w_{t}$, we essentially deploy the algorithm of [10] for sparse MAB, which is an instance of the standard follow-the-regularized-leader algorithm with a special hybrid regularizer, combining the entropy and the log-barrier (Lines 3 and 3). However, note that the loss $c_{t}$ we feed to this algorithm is not sparse and we cannot directly apply the guarantee from [10], but it turns out that one can still utilize the implicit exploration of this algorithm, as shown in our analysis. Compared to Algorithm 1, we also incorporate an extra bias term $-\eta z_{t}(i)\widehat{\ell}_{t}(i)^{2}$ in the definition of $c_{t}$ (Line 3), which is important for canceling the large variance of the loss estimator.

For learning $z_{t}(i)$ for each $i$, we design a new algorithm that is an instance of the standard Online Mirror Descent algorithm (see e.g., [22]). Recall that this is a one-dimensional problem, as we are trying to learn the distribution $(1-z_{t}(i),z_{t}(i))$ over actions $\left\{0,1\right\}$. We design a special one-dimensional regularizer $\phi(z)=\frac{1}{\eta}\ln\frac{1}{z}$, which can be seen as a one-sided log-barrier,777The usual log-barrier regularizer (see e.g. [16, 3, 36]) would be $\frac{1}{\eta}(\ln\frac{1}{z}+\ln\frac{1}{1-z})$ in this case. to bias towards action “1”. Technically, this provides a special “local-norm” guarantee that is critical for our analysis and may be of independent interest (see Lemma 14 in Appendix C.2). In addition, we remove the bias term in the loss for action “1” (so it is only $-r_{t}(i)$ now) as it does not help in the bandit case, and we also force $z_{t}(i)$ to be at least $\delta$ for some parameter $\delta$, which is important for achieving switching regret. Line 3 summarizes the update for $z_{t}(i)$.

Finally, we also enforce a small amount of uniform exploration by sampling $I_{t}$ from $\tilde{p}_{t}$, a smoothed version of $p_{t}$ (Line 3). We present the main result of our algorithm below (proven in Appendix C.2).

Theorem 8**.**

With $\eta=\max\left\{S^{\frac{1}{3}}\rho^{-\frac{2}{3}}(nT)^{-\frac{1}{3}},\sqrt{\frac{\ln K}{T\rho}}\right\},\delta=\sqrt{\frac{S}{T\eta n}},\gamma=200K^{2}$, Algorithm 3 ensures

[TABLE]

for any sequence of $\rho$-sparse losses $\ell_{1},\ldots,\ell_{T}$ and any benchmark sequence $i_{1},\dots,i_{T}$ such that $\sum_{t=2}^{T}\boldsymbol{1}\{i_{t}\neq i_{t-1}\}\leq S-1$ and $|\{i_{1},\dots,i_{T}\}|\leq n$.

In the case when $\rho$ and $n$ are constants, our bound (9) becomes $\mathcal{O}(S^{\frac{1}{3}}T^{\frac{2}{3}}+K^{3}\ln T)$, which improves over the existing bound $\mathcal{O}(\sqrt{TKS\ln(TK)})$ when $(\frac{T}{S})^{\frac{1}{3}}<K<(TS)^{\frac{1}{5}}$ (also recall the example in Section 1 where our bound is sublinear in $T$ while existing bounds become vacuous).

As a final remark, one might wonder if similar best-of-both-worlds results are also possible for MAB in terms of switching regret, given the positive results for static regret [9, 33, 5, 32, 36, 38]. We point out that the answer is negative — the proof of [37, Theorem 4.1] implicitly implies that even with one switch, logarithmic regret is impossible for MAB in the stochastic setting.

5 Conclusion

In this work, we propose a simple reduction-based approach to obtaining long-term memory regret guarantee. By plugging various existing algorithms into this framework, we not only obtain new algorithms for this problem in the adversarial case, but also resolve the open problem of Warmuth and Koolen [35] that asks for a single algorithm achieving the best of both stochastic and adversarial environments in this setup. We also extend our results to the bandit setting and show both negative and positive results.

One clear open question is whether our bound for the bandit case (Theorem 8) can be improved, and more generally what is the best achievable bound in this case.

Acknowledgments.

The authors would like to thank Alekh Agarwal, Sébastien Bubeck, Dylan Foster, Wouter Koolen, Manfred Warmuth, and Chen-Yu Wei for helpful discussions. Kai Zheng and Liwei Wang were supported by Natioanl Key R&D Program of China (no. 2018YFB1402600), BJNSF (L172037). Haipeng Luo was supported by NSF Grant IIS-1755781. Ilias Diakonikolas was supported by NSF Award CCF-1652862 (CAREER) and a Sloan Research Fellowship.

Appendix A Examples of Sub-routines

In this section, we briefly discuss why the algorithms listed in Propositions 1, 2, and 4 satisfy Conditions 1, 2, and 3 respectively. We first note that except for AdaNormalHedge [27], all other algorithms satisfy even tighter bounds with the absolute value replaced by square (also see Footnote 3).

A.1 Condition 1

Prod [12] with learning rate $\eta$ satisfies Eq. (4) according to its original analysis. Adapt-ML-Prod [19], AdaNormalHedge [27], and iProd/Squint [25] are all parameter-free algorithms that satisfy for all $i\in[K]$,

[TABLE]

By AM-GM inequality the square root term can be upper bounded by $\frac{\ln K}{4\eta}+\eta\sum_{t=1}^{T}\left|w_{t}^{\top}c_{t}-c_{t}(i)\right|$ for any $\eta$. Also the constraint $\eta\leq 1/5$ in Condition 1 allows one to bound the extra $\ln K$ term by $\frac{\ln K}{5\eta}$. This leads to Eq. (4).

Finally, for completeness we present a variant of Hedge (Algorithm 4) that can be extracted from [20, 34] and that satisfies Eq. (3).

Proposition 9**.**

Algorithm 4 satisfies Eq. (3).

Proof.

Define $\Phi_{t}=\sum_{i=1}^{K}\exp\left(\eta R_{t}(i)-\eta^{2}G_{t}(i)\right)$ where $R_{t}(i)=\sum_{\tau=1}^{t}r_{\tau}(i)$ with $r_{\tau}(i)=w_{\tau}^{\top}c_{\tau}-c_{\tau}(i)$ and $G_{t}(i)=\sum_{\tau=1}^{t}c^{2}_{\tau}(i)$. The goal is to show $\Phi_{T}\leq\Phi_{T-1}\leq\cdots\leq\Phi_{0}=K$, which implies for any $i$, $\exp\left(\eta R_{T}(i)-\eta^{2}G_{T}(i)\right)\leq K$ and thus Eq. (3) after rearranging. Indeed, for any $t$ we have

[TABLE]

where the first inequality uses the fact $\exp(x-x^{2})\leq 1+x$ for any $x\geq-1/2$, the second inequality uses the fact $-\exp(-x)\leq x-1$ for any $x$, and the last equality holds since $w_{t}(i)\propto\exp\left(\eta R_{t-1}(i)-\eta^{2}G_{t-1}(i)\right)$ and $\sum_{i}w_{t}(i)r_{t}(i)=0$. ∎

A.2 Condition 2

We first note that the three algorithms we include in Proposition 2 all work for an arbitrary number of actions $K$ (instead of just two actions) and the general guarantee will be in the same form of Eq. (5),  (6), and (7) except that $\ln T$ is replaced by $\ln(KT)$.

Fixed-share [23] with learning rate $\eta$ satisfies Eq. (7) and the proof can be extracted from the proof of [6, Theorem 8.1] or [28, Theorem 2]. AdaNormalHedge.TV [27] is again a parameter-free algorithm and achieves the bound of (6) using similar tricks mentioned earlier for Condition 1.

Finally we provide a variant of Fixed-share that satisfies Eq. (5). The pseudocode is in Algorithm 5, where we adopt the notation from Condition 2 ($q_{t}$ for distribution, $h_{t}$ for loss, $b$ for action index) but present the general case with $K$ actions.

Proposition 10**.**

Algorithm 5 satisfies Eq. (5).

Proof.

We first write the algorithm as an instance of Online Mirror Descent. Let $\psi(q)=\sum_{b=1}^{K}q(b)\ln q(b)$ be the entropy regularizer, and $\bar{q}_{t+1}$ be such that $\nabla\psi(\bar{q}_{t+1})=\nabla\psi(q_{t})-\eta h_{t}-\eta^{2}h_{t}^{2}$ where $h_{t}^{2}$ represents the element-wise square. Then one can verify $\bar{q}_{t+1}(b)=q_{t}(b)\exp(-\eta h_{t}(b)-\eta^{2}h_{t}^{2}(b))$ and $\tilde{q}_{t+1}=\operatorname{\arg\!\min}_{q\in\Delta_{K}}D_{\psi}(q,\bar{q}_{t+1})$, where $D_{\psi}(q,q^{\prime})=\sum_{b}\left(q(b)\ln\frac{q(b)}{q^{\prime}(b)}+q^{\prime}(b)-q(b)\right)$ is the Bregman divergence associated with $\psi$. Now we have for any $q\in\Delta_{K}$,

[TABLE]

where the first inequality is by the generalized Pythagorean theorem, the second inequality is by the fact $\exp(x-x^{2})\leq 1+x$ for all $x\geq-1/2$, and the last one is by the definition of $q_{t+1}$ and the fact $\ln\frac{1}{1-\gamma}\leq 2\gamma$ for any $\gamma\leq 1/2$. Rearranging then gives

[TABLE]

A benchmark sequence with $S-1$ switches naturally divides the sequence into $S$ intervals, and for each interval $1\leq s,\ldots,e\leq T$, by summing up the inequality above from $t=s$ to $t=e$ and telescoping we have

[TABLE]

Finally summing over all intervals, setting $q$ to put all weight on the corresponding competitor, and realizing $\gamma=1/T$ finish the proof. ∎

A.3 Condition 3

To simplify notation, we use $K$ to denote the number of actions (instead of $KM$) and prove the following

[TABLE]

which clearly implies Eq. (8). Once again since Adapt-ML-Prod [19], AdaNormalHedge [27], and iProd/Squint [25] are all parameter-free algorithms satisfying Eq. (10), they also ensure Eq. (11) for any $\eta_{i}\leq 1/5$ by the same reasoning mentioned for Condition 1. Next we present a variant of Hedge (Algorithm 6) with individual learning rate for each action and prove the following.

Proposition 11**.**

Algorithm 6 satisfies Eq. (11).

Proof.

Define $\Phi_{t,i}=\exp\left(\sum_{\tau=1}^{t}(\eta_{i}r_{\tau}(i)+\eta_{i}^{2}r_{\tau}^{2}(i))\right)$. We have

[TABLE]

where the inequality holds by the fact $\exp(x-x^{2})\leqslant 1+x$ for any $x\geq-1/2$ and the equality holds because $w_{t}(i)\propto\eta_{i}\Phi_{t-1,i}$ and $\sum_{i}w_{t}(i)r_{t}(i)=0$. Therefore,

[TABLE]

Solving for $\sum_{t}r_{t}(i)$ then proves Eq. (11). ∎

Appendix B Proofs for Section 3

In this section we provide proofs and related discussions for our algorithms under full-information feedback (i.e. the expert problem).

B.1 Proof of Theorem 3

Proof.

For each distinct action $i$ in ${\mathcal{J}}=\{i_{1},\dots,i_{T}\}$, we first apply the static regret bound of ${\mathcal{A}}$ stated in Condition 1 (either Eq. (3) or Eq. (4)). With the fact $w_{t}^{\top}c_{t}=0$ and $|r_{t}(i)|\leq 2$ this gives

[TABLE]

Next we apply the switching regret bound of ${\mathcal{A}}_{i}$ stated in Condition 2 with $b_{t}=0$ if $i_{t}\neq i$ and $b_{t}=1$ otherwise (note that $q_{t}=(1-z_{t}(i),z_{t}(i))$ and $h_{t}=(0,5\eta-r_{t}(i))$). This gives with $S_{i}=1+\sum_{t=2}^{T}\boldsymbol{1}\left\{b_{t}\neq b_{t-1}\right\}$ and $T_{i}=|\{t:i_{t}=i\}|$

[TABLE]

where $B$ is

[TABLE]

In either case, using the fact $|5\eta-r_{t}(i)|\leq 3$ we have

[TABLE]

Combining this inequality with Eq. (13) and rearranging give

[TABLE]

Further combining inequalities (12) and (14) and canceling terms give

[TABLE]

Finally summing over $i\in{\mathcal{J}}$, using the fact ${\mathcal{R}}(i_{1:T})=\mathbb{E}\left[\sum_{i\in{\mathcal{J}}}\sum_{t:i_{t}=i}r_{t}(i)\right]$, $\sum_{i\in{\mathcal{J}}}S_{i}\leq 2S+n\leq 3S$, $\sum_{i\in{\mathcal{J}}}T_{i}=T$, $\left|{\mathcal{J}}\right|\leq n$ and the choice of $\eta$ finish the proof. ∎

B.2 A weaker bound via weaker conditions

Condition 1 and Condition 2 require some data-dependent regret bounds. In fact, one can even relax these conditions and replace the data-dependent regret bounds with worst-case $T$-dependent bounds, leading to a slightly weaker long-term memory guarantee. Specifically, if we replace the bounds in Condition 1 and Condition 2 by standard worst-case static and switching regret bounds

[TABLE]

respectively, then by setting $\eta=0$ in Algorithm 1 (that is, removing the bias term in the loss for ${\mathcal{A}}_{i}$) and redoing the proof of Theorem 3 in a similar way one can verify that Eq. (15) now becomes

[TABLE]

which finally leads to

[TABLE]

via Cauchy-Schwarz inequality. Compared to our bound in Theorem 3, this leads to an extra $\sqrt{n}$ factor.

B.3 Proof of Theorem 5

Proof.

The first step is to prove that for each distinct action $i\in{\mathcal{J}}=\left\{i_{1},\ldots,i_{T}\right\}$, Algorithm 2 ensures

[TABLE]

The proof is similar to that of Theorem 3. We first apply the static regret bound of ${\mathcal{A}}$ stated in Condition 3, which gives for any $i\in[K]$ and $j\in[M]$,

[TABLE]

Here we use the fact

[TABLE]

where ${\mathcal{Z}}=\sum_{i,j}w_{t}(i,j)z_{t}(i,j)$ is the normalization factor. Next we apply the switching regret bound of ${\mathcal{A}}_{ij}$ stated in Condition 2 with $\eta=\eta_{j}$, $b_{t}=0$ if $i_{t}\neq i$ and $b_{t}=1$ otherwise (note that $q_{t}=(1-z_{t}(i,j),z_{t}(i,j))$ and $h_{t}=(0,5\eta_{j}|r_{t}(i)|-r_{t}(i))$). This gives with $S_{i}=1+\sum_{t=2}^{T}\boldsymbol{1}\left\{b_{t}\neq b_{t-1}\right\}$,

[TABLE]

where $B$ is

[TABLE]

In either case, using the fact $5\eta_{j}\leq 1$ and thus $|5\eta_{j}|r_{t}(i)|-r_{t}(i)|\leq 2|r_{t}(i)|$, we have

[TABLE]

Combining this inequality with Eq. (18) and rearranging give

[TABLE]

Further combining inequalities (17) and (19) and canceling terms give

[TABLE]

Now we pick $j$ such that

[TABLE]

which is always possible by the construction of $\eta_{1},\ldots,\eta_{M}$. This proves Eq. (16).

Adversarial setting.

We simply bound $|r_{t}(i)|$ by 2 in Eq. (16). The rest is the same as the proof of Theorem 3: summing over $i\in{\mathcal{J}}$, applying Cauchy-Schwarz inequality, and using the fact ${\mathcal{R}}(i_{1:T})=\mathbb{E}\left[\sum_{i\in{\mathcal{J}}}\sum_{t:i_{t}=i}r_{t}(i)\right]$, $\sum_{i\in{\mathcal{J}}}S_{i}\leq 2S+n\leq 3S$, $\sum_{i\in{\mathcal{J}}}\sum_{t:i_{t}=i}1=T$, $\left|{\mathcal{J}}\right|\leq n$, $M=\Theta(\ln T)$ prove ${\mathcal{R}}(i_{1:T})=\mathcal{O}\left(\sqrt{T(S\ln T+n\ln(K\ln T))}\right)$.

Stochastic setting.

The proof is similar to that of [27] and solely replies on the adaptive bound (16). Recall that in the stochastic setting, without loss of generality we assume $\left\{i_{1},\ldots,i_{T}\right\}=[n]$. For each $i\in[n]$ there exists a constant gap $\alpha_{i}$ such that $\mathbb{E}_{t}\left[\ell_{t}(j)-\ell_{t}(i)\right]\geq\alpha_{i}$ for all $j\neq i$ and all $t$ such that $i_{t}=i$. This implies

[TABLE]

On the other hand, we have

[TABLE]

Combining the two inequalities above with Eq. (16) and by AM-GM inequality, we know that there exists a constant $C^{\prime}$ such that

[TABLE]

Rearranging proves

[TABLE]

and thus

[TABLE]

Summing over $i\in[n]$ finishes the proof. ∎

B.4 A weaker best-of-both-worlds result

In this section we present a version of the “Mixing Past Posteriors” algorithm of [8, 2, 13] with a particular doubling trick and show that it also provides some similar but weaker best-of-both-worlds results. As far as we know this is unknown previously.

The pseudocode is in Algorithm 7. It is a variant of Hedge where each time the sampling distribution mixes all the past distributions. We apply a standard doubling trick to the quantity $\sum_{t}\sum_{i}p_{t}(i)r_{t}^{2}(i)$, an important data-dependent quantity that turns out to be useful for adapting to the stochastic setting (similar to the role of $\sum_{t}\sum_{i}|r_{t}(i)|$ in Eq. (16)). Specifically the algorithm satisfies the following adaptive switching regret bound.

Theorem 12**.**

Algorithm 7 ensures

[TABLE]

for any loss sequence $\ell_{1},\ldots,\ell_{T}$ and benchmark sequence $i_{1},\dots,i_{T}$ such that $\sum_{t=2}^{T}\boldsymbol{1}\{i_{t}\neq i_{t-1}\}\leq S-1$ and $|\{i_{1},\dots,i_{T}\}|\leq n$. This implies that

• •

in the adversarial setting, we have ${\mathcal{R}}(i_{1:T})=\mathcal{O}\left(\sqrt{T(S\ln T+n\ln K)}\right);$

• •

in the stochastic setting (defined in Section 2), we have ${\mathcal{R}}(i_{1:T})=\mathcal{O}\left(\frac{S\ln T+n\ln K}{\min_{i\in[n]}\alpha_{i}}\right).$

Compared to our bounds in Theorem 5, one can see that the stochastic bound here is weaker in the sense that all $\alpha_{i}$’s are replaced by $\min_{i}\alpha_{i}$. At a technical level, this is because this algorithm only admits an adaptive regret bound (20) over the entire horizon, instead of a bound like Eq. (16) that holds over segments with the same competitor.

Proof.

Similar to the proof of Proposition 10, we start by writing the algorithm as an instance of Online Mirror Descent. Let $\psi(p)=\sum_{i=1}^{K}p(i)\ln p(i)$ be the entropy regularizer, and $\bar{p}_{t+1}$ be such that $\nabla\psi(\bar{p}_{t+1})=\nabla\psi(p_{t})+\eta r_{t}$. Then one can verify $\bar{p}_{t+1}(i)=p_{t}(i)\exp(\eta r_{t}(i))$ and $\tilde{p}_{t+1}=\operatorname{\arg\!\min}_{p\in\Delta_{K}}D_{\psi}(p,\bar{p}_{t+1})$, where $D_{\psi}(p,p^{\prime})=\sum_{i}\left(p(i)\ln\frac{p(i)}{p^{\prime}(i)}+p^{\prime}(i)-p(i)\right)$ is the Bregman divergence associated with $\psi$. Now we have for any $p\in\Delta_{K}$, we have

[TABLE]

Now consider a period between two resets of the algorithm that starts at time $t_{0}$ and ends at time $t_{1}$. Let $s_{t}=1+\max\{t_{0}\leq s<t:i_{s}=i_{t}\}$ be one plus the most recent time when $i_{t}$ is the competitor (if the set is empty, $s_{t}$ is defined as $1$). Note that by the definition of $p_{t}$ we have

[TABLE]

Therefore, combining previous bounds we have for any $j\in[K]$,

[TABLE]

Summing over $t$ in this period and telescoping lead to

[TABLE]

Finally suppose there are $k=\mathcal{O}(\ln T)$ periods in total, then

[TABLE]

Note that in this case by the restart condition one must also have $\sum_{t=1}^{T}\sum_{i=1}^{K}p_{t}(i)r_{t}^{2}(i)\geq 2^{k-1}$, which implies Eq. (20) (by dropping the lower order term $(n\ln K+S\ln T)\ln T$ for simplicity).

Adversarial setting.

Simply upper bound $\sum_{t=1}^{T}\sum_{i=1}^{K}p_{t}(i)r_{t}^{2}(i)$ by $4T$.

Stochastic setting.

This is similar to the proof of Theorem 5. We make the following two observations. First, by the definition of the stochastic setting we have

[TABLE]

On the other hand, we have $r_{t}^{2}(i_{t})\leq 2|\sum_{i\neq i_{t}}p_{t}(i)(\ell_{t}(i)-\ell_{t}(i_{t}))|\leq 4(1-p_{t}(i_{t}))$ and thus

[TABLE]

Combining the two inequalities above with Eq. (20) and by AM-GM inequality, we know that there exists a constant $C^{\prime}$ such that

[TABLE]

Rearranging proves

[TABLE]

and thus the claimed regret bound. ∎

Appendix C Proofs for Section 4

In this section we provide the omitted proofs for Section 4.

C.1 Negative results

Proof of Theorem 6.

Divide the whole horizon evenly into $S/2$ intervals. Our goal is to show that for any algorithm ${\mathcal{A}}$, there exists a sequence of $2$-sparse loss vectors such that the switching regret of ${\mathcal{A}}$ against a benchmark with at most 2 switches on each of these intervals is at least $\Omega(\sqrt{TK/S})$, this clearly implies that the overall switching regret against a benchmark with at most $S$ switches is at least $\Omega(\sqrt{TKS})$.

To show this, consider a fixed interval and consider the behavior of ${\mathcal{A}}$ against a fixed loss vector $-\frac{1}{2}e_{1}$ for the entire interval ($e_{i}$ represents a basis vector). Let ${\mathcal{N}}$ be the expected number of times that action 1 is not selected by ${\mathcal{A}}$ on this interval (a fixed number conditioned on everything prior to this interval). If ${\mathcal{N}}\geq\sqrt{TK/S}$, then the (static) regret of ${\mathcal{A}}$ against action 1 on this interval is already $\Omega(\sqrt{TK/S})$. Otherwise, there must exist an action $i\neq 1$ such that in expectation it is selected for less than $\frac{\sqrt{TK/S}}{K-1}\leq 2\sqrt{T/(KS)}$ times. In this case, there must also exist a subinterval of length $\frac{2T/S}{4\sqrt{T/(KS)}}=\frac{1}{2}\sqrt{TK/S}$ where in expectation action $i$ is selected for less than $1/2$ times. This means that with probability at least $1/2$, action $i$ is not selected at all on this subinterval. If we switch the loss vector from $-\frac{1}{2}e_{1}$ to $-\frac{1}{2}e_{1}-e_{i}$ starting from the beginning of this subinterval, ${\mathcal{A}}$ suffers expected regret $\Omega(\sqrt{TK/S})$ against action $i$ after the switch point. In other words, in this case the switching regret of ${\mathcal{A}}$ (first against $1$ and then against $i$) is $\Omega(\sqrt{TK/S})$, finishing the proof. ∎

To prove Corollary 7, we first remind the reader the contextual bandit setting [6, 26]. It is a generalization of the MAB problem where at the beginning of each round $t$, the learner first observes a context $x_{t}$ from some arbitrary context space ${\mathcal{X}}$, and then selects an action $I_{t}$ and observes its loss $\ell_{t}(I_{t})$. The learner is given a fixed set of policies $\Pi$ beforehand where each policy is a mapping from ${\mathcal{X}}$ to $[K]$. The (static) regret of the learner against a fixed policy $\pi\in\Pi$ is now defined as

[TABLE]

The optimal regret for a finite policy class $\Pi$ is known to be $\Theta(\sqrt{TK\ln|\Pi|})$.

It is well-known that one can reduce the problem of achieving switching regret (with $S$ switches) for MAB to the problem of achieving static regret for contextual bandit. To do this, simply let $x_{t}=t$ and $\Pi$ be the set of action sequences with length $T$ and $S$ switches. For a policy $\pi$ that corresponds to the action sequence $i_{1},\ldots,i_{T}$, its output at time $t$ is simply $\pi(x_{t})=i_{t}$. Comparing the regret definitions it is clear that the static regret for this contextual bandit problem exactly corresponds to the switching regret for MAB. Moreover, since the size of $\Pi$ in this case is $\mathcal{O}((TK)^{S})$, a static regret of form $\Theta(\sqrt{TK\ln|\Pi|})$ exactly recovers the typical switching regret bound of form (2). Now it is clear that Corollary 7 is directly implied by Theorem 6.

C.2 Proof of Theorem 8

The proof relies on the following two lemmas, which respectively state the static and switching regret guarantees for algorithm ${\mathcal{A}}$ (that learns $w_{t}$) and algorithm ${\mathcal{A}}_{i}$ (that learns $z_{t}(i)$).

Lemma 13**.**

With $\gamma=200K^{2}$, Algorithm 3 ensures for any $i\in[K]$,

[TABLE]

Lemma 14**.**

For any $i\in[K]$, Line 3 of Algorithm 3 ensures

[TABLE]

for any sequence of $r_{1}(i),\ldots,r_{T}(i)\in\mathbb{R}$ and any competitor sequence $u_{1},\ldots,u_{T}\in[\delta,1]$ with $\sum_{t=2}^{T}\boldsymbol{1}\left\{u_{t}\neq u_{t-1}\right\}\leq S_{i}-1$.

The bound in Lemma 13 resembles the one of [10] for sparse MAB, but as mentioned since $c_{t}$ is not sparse (nor can it be made sparse after shifting), it requires a different analysis. The bound in Lemma 14 contains a “local-norm” term $\sum_{t=1}^{T}z_{t}(i)r_{t}^{2}(i)$ that resembles the one achieved by Hedge in the full information setting. However, importantly this holds for any real-valued sequence of $r_{1}(i),\ldots,r_{T}(i)$, while Hedge requires the losses to be bounded from one side. We are not able to prove the same bound with the usual log barrier regularizer (see Footnote 7) either. As far as we know this lemma is new and might be of independent interest.

Combining these two lemmas we now provide the proof for Theorem 8, followed by the proofs of these lemmas.

Proof of Theorem 8.

First note that by the definition of $c_{t},r_{t}$ and $p_{t}$ one has

[TABLE]

For each distinct action $i\in{\mathcal{J}}=\{i_{1},\ldots,i_{T}\}$, applying Lemma 13 and rearranging then lead to

[TABLE]

Next we apply Lemma 14 by setting $u_{t}=\delta$ if $i_{t}\neq i$ and $u_{t}=1$ otherwise, which gives

[TABLE]

Let $\mathbb{E}_{t}$ denote the expectation conditioned on the history up to the beginning of round $t$. It is clear that $\widehat{\ell}_{t}$ is unbiased: $\mathbb{E}_{t}[\widehat{\ell}_{t}]=\ell_{t}$, and thus $\mathbb{E}_{t}[-r_{t}(i)]=\ell_{t}(i)-p_{t}^{\top}\ell_{t}(i)\leq 2.$ Also we have

[TABLE]

where the first inequality uses the fact $\ell_{t}(I_{t})\widehat{\ell}_{t}(i)\geq 0$ (since it is either [math] or $\ell_{t}(i)^{2}/\tilde{p}_{t}(i)$), the second inequality uses the definition of $\tilde{p}_{t}$, and the last one uses $\eta\leq 1/2$. Combining these with Eq. (22) and Eq. (23) gives

[TABLE]

It remains to bound

[TABLE]

which implies

[TABLE]

Summing over $i\in{\mathcal{J}}$ and using the fact $\sum_{i\in{\mathcal{J}}}S_{i}\leq 3S$ and ${\mathcal{R}}(i_{1:T})\leq\mathbb{E}\left[\sum_{t=1}^{T}r_{t}(i_{t})\right]+T\eta$ give

[TABLE]

Plugging in the parameters $\eta$ and $\delta$ proves the theorem. ∎

Proof of Lemma 13.

The proof is in similar spirit of those of [10, 11]. Define for a semi-definite matrix $M$ the associated norm for a vector $a$ as $\left\|{a}\right\|_{M}=\sqrt{a^{\top}Ma}$. By standard analysis of Follow-the-Regularized-Leader, we have for any $w\in\Delta_{K}$,

[TABLE]

where $w_{t}^{\prime}$ is some point on the segment connecting $w_{t}$ and $w_{t+1}$, and $D_{\psi}$ is the Bregman divergence associated with $\psi$. Set $w=(1-\frac{1}{T})e_{i}+\frac{1}{TK}\boldsymbol{1}$. One can verify $\mathbb{E}\left[\sum_{t=1}^{T}w^{\top}c_{t}-c_{t}(i)\right]=\mathcal{O}(K)$ and $D_{\psi}(w,w_{1})=\frac{\ln K}{\eta}+\gamma K\ln T$, and thus

[TABLE]

The rest of the proof consists of two steps. First, we prove that the algorithm is stable in the sense that $\frac{1}{2}\leq\frac{w_{t+1}(i)}{w_{t}(i)}\leq 2$ for all $t$ and $i$, which implies $\left\|{c_{t}}\right\|^{2}_{\nabla^{-2}\psi(w_{t}^{\prime})}=\mathcal{O}\left(\left\|{c_{t}}\right\|^{2}_{\nabla^{-2}\psi(w_{t})}\right)$. The second step is to show $\mathbb{E}_{t}\left[\left\|{c_{t}}\right\|^{2}_{\nabla^{-2}\psi(w_{t})}\right]=\mathcal{O}(\rho\eta)$. Combining these two steps finishes the proof.

First step.

To prove the stability, it suffices to show $\left\|{w_{t}-w_{t+1}}\right\|_{\nabla^{2}\psi(w_{t})}\leq\frac{1}{2}$. Indeed, this is because $\nabla^{2}\psi(w_{t})\succcurlyeq\gamma\left[\frac{1}{w_{t}(i)^{2}}\right]_{\text{diag}}$, where $\left[\frac{1}{w_{t}(i)^{2}}\right]_{\text{diag}}$ represents the $K$ dimensional diagonal matrix whose $i$-th diagonal element is $\frac{1}{w_{t}(i)^{2}}$, and thus $\left\|{w_{t}-w_{t+1}}\right\|_{\nabla^{2}\psi(w_{t})}\leq\frac{1}{2}$ implies $\left\|{w_{t}-w_{t+1}}\right\|_{\gamma[1/w_{t}(i)^{2}]_{\text{diag}}}\leq\frac{1}{2}$, which further implies $1-\frac{1}{2\sqrt{\gamma}}\leqslant\frac{w_{t+1}(i)}{w_{t}(i)}\leqslant 1+\frac{1}{2\sqrt{\gamma}}$ and thus $\frac{1}{2}\leq\frac{w_{t+1}(i)}{w_{t}(i)}\leq 2$.

To prove $\left\|{w_{t}-w_{t+1}}\right\|_{\nabla^{2}\psi(w_{t})}\leq\frac{1}{2}$, define $F_{t}(w)=\sum_{s=1}^{t}w_{s}^{\top}c_{s}+\psi(w)$ so that $w_{t+1}=\operatorname{\arg\!\min}_{w\in\Delta_{K}}F_{t}(w)$. We will prove $F_{t}(w^{\prime})\geq F_{t}(w_{t})$ for any $w^{\prime}$ such that $\left\|{w^{\prime}-w_{t}}\right\|_{\nabla^{2}\psi(w_{t})}=\frac{1}{2}$, which then implies $\left\|{w_{t}-w_{t+1}}\right\|_{\nabla^{2}\psi(w_{t})}\leq\frac{1}{2}$ by the convexity of $F_{t}$.

Indeed, by Taylor’s expansion, there exists some $\xi$ on the line segment joining $w^{\prime}$ and $w_{t}$, such that

[TABLE]

where the first inequality is by the first order optimality of $w_{t}$ and the second is by Hölder’s inequality. Note that $\xi$ is between $w_{t}$ and $w^{\prime}$, which implies $\left\|{\xi-w_{t}}\right\|_{\nabla^{2}\psi(w_{t})}\leq\frac{1}{2}$ and $\frac{\xi_{i}}{w_{t}(i)}\leq 1+\frac{1}{2\sqrt{\gamma}}\leq\frac{11}{10}$ similar to previous discussions. Therefore, we have $\nabla^{2}\psi(\xi)\succcurlyeq\frac{100}{121}\nabla^{2}\psi(w_{t}),$ and thus

[TABLE]

Next we show $\left\|{c_{t}}\right\|^{2}_{\nabla^{-2}\psi(w_{t})}\leq\frac{1}{25}$, which will finish the proof for the stability.

[TABLE]

Second step.

With the stability, it is clear that $\left\|{c_{t}}\right\|^{2}_{\nabla^{-2}\psi(w_{t}^{\prime})}=\mathcal{O}\left(\left\|{c_{t}}\right\|^{2}_{\nabla^{-2}\psi(w_{t})}\right)$. Now we show $\mathbb{E}_{t}\left[\left\|{c_{t}}\right\|^{2}_{\nabla^{-2}\psi(w_{t})}\right]=\mathcal{O}(\rho\eta)$. Note that this is similar to previous calculations, but the expectation allows us to bound the term by something even smaller. Specifically, we continue from the intermediate step of the previous calculation

[TABLE]

Now we use the fact $\mathbb{E}_{t}\left[\widehat{\ell}_{t}^{2}(i)\right]\leq\frac{\ell_{t}^{2}(i)}{\tilde{p}_{t}(i)}\leq\frac{2\ell_{t}^{2}(i)}{p_{t}(i)}$ and $\mathbb{E}_{t}\left[\widehat{\ell}_{t}^{4}(i)\right]\leq\frac{\ell_{t}^{2}(i)}{\tilde{p}_{t}^{3}(i)}\leq\frac{4K\ell_{t}^{2}(i)}{\eta p_{t}^{2}(i)}$ to continue with

[TABLE]

This finishes the proof. ∎

Proof of Lemma 14.

By the definition of $z_{t+1}(i)$ and first order optimality, one has

[TABLE]

which after rearranging gives

[TABLE]

Summing over $t$, telescoping, and realizing $D_{\phi}(u_{t},z_{t}(i))=\frac{1}{\eta}\left(\frac{u_{t}}{z_{t}(i)}+\ln\frac{z_{t}(i)}{u_{t}}-1\right)\leq\frac{2}{\eta\delta}$ since $u_{t}$ and $z_{t}(i)$ are in $[\delta,1]$, we arrive at

[TABLE]

It remains to prove $(z_{t+1}(i)-z_{t}(i))r_{t,i}\leq\eta\sum_{t}z_{t}(i)r_{t}^{2}(i)$. For notational convenience, given any $L,\xi\in\mathbb{R}$, let $z_{1}=\operatorname{\arg\!\min}_{z\in[\delta,1]}Lz+\phi(z)$ and $z_{2}=\operatorname{\arg\!\min}_{z\in[\delta,1]}(L+\xi)z+\phi(z)$. If we can prove $\xi(z_{1}-z_{2})\leq\eta z_{1}\xi^{2}$, then we finish the proof by setting $L=-\phi^{\prime}(z_{t}(i))$ and $\xi=-r_{t}(i)$ (which gives $z_{1}=z_{t}(i)$ and $z_{2}=z_{t+1}(i)$).

To show $\xi(z_{1}-z_{2})\leq\eta z_{1}\xi^{2}$. Realize that the optimizations are one dimensional and admit the following solutions with explicit forms

[TABLE]

The rest of the proof is simply to show $\xi(z_{1}-z_{2})\leq\eta z_{1}\xi^{2}$ holds in all of the nine possible cases.

• A.

If $z_{1}=z_{2}=1$, then $\xi(z_{1}-z_{2})=0\leq\eta z_{1}\xi^{2}$ holds trivially.

• B.

If $z_{1}=1$ and $z_{2}=\frac{1}{\eta(L+\xi)}$, then $L-\frac{1}{\eta}\leq 0$ and $\eta(L+\eta)\geq 1$ and thus

[TABLE]

• C.

If $z_{1}=1$ and $z_{2}=\delta$, then $\xi\geq 0$, $L\leq\frac{1}{\eta}$, and $\frac{1}{\eta\delta}-L\leq\xi$, and thus

[TABLE]

• D.

If $z_{1}=\frac{1}{\eta L}$ and $z_{2}=1$, then $\xi\leq 0$ and $\eta L-1\leq-\eta\xi$, and thus

[TABLE]

• E.

If $z_{1}=\frac{1}{\eta L}$ and $z_{2}=\frac{1}{\eta(L+\xi)}$, then $\frac{1}{L+\xi}\leq\eta$, and thus

[TABLE]

• F.

If $z_{1}=\frac{1}{\eta L}$ and $z_{2}=\delta$, then $\xi\geq 0$ and $\frac{1}{\eta\delta}-L\leq\xi$, and thus

[TABLE]

• G.

If $z_{1}=\delta$ and $z_{2}=1$, then $\xi\leq 0$, $\frac{1}{\eta\delta}\leq L$, and $L-\frac{1}{\eta}\leq\xi$, and thus

[TABLE]

• H.

If $z_{1}=\delta$ and $z_{2}=\frac{1}{\eta(L+\xi)}$, then $\xi\leq 0$, $\frac{1}{\eta L}\leq\delta$, $\frac{1}{\eta(L+\xi)}\leq 1$, and thus

[TABLE]

• I.

If $z_{1}=z_{2}=\delta$, then $\xi(z_{1}-z_{2})=0\leq\eta z_{1}\xi^{2}$ holds trivially.

This finishes the proof. ∎