Predictive Online Convex Optimization

TL;DR

This paper introduces a predictive step in online convex optimization that uses future gradient estimates to improve performance, especially in demand response scenarios, with proven regret bounds and empirical validation.

Contribution

It proposes a novel predictive update method for online convex optimization that leverages future gradient estimates without assuming environment predictability.

Findings

Predictive updates outperform standard methods in demand response tasks.

Regret bounds are established for the proposed algorithms.

Empirical results show improved performance over traditional online convex optimization.

Abstract

We incorporate future information in the form of the estimated value of future gradients in online convex optimization. This is motivated by demand response in power systems, where forecasts about the current round, e.g., the weather or the loads' behavior, can be used to improve on predictions made with only past observations. Specifically, we introduce an additional predictive step that follows the standard online convex optimization step when certain conditions on the estimated gradient and descent direction are met. We show that under these conditions and without any assumptions on the predictability of the environment, the predictive update strictly improves on the performance of the standard update. We give two types of predictive update for various family of loss functions. We provide a regret bound for each of our predictive online convex optimization algorithms. Finally, we apply our framework to an example based on demand response which demonstrates its superior performance to a standard online convex optimization algorithm.

Tables

Table 1. Table 1: Parameters for POCO simulations

Parameter	Value	Unit
$N$	$25$	loads
$h$	$30$	seconds
$\overline{𝐱}/h$	$\mathrm{Uniform}[1,3]$	kW
$\underset{¯}{𝐱}/h$	$-\overline{𝐱}$	kW
$𝐜$	$\mathrm{Uniform}[10,15]$	kWh
$ϵ$	$0.1$, $0.05$ & $0.01$	—
$\delta$	${10}^{-6}$	—
$\sigma$	$0.005$	—
$\eta$	$1$	—
$\gamma$	$L$	—
$\beta$	$1/L$	—

Table 2. Table 2: Different parameters for POCOb simulations

Parameter	Value
$\alpha$	$1.001$
$ϵ$	$0.1$, $0.01$ & $0.001$
$M$	$100$
$\sigma$	$5\times {10}^{-5}$
$\zeta$	$0.5$
$\beta$	$0.9$
$\eta$	$1/10\sqrt{T}$

Equations

$$\underset{{\mathbf{x}}_t\in \mathcal{X}}{\min }{f}_t({\mathbf{x}}_t)$$

$${\mathbf{x}}_{t+1}={\mathrm{proj}}_{\mathcal{X}}\left({\mathbf{x}}_t-\eta \nabla f_t({\mathbf{x}}_t)\right),$$

$${\text{Reg}}_T^d=\sum _{t=1}^T{f}_t({\mathbf{x}}_t)-f_t({\mathbf{x}}_t^{\ast }),$$

$${\mathbf{x}}_{t+1}={\mathrm{proj}}_{\mathcal{X}}\left({\overline{\mathbf{x}}}_{t+1}-{\beta }_t{\mathbf{g}}_t\right).$$

$$c_{t+1}=\{\begin{array}{ll}{c}_t+1& \text{if }\Vert {\mathbf{x}}_{t+1}-{\overline{\mathbf{x}}}_{t+1}\Vert \ge \delta \\ c_t& \text{otherwise}\end{array}$$

$${\text{Reg}}_T^d(POCO)\le {\text{Reg}}_T^d(OCO)-T\nu \delta .$$

$${\text{Reg}}_T^d(\text{POGD})$$

$$=\left(\frac{7X^2}{4}+\frac{G^2}{2}+X{V}_T-\delta \right)\sqrt{T},$$

$$\left|f_t(\mathbf{x})-f_{t+\tau }(\mathbf{x})\right|\le {\Delta }_t(\mathbf{x})|\tau |\le \Delta |\tau |$$

$${\mathbf{x}}_{t+1}={\overline{\mathbf{x}}}_{t+1}+{\beta }^m\left({\mathrm{proj}}_{\mathcal{X}}\left({\overline{\mathbf{x}}}_{t+1}-\zeta {\mathbf{g}}_t\right)-{\overline{\mathbf{x}}}_{t+1}\right)$$

$$f_{t+1}\left({\overline{\mathbf{x}}}_{t+1}+{\beta }^m{\mathbf{d}}_{t+1}\right)$$

$$+{\beta }^m\nabla f_{t+1}({\overline{\mathbf{x}}}_{t+1}{)}^{\top }{\mathbf{d}}_{t+1}.$$

$$f_t\left({\overline{\mathbf{x}}}_{t+1}+{\beta }^m{\mathbf{d}}_{t+1}\right)$$

$$-{\beta }^mϵ\Vert {\mathbf{d}}_{t+1}\Vert -2\Delta$$

$$\displaystyle\mathcal{S}=\Big{\{}\beta>0\Big{|}$$

$$\displaystyle+\beta\mathbf{g}_{t}^{\top}\mathbf{d}_{t+1}-\beta\epsilon\|\mathbf{d}_{t+1}\|-2\Delta\Big{\}}.$$

$${\text{Reg}}_T^d(POCOb)\le {\text{Reg}}_T^d(OCO)-2T\nu \Delta$$

$$\underset{{\mathbf{x}}_t\in \mathcal{X}}{\min }{\left(r_t-1^{\top }{\mathbf{x}}_t\right)}^2+\sigma {\Vert {\mathbf{s}}_{t-1}+{\mathbf{x}}_t-\frac{\mathbf{c}}{2}\Vert }^2.$$

$${\overline{\mathbf{x}}}_{t+1}$$

$${\text{Reg}}_T^d(\sigma \text{POGD})$$

$$\le O\left(V_T+1\right)-T\nu \delta .$$

$$f_t({\mathbf{x}}_t)={\left({\left[p_t-1^{\top }{\mathbf{x}}_t\right]}^{+}\right)}^2+\sigma {\Vert \alpha {\mathbf{s}}_{t-1}+{\mathbf{x}}_t-\frac{\mathbf{c}}{2}\Vert }^2$$

$$0$$

$$={\mathbf{g}}_t^{\top }({\mathbf{g}}_t-{\mathbf{e}}_t)$$

$$\Vert {\mathbf{g}}_t\Vert >\Vert {\mathbf{e}}_t\Vert \cos {\theta }_{{\mathbf{g}}_t,{\mathbf{e}}_t},$$

$$f_{t+1}(\mathbf{y})$$

$$+\frac{L}{2}{\Vert \mathbf{x}-\mathbf{y}\Vert }^2$$

$$f_{t+1}({\mathbf{x}}_{t+1}(\beta ))$$

$$+\nabla f_{t+1}({\overline{\mathbf{x}}}_{t+1}{)}^{\top }({\mathbf{x}}_{t+1}(\beta )-{\overline{\mathbf{x}}}_{t+1}).\text{vspace}-0.3cm$$

$$f_{t+1}({\mathbf{x}}_{t+1}(\beta ))$$

$$+\frac{L}{2}{\Vert {\mathbf{d}}_{t+1}\Vert }^2.\text{vspace}-0.5cm$$

$$\frac{1}{\beta }{\Vert {\mathbf{d}}_{t+1}\Vert }^2-\frac{L}{2}{\Vert {\mathbf{d}}_{t+1}\Vert }^2-ϵ\Vert {\mathbf{d}}_{t+1}\Vert \ge \delta .$$

$$\Vert {\mathbf{d}}_{t+1}\Vert =\Vert {\mathbf{x}}_{t+1}(\beta )-{\overline{\mathbf{x}}}_{t+1}\Vert \ge \frac{ϵ}{L}+\sqrt{\frac{{ϵ}^2}{L^2}+\frac{2\delta }{L}}.$$

$$i_t{\mathbb{I}}_{{\mathcal{A}}_t}=f_t({\hat{\mathbf{x}}}_t)-f_t({\mathbf{x}}_t),$$

$${\text{Reg}}_T^d(POCO)=\sum _{t=1}^T{f}_t({\mathbf{x}}_t)-f_t({\mathbf{x}}_t^{\ast }).$$

Full text

\savesymbol

AND

††thanks: This work was partly done while A. Lesage-Landry was visiting The University of Melbourne, Australia. This work was funded by the Fonds de recherche du Québec – Nature et technologies, the Ontario Ministry of Research, Innovation and Science and the Natural Sciences and Engineering Research Council of Canada. This paper was not presented at any IFAC meeting. Corresponding author A. Lesage-Landry Tel.: +1 416-978-6842; fax: +1 416-978-1145

, ,

1 Introduction

Online convex optimization (OCO) has found applications in fields like network resource allocation [8, 7, 6] and demand response in power systems [20, 21]. It is used for sequential decision-making when contextual information or feedback is only revealed to the decision maker at the end of the current round. Theoretical results showing that OCO algorithms have bounded regret guarantee the performance of these algorithms under mild assumptions.

In many applications, the decision maker has access to both revealed past information and estimates about future rounds. For example, in power systems, weather forecasts or historical load patterns can be used to estimate the future regulation needs [22, 4]. In this work, we present the predictive online convex optimization (POCO) framework. POCO works under the assumption that an estimate of the gradient of the loss function for the next round is available to the decision maker. In POCO, a standard OCO update is first applied using past information to compute the next decision. Then, the decision maker checks the quality of the estimated information available to them. If the estimated gradient is considered accurate enough, the decision maker implements an additional projected gradient step based on the estimated gradient to improve their decision for this round. This last step is referred as the predictive update.

We introduce explicit criteria for determining if the quality of the estimated gradient is high enough to guarantee an improvement over a standard OCO step when the predictive update is applied. A regret bound is obtained for all our algorithms. We conclude this work by presenting numerical examples where a POCO algorithm is used to improve on the performance of demand response with standard OCO. This example is motivated by the fact that a load aggregator often has access to an estimate of the power imbalance they have to counteract for regulation purposes.

Literature review. Recent work in online convex optimization has focused on including prior or future information. Reference [28], which builds on [9], assumes that the problem’s unknown and uncertain parameters follow a predictable process plus some noise [27] for their OCO algorithm. As in our setting, a second update with an estimated gradient-like term follows a mirror descent update. This second update is used by the algorithm in every step regardless of the quality of the estimated gradient. For this reason, the algorithm is referred to as optimistic. Optimistic algorithms were also studied in [31, 23, 34]. No conditions are provided about the estimated gradient in this case except that it comes from past observations and/or side information via an oracle. The authors of [28] show that the optimistic mirror descent can lead to a tighter bound than a standard online mirror descent algorithm if the process is indeed predictable. In [19], the authors provide a dynamic regret bound for the optimistic mirror descent. There is, however, no guarantee that in a given round the optimistic update does not do worse than the standard OCO update. An algorithm similar to [28] is given in [18]. In their work, they make the stronger assumption in which the exact gradient of the next round loss function is available and then provide a static regret bound for their setting. This differs from our setting in that we provide dynamic regret-bounded algorithms and use an estimated gradient which entails less restrictive assumptions. Several other authors have studied different ways to incorporate future information in OCO like using state information [17] or the direction of the loss function’s gradient in an online linear optimization setting [10].

The projected gradient descent, inexact gradient descent, and proximal algorithms [1, 29, 2] from conventional convex optimization resemble our setting. These algorithms differ from ours because they aim to minimize the same objective function throughout all descent steps. In OCO, we minimize a sequence of objective functions $\left\{f_{t}\right\}_{t=1}^{T}$ and at each time $t$ provide a decision to minimize the current loss function. The loss function in a given round is only observed after we have committed to a decision. OCO will be introduced formally in Section 2.

Model predictive control (MPC) [14, 3] is another widely-used sequential decision-making framework. In MPC, the decision maker solves to optimality a receding horizon optimization problem that relies on models of future round loss functions. This thus requires significantly more contextual information and computational resources. These limitations are absent in OCO, making it a more suitable tool for real-time decision making with small computational resources.

Because we characterize conditions under which the predictive step improves performance, we guarantee improvement over conventional OCO and require no predictability assumptions. These conditions can be checked at each round of OCO, and if satisfied, the predictive update is implemented. In sum, in this work we make the following contributions:

• •

We introduce a novel predictive online convex optimization framework and provide conditions for when to use side information.

• •

We propose a predictive update with a predetermined step size for loss functions that have a Lipschitz gradient. We show that this update leads to a strict improvement over an OCO update when used (Section 4).

• •

We give a predictive update with backtracking line search that applies to a broader family of problems. We show that it leads to strict improvement over an OCO update (Section 5).

• •

We obtain sublinear regret bounds in the number of rounds for all algorithms.

• •

We apply our framework to demand response in power systems and find that it outperforms a standard OCO algorithm (Section 6).

2 Background

In OCO, one must make a decision at each round to minimize their cumulative loss [30, 15]. The current round’s loss function and any other round-dependent parameters are not available at the moment when the decision is made. Only information about previous rounds can be used to make the decision. Once the decision has been made, information about the current round is observed.

Let $t$ denote the current round index and $T$ be the time horizon. Let $\mathcal{X}\subset\mathbb{R}^{N}$, $N\in\mathbb{N}$, be the decision set, and let $\mathbf{x}_{t}\in\mathcal{X}$ be the decision variable at time $t$. We denote the differentiable convex loss function by $f_{t}(\mathbf{x}_{t})$ for $t=1,2,\ldots,T$. Let $\|\cdot\|$ be the Euclidean norm. We denote the projection operator onto the set $\mathcal{Y}$ as $\operatorname{proj}_{\mathcal{Y}}(\mathbf{x})\in\operatorname*{arg\,min}_{\mathbf{y}\in\mathcal{Y}}\left\|\mathbf{x}-\mathbf{y}\right\|$.

The goal of the decision maker is to sequentially solve the following sequence of problems:

[TABLE]

for $t=1,2,\ldots,T$. The decision maker observes the loss function $f_{t}$ after choosing $\mathbf{x}_{t}$. For this reason, even if the loss function has a simple form, an analytical solution to the round optimization problem (1) is not obtainable. The decision $\mathbf{x}_{t}$ is computed using a gradient descent-based [35], mirrored descent-based [13] or Newton step-based rule [16]. For example, in the online gradient descent (OGD) [35] algorithm, the decision at round $t+1$, $\mathbf{x}_{t+1}$, is given by the update:

[TABLE]

where $\eta\propto T^{-1/2}$ to guarantee a sublinear upper bound on the dynamic regret of OGD [35, Theorem 2].

Throughout this work, we make the following assumptions [35, 30, 15].

Assumption 1**.**

The set $\mathcal{X}$ is convex and compact.

The decision set $\mathcal{X}$ represents all constraints on $\mathbf{x}_{t}$. In this version of OCO, we only consider time-invariant constraints.

Assumption 2**.**

The loss function is $B$-bounded: $|f_{t}(\mathbf{x}_{t})|\leq B$ for $t=1,2,\ldots,T$ and $B<\infty$.

Assumption 3**.**

The gradient of the loss function is $G$-bounded: $\|\nabla f_{t}(\mathbf{x}_{t})\|\leq G$ for $t=1,2,\ldots,T$ and $G<\infty$.

As a consequence of Assumption 1, the decision variable is also $X$-bounded: $\|\mathbf{x}_{t}\|\leq X$ for $t=1,2,\ldots,T$. We define the diameter of the compact set $\mathcal{X}$ as \mathrm{diam}\ \mathcal{X}=\sup\Big{\{}\|\mathbf{x}-\mathbf{y}\|\Big{|}\mathbf{x},\mathbf{y}\in\mathcal{X}\Big{\}}and let $D=\mathrm{diam}\ \mathcal{X}$, a positive scalar. The remainder of the assumptions will be stated when a specific technical result requires it.

The design tool of OCO algorithms is the regret [30, 15]. In this work, we use the dynamic regret [8, 19, 35, 24]:

[TABLE]

where $\mathbf{x}_{t}^{\ast}\in\operatorname*{arg\,min}_{\mathbf{x}\in\mathcal{X}}f_{t}(\mathbf{x})$. The dynamic regret compares the loss suffered by the decision maker to optimal performance in each round. Other versions of the regret exists, e.g., static regret [30, 15, 35], which is defined in terms of the optimal stationary decision, $\mathbf{x}^{\ast}\in\operatorname*{arg\,min}_{\mathbf{x}\in\mathcal{X}}\sum_{t=1}^{T}f_{t}(\mathbf{x})$ in (3). In this work, we only consider the dynamic regret because it yields a stronger theoretical guarantee. This theoretical guarantee is also more relevant in the context of time-varying optimization. For this reason, we refer to the dynamic regret, $\texttt{Reg}^{d}_{T}$, simply as the regret. Note that a bounded dynamic regret implies a bounded static regret [8]. The goal when designing an OCO algorithm is to show that the regret is sublinearly bounded above in the number of rounds. An OCO algorithm with a sublinearly bounded dynamic regret in the number of rounds will on average perform as well as the round optimal decision at each round [30, 15, 16].

We conclude this section by defining the quantity $V_{T}=\sum_{t=2}^{T}\left\|\mathbf{x}_{t}^{\ast}-\mathbf{x}_{t-1}^{\ast}\right\|$. The term $V_{T}$ quantifies the variation of the optimal predictions through all rounds.

3 Predictive OCO

We now introduce our POCO framework. We let $\overline{\mathbf{x}}_{t+1}\in\mathcal{X}$ be the decision computed by an OCO algorithm in round $t$. This OCO algorithm can be, for example, the aforementioned OGD. The decision $\overline{\mathbf{x}}_{t+1}$ is then given by the update (2). In POCO, we consider an $\epsilon$-forecaster introduced in Assumption 4. Let $\mathbf{g}_{t}(\overline{\mathbf{x}}_{t+1})\in\mathbb{R}^{N}$ be the estimated gradient of the loss function $f_{t+1}$ at $\overline{\mathbf{x}}_{t+1}$.

Assumption 4** ($\epsilon$-forecaster).**

The $\epsilon$-forecaster has access to an estimate of the gradient of the next round’s loss function evaluated at $\overline{\mathbf{x}}_{t+1}$, $\mathbf{g}_{t}\left(\overline{\mathbf{x}}_{t+1}\right)$, and the maximum estimation error, $\epsilon>0$, such that $\|\mathbf{g}_{t}\left(\overline{\mathbf{x}}_{t+1}\right)-\nabla f_{t+1}(\overline{\mathbf{x}}_{t+1})\|\leq\epsilon$ for all time $t=1,2,\ldots,T$.

In other words, we consider a forecaster that has access to limited information about the next round in the form of $\mathbf{g}_{t}\left(\overline{\mathbf{x}}_{t+1}\right)$. This could represent a prediction based on historical data, e.g., a weather or demand forecast. The decision maker uses this information to improve on the OCO update. For conciseness, we denote the estimated gradient by $\mathbf{g}_{t}$. We omit its dependency on $\overline{\mathbf{x}}_{t+1}$ because it is always evaluated at the OCO update output, $\overline{\mathbf{x}}_{t+1}$, and no other points. The decision maker can meet Assumption 4 by relying on an exogenous model to estimate the gradient $\nabla f_{t+1}\left(\overline{\mathbf{x}}_{t+1}\right)$. In the context of demand response, historical data of the load’s consumption and generator output’s patterns, weather history and the historical values of the gradient, for example, can be used to build a statistical model to estimate the value of the $\nabla f_{t+1}$ at the decision given by OCO update. The parameter $\epsilon$ can then be set according to, for example, a high confidence interval or a worst-case performance parameter. The forecaster would then provide $\mathbf{g}_{t}$ using this model.

Then, if certain conditions are met, the following update rule for our proposed POCO algorithm is used.

Definition 1** (Predictive update).**

Let $\beta_{t}>0$ be an appropriately chosen step size. The predictive update is

[TABLE]

The predictive update is to be used directly after the OCO update and will lead to a strict improvement over the OCO update under certain conditions. The aforementioned conditions will be discussed in the next sections and depend on the properties of the loss function. If the conditions are not met, $\overline{\mathbf{x}}_{t+1}$ is directly used. Let $\delta>0$ be the desired improvement when using the predictive update. We define the counter $c_{t}$:

[TABLE]

with $c_{0}=0$. The variable $c_{t}$ represents the number of predictive updates as described in Definition 4. Let $\nu=c_{T}/T$ be the ratio of rounds using the predictive update to the total number of rounds.

Depending on the loss function, any regret-bounded OCO update can be used in the POCO framework. Back to the OGD example, the predictive OGD uses the update (2) and if certain conditions are met, $\mathbf{x}_{t+1}$ is provided by (4) and if not, $\mathbf{x}_{t+1}=\overline{\mathbf{x}}_{t+1}$. We write $\mathbf{x}_{t+1}(\beta_{t})=\operatorname{proj}_{\mathcal{X}}\left(\overline{\mathbf{x}}_{t+1}-\beta_{t}\mathbf{g}_{t}\right)$ as a function of the step size $\beta_{t}>0$ and let $\mathbf{d}_{t+1}=\mathbf{x}_{t+1}(\beta_{t})-\overline{\mathbf{x}}_{t+1}$ be the descent direction.

Next, we provide sufficient conditions for the estimated gradient $\mathbf{g}_{t}$ to be a feasible descent direction. Later, we consider the step size selection problem. Particularly, two cases are considered where (i) the step sizes are constant and chosen a priori based on a property of the loss functions, or (ii) the step sizes are selected through the application of a backtracking line search that enforces a modified online version of the Armijo condition [2].

The following lemma introduces a sufficient condition for the estimated gradient $\mathbf{g}_{t}$ to be a descent direction of the OCO problem (1).

Lemma 1** (Estimated descent direction).**

The vector $-\mathbf{g}_{t}$ provided by the $\epsilon$-forecaster is a descent direction for $f_{t+1}(\overline{\mathbf{x}}_{t+1})$ if $\|\mathbf{g}_{t}\|>\epsilon$.

The proof of Lemma 1 is presented in Appendix A. The next lemma is adapted from [2] and ensures that the predictive step follows a feasible descent direction.

Lemma 2** (Feasible estimated descent direction).**

For all $\beta_{t}>0$ and $\overline{\mathbf{x}}_{t+1}\in\mathcal{X}$, if $\|\mathbf{g}_{t}\|>\epsilon$ and $\mathbf{x}_{t+1}(\beta_{t})\neq\overline{\mathbf{x}}_{t+1}$ , then $\mathbf{x}_{t+1}(\beta_{t})-\overline{\mathbf{x}}_{t+1}$ is a feasible descent direction at $\overline{\mathbf{x}}_{t+1}$ and $\mathbf{g}_{t}^{\top}\left(\mathbf{x}_{t+1}(\beta_{t})-\overline{\mathbf{x}}_{t+1}\right)\leq-\frac{1}{\beta_{t}}\left\|\mathbf{x}_{t+1}(\beta_{t})-\overline{\mathbf{x}}_{t+1}\right\|^{2}.$

Similarly, the proof of Lemma 2 is given in Appendix B.

4 POCO with fixed step size

We now present a predictive update where step sizes $\beta_{t}$ are fixed and based on a propriety of the sequence of loss functions. We conclude this section by providing regret-bounded algorithms using these updates. In this section, we add the following assumption:

Assumption 5**.**

Let $L<\infty$. The loss function $f_{t}(\mathbf{x})$ has an $L$-uniformly Lipschitz-continuous gradient: $\left\|\nabla f_{t}(\mathbf{x})-\nabla f_{t}(\mathbf{y})\right\|\leq L\|\mathbf{x}-\mathbf{y}\|$ for all $t=1,2,\ldots,T$ and $\mathbf{x},\mathbf{y}\in\mathcal{X}$.

We propose a predictive update with fixed step size next. We state sufficient conditions that guarantee a strict improvement over an OCO update. These sufficient conditions can be checked at each round to determine if the estimated information is accurate enough, and therefore if the predictive update should be used in the current round.

Lemma 3** (Predictive update with fixed step size).**

Suppose that Assumption 5 holds and $\|\mathbf{g}_{t}\|>\epsilon$. If $\beta\leq\frac{1}{L}$ and $\|\mathbf{d}_{t+1}\|=\|\operatorname{proj}_{\mathcal{X}}\left(\overline{\mathbf{x}}_{t+1}-\beta_{t}\mathbf{g}_{t}\right)-\overline{\mathbf{x}}_{t+1}\|\geq\frac{\epsilon}{L}+\sqrt{\frac{\epsilon^{2}}{L^{2}}+\frac{2\delta}{L}}$, then the predictive update (4) used by the $\epsilon$-forecaster strictly improves on the OCO update and the improvement is bounded below by $\delta>0$.

The proof of Lemma 3 is provided in Appendix C. We now present regret bounds for POCO algorithms. This algorithm uses the predictive update with fixed step size to improve the performance of OCO algorithms.

Theorem 1** (POCO regret bound).**

Consider an OCO algorithm with a sublinear regret upper bound. Suppose that the forecaster uses the predictive update (4) only at rounds $t$ when the estimated gradient $\mathbf{g}_{t}$ and feasible descent direction $\mathbf{d}_{t+1}=\operatorname{proj}_{\mathcal{X}}\left(\overline{\mathbf{x}}_{t+1}-\beta_{t}\mathbf{g}_{t}\right)-\overline{\mathbf{x}}_{t+1}$ satisfy the assumptions of Lemma 3. If the ratio of rounds satisfying these assumptions is greater than $\nu$, then the regret of the POCO algorithm is bounded above by

[TABLE]

The proof of Theorem 1 is presented in Appendix D. This theorem leads to the following corollary which provides a regret bound for the OGD with predictive updates (POGD).

Corollary 1** ($O\left(\sqrt{T}\right)$ regret bound for POGD).**

Suppose that the ratio $\nu$ of rounds that respects the assumptions of Lemma 3 is $\nu>\frac{1}{\sqrt{T}}$. Then the predictive OGD algorithm’s regret is bounded above by

[TABLE]

which is sublinear and tighter than the OGD regret bound.

The corollary follows from substituting $\texttt{Reg}^{d}_{T}(\texttt{OGD)}$ from [35] and $\nu>1/\sqrt{T}$ in Theorem 1.

5 POCO with backtracking line search

In this section, we do not require Assumption 5 to hold. We however use the following proposition:

Proposition 1**.**

The loss function $f_{t}(\mathbf{x})$ is $\Delta$-time-Lipschitz with $\Delta_{t}(\mathbf{x}),\Delta<\infty$, that is:

[TABLE]

for all $\tau\in\left\{\left.i\in\mathbb{Z}\right|0\leq t+i\leq T\right\}$ at all $t=1,2,\ldots,T$, and all $\mathbf{x}\in\mathcal{X}$.

Proposition 1 always holds because Assumption 2 implies that $\Delta=2B/\left|\tau\right|$ is sufficient. Under Proposition 1, we consider functions that are $(t,\mathbf{x})$-locally and globally Lipschitz in their time argument, respectively, for the intermediary bound ($\Delta_{t}(\mathbf{x})$) and the upper bound ($\Delta$). This can represent, for example, loss functions like squared tracking error functions, in which the time-varying targets are always contained in a closed set.

In the case of the POCO with backtracking (POCOb), we re-express the update (4) given in Definition 5. The backtracking line search for predictive update is given in Algorithm 1.

Definition 2** (POCOb update).**

Let $\zeta$ be a positive scalar and $\beta^{m}$ be determined by a backtracking line search algorithm. The predictive update with backtracking line search is

[TABLE]

The next lemma shows that the backtracking line search-based predictive update improves on the OCO update. Our claim relies on the modified Armijo condition for gradient projection. This condition ensures a sufficient decrease in the objective when using an estimated gradient projection descent direction [33]. We adapt this condition to the estimated gradient and online setting. The modified Armijo condition for gradient projection [2] on $f_{t+1}$ and feasible descent direction $\mathbf{d}_{t+1}=\operatorname{proj}_{\mathcal{X}}\left(\overline{\mathbf{x}}_{t+1}+\zeta\mathbf{g}_{t}\right)-\overline{\mathbf{x}}_{t+1}$ for some $\zeta>0$ with step size $\beta^{m}$ is given by:

[TABLE]

Lemma 4** (Sufficient decrease of POCOb update).**

Suppose $\|\mathbf{g}_{t}\|>\epsilon$. If Algorithm 1 terminates to a step size $\beta^{m}>0$, then the predictive update with backtracking line search (5) used by the $\epsilon$-forecaster satisfies the modified Armijo condition (6), and leads to a sufficient decrease in the loss function.

The proof of the lemma can be found in Appendix E.

Remark 1**.**

Algorithm 1 ensures that when $\beta\neq 0$, $\beta^{m}$ satisfied:

[TABLE]

Every element of (7) is available at time $t$, which is not the case in (6). This allows us to use a backtracking line search algorithm to determine $\beta_{t}$ in an OCO setting. Algorithm 1 also ensures that the step size is not too small (cf. [33, Section 3.1]).

Note that there is an additional $\epsilon\|\mathbf{d}_{t+1}\|$ term in the modified Armijo condition for estimated gradient projection. This is a consequence of not having access to the exact gradient of $f_{t}$. Hence, to ensure that the update is valid, the modified Armijo condition is augmented by a term proportional to the error of the estimated gradient. The second additional term, $2\Delta$, is due to the time-varying setting of OCO.

We now discuss the existence of step sizes $\beta_{t}$ that satisfy (7) at round $t$. Before stating the main result, for a given $\overline{\mathbf{x}}_{t+1}$ and $\mathbf{g}_{t}$, define the set of step sizes that comply with line 5 in the line search algorithm, which is the modified Armijo condition for online settings (7):

[TABLE]

Theorem 2**.**

Suppose $\mathbf{d}_{t+1}=\operatorname{proj}_{\mathcal{X}}\left(\overline{\mathbf{x}}_{t+1}-\zeta\mathbf{g}_{t}\right)-\overline{\mathbf{x}}_{t+1}\neq\mathbf{0}$ is a feasible descent direction and $f_{t}$ is bounded below for all $t$. Then there exists $\mathbf{x}\in\mathcal{X}$ such that $f_{t}(\overline{\mathbf{x}}_{t+1})-f_{t}(\mathbf{x})>2\Delta$ if and only if $\mathcal{S}\neq\emptyset$.

The proof for Theorem 2 is presented in Appendix F. We note that Theorem 2 does not guarantee that the backtracking algorithm, Algorithm 1, will find a non-zero step size. Other techniques like exact line searches, might be required to identify an adequate step size in some problem instances. Using Theorem 2, we can provide a lower bound on the improvement of the predictive update with backtracking line search.

Corollary 2** (POCOb update improvement).**

Suppose that the assumptions of Lemma 4 hold and $\beta^{m}>0$, then the predictive update with backtracking line search improves on the OCO update by a minimum of $2\Delta$.

The proof of Corollary 2 is given in Appendix G. We now state a regret bound for the POCOb algorithm.

Theorem 3** (POCOb regret bound).**

Consider an OCO algorithm with bounded regret. Suppose that the assumptions of Lemma 4 are met. If the ratio of rounds with $\beta>0$ and satisfying these assumptions to $T$ is greater than $\nu$, then the regret of the POCO algorithm with backtracking used by the $\epsilon$-forecaster is bounded above by

[TABLE]

and thus outperforms the OCO algorithm.

The proof of Theorem 3 is presented in Appendix H.

Remark 2**.**

Note that if the locally Lipschitz statement of Proposition 1 is used, then $2\Delta$ is replaced by $\Delta_{t,1}(\overline{\mathbf{x}}_{t+1}+\beta^{m}\mathbf{d}_{t+1})+\Delta_{t,1}(\overline{\mathbf{x}}_{t+1})$ in the modified Armijo condition for online settings (7), and the bound (8) can be recomputed accordingly.

6 Example

In this section, we apply POCO algorithms to demand response (DR) in power systems [5, 26], specifically regulation and curtailment. At each time step, a DR aggregator sends instructions to their loads to follow a regulation signal, e.g., a power imbalance due to a sudden change in renewable power generation [4, 32]. Each load responds to the signal by adjusting its power consumption. The power consumption is constrained by a storage capacity, which could represent physical storage like a battery or the load’s limits, e.g., thermal constraints. The regulation signal is unknown at the time the DR instructions are sent. This can be due, for example, to a drop in renewable power generation which is only assessed after the generator has committed to some amount of power. The objective of the DR aggregator is, therefore, to predict the DR dispatch at each time instance. This problem can be formulated as POCO, in which an estimate of the regulation signal is available to the load aggregator.

We consider $N$ loads. Let $\mathbf{x}_{t}\in\mathbb{R}^{N}$ denote the decision variable at round $t$. The variable $\mathbf{x}_{t}$ represents the instructions sent to the loads. Let $r_{t}\in\mathbb{R}$ be the regulation signal at time $t$. Let $\overline{\mathbf{x}},\underline{\mathbf{x}}\in\mathbb{R}^{N}$ be the maximum and minimum power that can be consumed or delivered for all loads. Define the decision set $\mathcal{X}=\left\{\left.\mathbf{x}\in\mathbb{R}^{N}\right|\underline{\mathbf{x}}\leq\mathbf{x}\leq\overline{\mathbf{x}}\right\}$. We let $\mathbf{s}_{t}\in\mathbb{R}^{N}$ denote the state of charge vectors of the loads at time $t$ and $\mathbf{c}\in\mathbb{R}^{N}$ the vector of load energy capacities. The state of charge of a load $i$ at time $t$ is $s_{t}(i)=s_{0}(i)+\sum_{n=1}^{t}x_{n}(i)$. In the current case, we assume that there is no leakage nor energy losses. The OCO problem takes the following form:

[TABLE]

The loss function has two terms: (i) a regulation term where the aggregated loads are dispatched to follow a regulation signal $r_{t}$ and (ii) a state of charge objective added to keep the loads near half their energy capacity. The loss function given in (9) is $\sigma$-strongly convex. For this reason we use the OGD for strongly convex functions ($\sigma$OGD) proposed in [24], which offers tighter regret bound than the standard OGD. The following corollary gives an upper bound on the regret of predictive OGD for strongly convex function ($\sigma$POGD).

Corollary 3** (POGD for strongly convex functions).**

Suppose $f_{t}$ is $\sigma$-strongly convex and satisfies Assumption 5 for all $t$. Consider the $\sigma$OGD update

[TABLE]

where $\eta\in(0,1]$ and $0<\gamma\leq L$. Then, the $\sigma$POGD with fixed step size, given that the assumptions of Lemma 3 hold for a ratio of the total rounds greater than $\nu$, has a regret bounded above by

[TABLE]

The result follows from the proof of Theorem 1 and the $\sigma$OGD regret bound from [24]. We now present simulation results. All optimizations are solved using CVXPY [11] and the ECOS [12] solver.

Fixed step size example.

The load and algorithm parameters for this example are gathered in Table 1. The initial state of charge of each load is set to half its capacity. The regulation signal is $r_{t}=0.2\sin\left(\frac{2\pi}{T}t\right)+w_{t}$. The parameter $w_{t}\sim\mathrm{N}(0,0.01)$ is a Gaussian noise used to model sudden changes. We assume that the aggregator has access to estimated gradient for different level of accuracy $\epsilon$. This represents, for example when $\epsilon=0.01$, a relative error of at least $4\%$ of the actual gradient norm. The parameter $\sigma$ is set to achieve adequate regulation performance without deviating too much from each load’s desired state of charge.

We now present the performance of our POCO algorithm with a fixed step size. We implement the OMD from [28] for comparison. This algorithm uses $\mathbf{g}_{t}$ without validating the estimated information. Figure 1 shows an instance of the experimental regret for the POCO with three different values of $\epsilon$, the conventional OCO algorithm, their respective regret bounds and OMD’s regret. POCO outperforms its bound, the OCO and the OMD algorithms. We remark that as expected the number of predictive updates increases with the accuracy of the estimated gradient, the performance of the POCO algorithm also improves.

Backtracking line search example.

We now present an example of POCO with backtracking. We consider a curtailment scenario. We let $p_{t}$ be the total power to be curtailed by the loads at time $t$ for $t=1,2,\ldots,T$. When a contingency occurs in the network, flexible loads are called to curtail their power consumption, e.g., by temporarily shutting down their HVAC system. Contrary to the regulation case, the loads are not contracted to follow a setpoint and no penalties are assessed on loads curtailing more than asked. Similar to the regulation setting, the curtailment signal is unknown until immediately after the current round. This setting can be modeled as POCO where an estimated curtailment signal is available to the aggregator at each round. We use the same notation as the previous examples. Let $[\cdot]^{+}=\max\{0,\cdot\}$. This curtailment scenario is modeled by loss function given below:

[TABLE]

where we have added a recovery coefficient to the state of charge objective term used previously. This coefficient models the usual evolution of the load (e.g., ambient temperature heating for a thermostatic load). We let $\alpha=1.001$. This is equivalent to a recovery coefficient of $1.13$ per hour. The function $f_{t}$ given in (10) is not gradient Lipschitz and Assumption 5 does not hold. We model the curtailment signal to be quickly increasing at first and then slowly plateauing to represent new level of available generation. This event is assumed to be limited in time, after which the network goes back to its normal state and no curtailment is then required. We let $p_{t}=0.04t^{0.3}+w_{t}$ where $w_{t}\sim\mathrm{N}(0,0.01)$ for $t=1,2,\ldots,T/4$ and then $p_{t}=0.04\left(T/4\right)^{0.3}+w_{t}^{\prime}$ where $w_{t}^{\prime}\sim\mathrm{N}(0,0.001)$ for $t=T/4,T/4+1,\ldots,T$. The noise variance is equivalent to approximatively $10\%$ of curtailment signal’s value at first and then about $1\%$.

We use the same parameters as in the previous section, except for those in Table 2. The POCOb experimental regret shown in Figure 2 is sublinear in the numbers of rounds and outperforms the OCO’s regret. While the performance is not as strong as POCO with fixed step size, this algorithm can be applied to a broader family of functions because it does not require the loss function to be gradient Lipschitz continuous. The difference in performances between the two POCO updates is explained by the fact that the sufficient conditions for the POCOb are rarely satisfied in this simulation. Improved estimation accuracy, $\epsilon$, and a loss function with a lower maximum temporal change, $\Delta$, could increase the number of times the backtracking line search is used. Nevertheless, the POCOb achieves a regret reduction of 29% when $\epsilon=1\%$ over a standard OCO algorithm. Lastly, we note that POCOb performs better for larger variations in $p_{t}$ and smaller values of $\epsilon$.

7 Conclusion

In this work, we have presented the predictive online convex optimization framework. In POCO, a second update is used after the OCO update to improve performance using an estimated gradient. We have presented two versions of the predictive update that can be used under different assumptions. We have shown a regret upper bound for all of our POCO algorithms. We have applied POCO to demand response in electric power systems and found that they outperform conventional OCO using commonly available forecast information. In the case of fixed step size update, we observed an improvement of $95\%$ in the final regret and of $29\%$ in the backtracking case when having access to a $(\epsilon=0.01)$-forecaster.

{ack}

A.L.-L. thanks P. Mancarella for his support and for co-hosting him at The University of Melbourne.

Appendix A Proof of Lemma 1

Define $\mathbf{e}_{t}\in\mathbb{R}^{n}$ as $\mathbf{e}_{t}=\mathbf{g}_{t}-\nabla f_{t+1}(\overline{\mathbf{x}}_{t+1})$ where $\|\mathbf{e}_{t}\|\leq\epsilon$ by the definition of the $\epsilon$-forecaster. A vector $\mathbf{v}$ is a descent direction if $\mathbf{v}^{\top}\nabla f_{t+1}(\overline{\mathbf{x}}_{t+1})<0$. Thus, we have

[TABLE]

Equivalently, we have $\mathbf{g}_{t}^{\top}\mathbf{g}_{t}>\mathbf{g}_{t}^{\top}\mathbf{e}_{t}$. Taking the norm of both sides and dividing by the norm of $\mathbf{g}_{t}$ gives

[TABLE]

where $\theta_{\mathbf{g}_{t},\mathbf{e}}$ is the angle between $\mathbf{g}_{t}$ and $\mathbf{e}_{t}$. By assumption, $\|\mathbf{g}_{t}\|>\epsilon$ and $\|\mathbf{e}_{t}\|\leq\epsilon$. Therefore (11) always holds and we have proved the lemma. $\square$

Appendix B Proof of Lemma 2

The identity follows from [2, Proposition 6.1.1] with $\mathbf{g}_{t}$ instead of the gradient of the loss function. It then follows from Lemma 1 that $-\beta_{t}\mathbf{g}_{t}$ with $\beta_{t}>0$ is a descent direction at $\overline{\mathbf{x}}_{t+1}$. Thus, $\mathbf{d}_{t+1}=\mathbf{x}_{t+1}(\beta_{t})-\overline{\mathbf{x}}_{t+1}$ is a feasible descent direction because $\mathbf{d}_{t+1}\in\mathcal{X}$ and $\mathbf{g}_{t}^{\top}\mathbf{d}_{t+1}<0$ for all $t$ and $\overline{\mathbf{x}}_{t+1}\in\mathcal{X}$. $\square$

Appendix C Proof of Lemma 3

By Assumption 5, $f_{t+1}$ has an $L$-Lipschitz gradient. We use the following inequality from  [25, Theorem 2.1.5]

[TABLE]

for all $\mathbf{x},\mathbf{y}\in\mathcal{X}$. We substitute $\mathbf{y}=\mathbf{x}_{t+1}(\beta)$ and $\mathbf{x}=\overline{\mathbf{x}}_{t+1}$ into (12) to obtain

[TABLE]

For the reminder of the proof, we use $\mathbf{d}_{t+1}=\mathbf{x}_{t+1}(\beta)-\overline{\mathbf{x}}_{t+1}$ to simplify the notation. We rewrite the gradient in term of the estimated gradient, which yields

[TABLE]

By assumption, $\mathbf{x}_{t+1}(\beta)\neq\overline{\mathbf{x}}_{t+1}$, which ensures that Lemma 2 holds. We use Lemma 2 to upper bound the second term of the right-hand side of (13). We then have: $f_{t+1}(\mathbf{x}_{t+1}(\beta))\leq f_{t+1}(\overline{\mathbf{x}}_{t+1})-\left(\frac{1}{\beta}-\frac{L}{2}\right)\left\|\mathbf{d}_{t+1}\right\|^{2}+\epsilon\|\mathbf{d}_{t+1}\|$. Therefore, the predictive update with fixed step size will improve on the OCO update by a minimum of $\delta>0$ if the following condition is satisfied:

[TABLE]

Assuming $0<\beta\leq\frac{1}{L}$, then $\frac{1}{\beta}\geq L$, and if $\frac{L}{2}\left\|\mathbf{d}_{t+1}\right\|^{2}-\epsilon\|\mathbf{d}_{t+1}\|\geq\delta$, then (14) also holds for any $\beta\in]0,\frac{1}{L}]$. Solving for the norm of the feasible descent direction $\|\mathbf{d}_{t+1}\|$, we have

[TABLE]

Thus, by setting $0<\beta\leq\frac{1}{L}$ and satisfying (15), we obtain $f_{t+1}(\mathbf{x}_{t+1}(\beta))\leq f_{t+1}(\overline{\mathbf{x}}_{t+1})-\delta,$ where $\delta>0$. This implies that the predictive update strictly improves over the OCO update when the feasible descent direction satisfies the condition (15). The improvement is bounded below by $\delta$. $\square$

Appendix D Proof of Theorem 1

Let $\hat{\mathbf{x}}_{t}$ denote the decision variable with $\beta_{t}=0$ for all $t$. In other words, $\hat{\mathbf{x}}_{t}$ represents the decision variable computed without the predictive algorithm. Denote the set of assumptions of Lemma 3 at round $t$ by $\mathcal{A}_{t}$. Let $\mathbb{I}_{\mathcal{A}_{t}}$ be the indicator function where $\mathbb{I}_{\mathcal{A}_{t}}=1$ if the assumptions are satisfied and [math] otherwise. Observe that the improvement, $i_{t}$, is given by

[TABLE]

where $i_{t}$ is the improvement when $\mathbb{I}_{\mathcal{A}_{t}}=1$. The regret of the POCO algorithm is

[TABLE]

Using (16), we re-express $f_{t}(\mathbf{x}_{t})$ in (17):

[TABLE]

By Lemma 3, the improvement $i_{t}$ is bounded below by $\delta$. We rewrite (18) as $\texttt{Reg}^{d}_{T}(POCO)\leq\texttt{Reg}^{d}_{T}(OCO)-\sum_{t=1}^{T}\delta\mathbb{I}_{\mathcal{A}_{t}}.$ A minimum of $T\nu$ rounds satisfy $\mathcal{A}_{t}$ and hence $\texttt{Reg}^{d}_{T}(POCO)\leq\texttt{Reg}^{d}_{T}(OCO)-T\nu\delta$. $\square$

Appendix E Proof of Lemma 4

We show that for some step size $\beta^{m}$, the estimated gradient descent projection leads to a sufficient decrease thus outperforming the OCO update. We show that if $\beta^{m}$ satisfies (7), then it also satisfies satisfies (6), ensuring a sufficient decrease in the objective function. Note that (7) is the condition under which the backtracking algorithm, Algorithm 1, is used. We can see from the left-hand side of the condition (7) that the update improves over the OCO update because the three last terms are bounded above by [math], i.e., $\mathbf{g}_{t}^{\top}\mathbf{d}_{t+1}\leq-\frac{1}{\zeta}\|\mathbf{d}_{t+1}\|^{2}<0$ by Lemma 2. Thus all three terms are less or equal to zero. By assumption, $\beta>0$, and these terms are also bounded away from zero since $\mathbf{x}_{t+1}\neq\overline{\mathbf{x}}_{t+1}$.

We start from (7) and shows it implies (6). By assumption, $\|\mathbf{e}_{t}\|\leq\epsilon$ for all $t$ and hence (7) implies,

[TABLE]

Rearranging the terms, we have

[TABLE]

By assumption, $f_{t+1}$ is time-Lipschitz with constant $\Delta>0$ for all $\mathbf{x}\in\mathcal{X}$ and all $t$. We can therefore bound below and above respectively the left-hand and right-hand side of (19). This leads to

[TABLE]

the modified Armijo condition (6). $\square$

Appendix F Proof of Theorem 2

Assume $f_{t}(\overline{\mathbf{x}}_{t+1})-f_{t}(\mathbf{x})>2\Delta$. This assumption implies that $f_{t+1}(\overline{\mathbf{x}}_{t+1})-f_{t+1}(\mathbf{x})>0$ by Proposition 1. Thus, $\overline{\mathbf{x}}_{t+1}$ is not the minimum point of $f_{t+1}$. It follows that $\nabla f_{t+1}(\overline{\mathbf{x}}_{t+1})\neq 0$. By assumption, $\mathbf{d}_{t+1}\neq\mathbf{0}$ is a feasible descent direction and we have

[TABLE]

Let $a\in(0,1)$. Subtracting $\nabla f_{t}\left(\overline{\mathbf{x}}_{t+1}+a\beta\mathbf{d}_{t+1}\right)^{\top}\mathbf{d}_{t+1}$ on both side of (20) we obtain,

[TABLE]

If the following condition holds, then (21) also holds:

[TABLE]

Under Assumption 3, for all $\mathbf{x},\mathbf{z}\in\mathcal{X}$ we have

[TABLE]

and by Assumption 1, we have $\|\mathbf{d}_{t+1}\|\leq D$. Then, if

[TABLE]

holds, so does (22). We rewrite (23) as

[TABLE]

Recalling Taylor’s Theorem [33, Theorem 2.1]:

[TABLE]

where $\mathbf{y},\mathbf{x}\in\mathcal{X}$, $\mathbf{p}\in\mathbb{R}^{n}$ and for some $a\in(0,1)$. We let $\mathbf{y}=\overline{\mathbf{x}}_{t+1}$ and $\mathbf{p}=\beta^{m}\mathbf{d}_{t+1}$. We have,

[TABLE]

We bound above the the last term of (25) using (24) and obtain

[TABLE]

By setting $\beta\leq\frac{\Delta}{GD}$ in (26), we have $f_{t}(\overline{\mathbf{x}}_{t+1}+\beta\mathbf{d}_{t+1})<f_{t}(\overline{\mathbf{x}}_{t+1})-2\Delta$. This shows that there always exists at least one point which satisfies the assumption on the existence of $\mathbf{x}\in\mathcal{X}$ such that $f_{t}(\overline{\mathbf{x}}_{t+1})-f_{t}(\mathbf{x})>2\Delta$ that is along the feasible descent direction $\mathbf{d}_{t+1}$ from $\overline{\mathbf{x}}_{t+1}$.

Next, adapting the proof of [33, Lemma 3.1] for the modified Armijo condition for online settings (7), it follows that there exists $\overline{\beta}\leq\frac{\Delta}{GD}$ such that

[TABLE]

The set $\mathcal{S}$ is therefore non-empty if there exists $\mathbf{x}\in\mathcal{X}$ such that $f_{t}(\overline{\mathbf{x}}_{t+1})-f_{t}(\mathbf{x})>2\Delta$.

We now show the converse. Assuming $\mathcal{S}\neq\emptyset$, then there exists $\underline{\beta}\in\mathcal{S}$ and

[TABLE]

holds since $\mathbf{g}_{t}^{\top}\mathbf{d}_{t+1}<0$ by Lemma 2 and $\epsilon>0$. Thus, (27) implies that there exists $\mathbf{x}\in\mathcal{X}$ such that $f_{t}(\overline{\mathbf{x}}_{t+1})-f_{t}(\mathbf{x})>2\Delta$ and one of such point is $\mathbf{x}=\overline{\mathbf{x}}_{t+1}+\beta_{2}\mathbf{d}_{t+1}$. This completes the proof. $\square$

Appendix G Proof of Corollary 2

Since $\beta>0$, then $\mathcal{S}\neq\emptyset$. By the converse of Theorem 2, we have $f_{t}(\overline{\mathbf{x}}_{t+1})-f_{t}(\mathbf{x})>2\Delta,$ where $\mathbf{x}=\overline{\mathbf{x}}_{t+1}+\beta\mathbf{d}_{t+1}$, the decision played by the predictive update (5). The predictive update hence improves on the OCO update by at least $2\Delta$. $\square$

Appendix H Proof of Theorem 3

Let $\mathbb{I}_{\mathcal{A}^{\prime}_{t}}$ be the indicator function where $\mathbb{I}_{\mathcal{A}^{\prime}_{t}}=1$ if at round $t$, $\beta>0$ and $\|\mathbf{g}_{t}\|>\epsilon$ or [math] otherwise. Using the same approach as in Theorem 1’s proof with Corollary 2, we obtain the regret bound. The last term of (8) is strictly positive and thus the POCOb regret is always bounded above by the OCO algorithm regret. $\square$