Online Convex Optimization with Perturbed Constraints

TL;DR

This paper introduces a primal-dual proximal gradient algorithm for online convex optimization with time-varying, unknown perturbations in constraints, achieving sublinear regret and constraint violation with adaptive learning rates.

Contribution

It proposes a novel algorithm that balances regret and constraint violation, handling unknown, non-i.i.d. perturbations with adaptive learning rates.

Findings

Achieves $O(T^ hreshold)$ regret and $O(T^ hreshold)$ constraint violation.

Defines regret with a time-varying set of best fixed decisions.

Supports any time horizon with adaptive learning rates.

Abstract

This paper addresses Online Convex Optimization (OCO) problems where the constraints have additive perturbations that (i) vary over time and (ii) are not known at the time to make a decision. Perturbations may not be i.i.d. generated and can be used to model a time-varying budget or commodity in resource allocation problems. The problem is to design a policy that obtains sublinear regret while ensuring that the constraints are satisfied on average. To solve this problem, we present a primal-dual proximal gradient algorithm that has $O(T^\epsilon \vee T^{1-\epsilon})$ regret and $O(T^\epsilon)$ constraint violation, where $\epsilon \in [0,1)$ is a parameter in the learning rate. Our results match the bounds of previous work on OCO with time-varying constraints when $\epsilon = 1/2$; however, we (i) define the regret using a time-varying set of best fixed decisions; (ii) can balance between regret and constraint violation; and (iii) use an adaptive learning rate that allows us to run the algorithm for any time horizon.

Tables

Table 1. Table 1: The feasible set is the set from which the fixed decision in hindsight is selected. In [ JHA16 ] , β ∈ ( 0 , 1 ) 𝛽 0 1 \beta\in(0,1) . In [ YNW17 ] , the bounds hold with probability at least 1 − δ 1 𝛿 1-\delta where δ ∈ ( 0 , 1 ) 𝛿 0 1 \delta\in(0,1) , and the feasible set is defined in expectation. In this work, ϵ ∈ [ 0 , 1 ) italic-ϵ 0 1 \epsilon\in[0,1) . The works with ( † † \dagger ) consider only static (or “offline”) constraints.

Paper	Feasible Set	Regret	Constraint Violation	Learning Rate
[MJY12]†	fixed	$O\left(T^{\{\frac{1}{2},\frac{2}{3}\}}\right)$	$O\left(T^{\{\frac{3}{4},\frac{2}{3}\}}\right)$	constant
[JHA16]†	fixed	$O\left(T^{\beta }\vee T^{1-\beta }\right)$	$O\left(T^{1-\beta /2}\right)$	adaptive
[NY17]	fixed	$O\left(\sqrt{T}\right)$	$O\left(\sqrt{T}\right)$	constant
[YNW17]	fixed∗	$O\left(\sqrt{T}\log \left(T\right)\log \left(\frac{1}{\delta }\right)\right)$	$O\left(\sqrt{T}\log \left(T\right){\log }^{3/2}\left(\frac{1}{\delta }\right)\right)$	constant
This work	time-varying	$O\left(T^{ϵ}\vee T^{1-ϵ}\right)$	$O\left(T^{ϵ}\right)$	adaptive

Equations

$$R(T)$$

$$x_{t+1}$$

$$X_T:=\{x\in C\mid g(x)+b_T⪯0\},\text{where}{b}_T^{(j)}\in [{\underline{b}}_T^{(j)},{\bar{b}}_T^{(j)}],j=1,\dots ,m$$

$$X_T^{\text{min}}\subseteq X_T\subseteq X_T^{\text{max}},$$

$$\tilde{R}(T):=\sum _{t=1}^T{f}_t(x_t)-\underset{x\in X_T}{\min }\sum _{t=1}^T{f}_t(x).$$

$$V(T)$$

$${\mathcal{L}}_t(x,y)=⟨f_t^{\prime }(x_t),x⟩+⟨y,g(x)+b_t⟩,$$

$$B_{\psi }(a,b)=\psi (a)-\psi (b)-⟨a-b,\nabla \psi (b)⟩,$$

$$\arg \underset{u\in C}{\min }{\mathcal{L}}_t(u,y_t)=\arg \underset{u\in C}{\min }⟨f_t^{\prime }(x_t),u⟩+⟨y_t,g(u)+b_t⟩=\arg \underset{u\in C}{\min }⟨f_t^{\prime }(x_t),u⟩+⟨y_t,g(u)⟩,$$

$$\tilde{R}(T)\le \frac{1}{{\rho }_T}\left(\frac{L_{\psi }}{2}{D}^2+\frac{L_{\phi }}{2}{E}^2\right)+\left(\frac{2F_{\ast }^2}{{\sigma }_{\psi }}+\frac{2G_{\ast }^2}{{\sigma }_{\phi }}\right)\sum _{t=1}^T{\rho }_t=O\left(T^{ϵ}\vee T^{1-ϵ}\right)$$

$$V(T)\le G+\frac{L_{\phi }}{2{\rho }_T}E=O(T^{ϵ})$$

$$\text{(i)}f({\bar{x}}_T)-f^{\star }\le O\left(\frac{1}{\alpha T^{1-ϵ}}+\frac{\alpha }{T^{ϵ}}\right)\text{and}\text{(ii)}\Vert [g({\bar{x}}_T)+b{]}^{+}\Vert \le O\left(\frac{1}{\alpha T^{1-ϵ}}\right)$$

$$B_{\psi }(c,a)+B_{\psi }(a,b)-B_{\psi }(c,b)=⟨\nabla \psi (b)-\nabla \psi (a),c-a⟩,$$

$$\text{(a)}{B}_{\psi }(a,b)=\psi (a)-\psi (b)-⟨a-b,\nabla \psi (b)⟩,$$

$$\text{(b)}{B}_{\psi }(c,a)=\psi (c)-\psi (a)-⟨c-a,\nabla \psi (a)⟩,$$

$$\text{(c)}-B_{\psi }(c,b)=\psi (b)-\psi (c)-⟨b-c,\nabla \psi (b)⟩.$$

$$x^{+}=\arg \underset{u\in C}{\min }\left\{\varphi (u)+\frac{1}{\rho }{B}_{\psi }(u,x)\right\},$$

$$\varphi (x^{+})-\varphi (z)\le \frac{1}{\rho }\left(B_{\psi }(z,x)-B_{\psi }(z,x^{+})-B_{\psi }(x^{+},x)\right).$$

$$f(x)-f(z)+\theta (x^{+})-\theta (z)\le \frac{1}{\rho }\left(B_{\psi }(z,x)-B_{\psi }(z,x^{+})\right)+\frac{2\rho }{{\sigma }_{\psi }}\Vert f^{\prime }(x){\Vert }_{\ast }^2.$$

$$x_{t+1}$$

$$\tilde{R}(T)\le -\sum _{t=1}^T⟨y_t,g(x_{t+1})+b_{t+1}⟩+\frac{L_{\psi }}{2{\rho }_T}{D}^2+\frac{2F_{\ast }^2}{{\sigma }_{\psi }}\sum _{t=1}^T{\rho }_t$$

$$y_{t+1}$$

$$-\sum _{t=1}^T⟨y_t,g(x_{t+1})+b_{t+1}⟩\le \frac{L_{\phi }}{2{\rho }_T}{E}^2+\frac{2G_{\ast }^2}{{\sigma }_{\phi }}\sum _{t=1}^T{\rho }_t.$$

$$V(T)\le G+\frac{L_{\phi }}{2{\rho }_T}\Vert y_T\Vert$$

$${\Psi }_t(y)=\underset{u\in C}{\min }\left\{⟨f_t^{\prime }(x_t),u⟩+\frac{1}{{\rho }_t}{B}_{\psi }(u,x_t)+⟨y,g(u)+b_{t+1}⟩\right\}.$$

$$\Vert y_t^{\ast }\Vert$$

$$S:=\{t,\dots ,t+\lceil t^{ϵ}\rceil \}t\in \mathrm{I}\mathrm{N},$$

$$\text{(i)}\log (2)\le \sum _{i\in S}{\rho }_i\le 3\text{and}\text{(ii)}\sum _{i\in S}{\rho }_i^2\le 3.$$

$$B_{\phi }(0,y_{t+k})-B_{\phi }(0,y_t)\le \chi -\eta \sum _{i\in S}{\rho }_i\Vert y_t\Vert$$

$$\Vert y_t\Vert \le E:=\sqrt{\frac{L_{\phi }}{{\sigma }_{\phi }}{\left(\frac{2\chi }{\eta }\right)}^2+\frac{2}{{\sigma }_{\phi }}\chi },$$

$$0\in {\varphi }^{\prime }(x^{+})+\frac{1}{\rho }\nabla B_{\psi }(x^{+},x)+s,$$

$$0$$

Full text

1 Introduction

The Online Convex Optimization (OCO) framework was introduced in [Zin03] and it is widely used to model applications such as spam filtering, portfolio selection, recommendation systems, among many others [Haz16]. In short, OCO consists of a sequence of games where in each round $t\in{\rm I\!N}$ an agent selects an action $x_{t}$ from a convex set $X\subset{\rm I\!R}^{n}$ and suffers a cost $f_{t}(x_{t})$, where $f_{t}:{\rm I\!R}^{n}\to{\rm I\!R}$ is convex. Crucially, the cost function is not known at the time of making a decision, and it may even be selected by an adversary after the action has been played. The goal is to design a policy or algorithm that selects a sequence of actions $\{x_{t}\}$, $t=1,\dots,T$ from $X$ so that the regret

[TABLE]

increases sublinearly, i.e., ${\lim\sup}_{T\to\infty}R(T)/T\leq 0$. Hence, the incurred cost is asymptotically as good as the best fixed decision in hindsight.111The regret captures the difference between the incurred cost and the cost obtained by an “offline” algorithm that has knowledge of all the cost functions from $t=1,\dots,T$. The decision of the offline algorithm is, however, more restricted, as it can only choose one vector from $X$.

[Zin03] showed that the online gradient descent (OGD) algorithm can obtain sublinear regret when the action set $X$ is bounded and the convex cost functions $f_{t}$, $t\in{\rm I\!N}$ have bounded subgradients. The algorithm consists of update

[TABLE]

where $\alpha_{t}=1/\sqrt{t}$ is the learning rate (or, step size), $f_{t}^{\prime}(x_{t})$ a subgradient of the previous cost function at $x_{t}$, and $\mathcal{P}_{X}$ the Euclidean projection onto the convex set $X$. Note from Eq. (2) that action $x_{t+1}$ is selected using only information available at time $t$.

1.1 OCO with long-term (LT) constraints

In the standard OCO setting, set $X$ encompasses all the constraints that an online policy and a fixed decision in hindsight must satisfy. However, sometimes it is useful to treat constraints differently depending on whether they are (i) instantaneous constraints that have to be satisfied in every iteration, or (ii) long-term constraints that must be satisfied only in the long-term or average sense (the formal definition is given in Sec. 2).

The two most prominent reasons for considering long-term constraints are the following. First, more freedom of choice. By not requiring that the long-term constraints are satisfied in every iteration, it is possible to devise policies with specific properties (e.g., lower-complexity per iteration; see Sec. 1.2) and to model new classes of OCO problems. For instance, in wireless communications systems, the power needed to transmit a message is not known a priori (as it depends on the channel conditions, the behavior of other users, etc.) and the goal is to adjust the transmission power to maximize the rate while keeping the average power consumption below a predefined threshold [MTYY09]. That is, it is possible to transmit a message using more power than what it is allowed to use on average as long as the average power consumption constraint is met.

The second reason is that they allow us to handle online constraints. More specifically, constraints that (i) change over time and (ii) are not known by the decision maker at the time of making a decision. For example, in online network flow problems with “offline” constraints (e.g., online shortest path routing [Haz16, pp. 7]) the amount of flow $x_{t}$ allocated to each of the links has to satisfy the equation $Ax_{t}=b$ in each time $t\in{\rm I\!N}$,222See [Roc84, GNT06] for an introduction to modeling different types of network flow problems. where $A$ is the routing matrix and $b$ a request vector that indicates the supply/demand of flow (of material, traffic, information, etc.) at each of the nodes. With online constraints, we have $b_{t}$ instead of $b$ (i.e., $Ax_{t}=b_{t}$) and must select $x_{t}$ without knowledge of $b_{t}$. Hence, it is not possible to design an algorithm that guarantees that $Ax_{t}$ is equal to $b_{t}$ in every iteration. An example of this type of constraint is in power allocation problems in data centers where the number of machines that run at a given time ($x_{t}$) has to be decided before the real workload ($b_{t}$) can be observed [GIN10].333See [MTYY09, Sec. 8] for a similar example with CPUs. It is important to emphasize that like the cost functions in standard OCO, the perturbation can be a function of the past actions. For instance, in network flow problems [GNT06], the previous resource allocation decisions ($x_{t-1},\dots,x_{1}$) affect the quality of experience of the users and therefore, how those generate resource requests $(b_{t})$. Another example is online display advertising [AD15] where the perturbations $b_{t}$ represent a budget that varies with time and depends on the past rewards. Or more precisely, past actions affect the rewards and, therefore, the future advertising budget.

1.2 Contributions and related work

In this paper, we consider OCO problems where the long-term constraints have additive perturbations $b_{t}$ that (i) change over time and (ii) are not known by the decision maker at the time of making a decision $x_{t}$. We do not require the perturbations to be i.i.d.  or to have any other statistical property (see assumptions in Sec. 3.2). The problem is to design and algorithm that obtains sublinear regret and ensures that the constraints are satisfied on average. The problem addressed is important because non-i.i.d. perturbations can be used to model more accurately a time-varying budget or commodity in online resource allocation problems; for instance, when the actions that the agent made in the past affect the future constraints (i.e., the perturbations).

We solve the OCO problem with perturbed constraints using a novel primal-dual proximal gradient algorithm (Algorithm 1) that obtains $O(T^{\epsilon}\vee T^{1-\epsilon})$ regret and $O(T^{\epsilon})$ accumulated constraint violation, where $\epsilon\in[0,1)$ is a parameter in the learning rate. The best decision in hindsight is selected from a feasible set444i.e., a set where the best fixed decisions satisfies the constraints on average. that changes over time depending on the perturbations. Our algorithm allows us to balance between regret and constraint violation by choosing $\epsilon$ accordingly. When $\epsilon=1/2$, our bounds match the best well-known rates [NY17], but we can also obtain faster violation rate than $O(\sqrt{T})$. For example, with $\epsilon=1/4$, we have $O(T^{3/4})$ regret and $O(T^{1/4})$ constraint violation. Another key characteristic of our algorithm is that the learning rate is adaptive. Hence, we do not need to fix in advance the time the algorithm will run (which is not known in many resource allocation problems). Furthermore, adaptive learning rates are preferable than extending the horizon with the “doubling trick” [CBL06, Sec. 2.3]. Table 1 shows a summary of how our technical approach compares to previous approaches. However, we must emphasize that the problem we address in this paper is fundamentally different from the one considered in previous work; especially, [MJY12, JHA16]. Our motivation for relaxing the constraints is not the complexity of the projection onto the feasible set, but actually that the feasible set is not known and changes over time (because of the perturbations).

Related work: The first works of OCO with long-term constraints were motivated by the complexity of the projection step in OGD. In brief, when set $X$ is composed of general convex constraints, the projection step involves solving a convex program that can be computationally burdensome. For example, projections onto the semidefinite cone. Expensive projections are dealt with in offline convex optimization by carrying them out only in the last iteration [MYJ*+*12] or less often [CGP16]. However, such approaches cannot be used in OCO problems as every action played incurs an instantaneous cost. The latter was noted in [MJY12], which formalized the OCO problem with static long-term constraints and proposed two algorithms based on variational inequalities [Nem94]. First, a gradient-based algorithm that obtains $O(\sqrt{T})$ regret and $O(T^{3/4})$ constraint violation for general OCO problems; and second, a mirror-prox algorithm that obtains $O(T^{2/3})$ regret and constraint violation when the constraints are polyhedral. The paper in [JHA16] extends the work [MJY12] by proposing an algorithm for general OCO problems with long-term constraints that can balance regret and constraint violation. In particular, the algorithm obtains $O(T^{\beta}\vee T^{1-\beta})$ regret and $O(T^{1-\beta/2})$ constraint violation where $\beta\in(0,1)$ is a design parameter. Furthermore, and unlike in [MJY12], the learning rate is adaptive, and so the algorithm can run for any time horizon.

Regarding online constraints, the work in [MTYY09] considers online learning problems with constraints that can vary in an arbitrary and possibly adversarial manner. The paper shows that the highest reward-in-hindsight while satisfying the constraints is not attainable in general, except for the case where the feasible set (i.e., the set from which the best fixed decision in hindsight is selected) is convex. The latter result motivated the work in [NY17] to consider OCO problems with convex time-varying constraints (not only with additive perturbations) and proposed an algorithm that obtains $O(\sqrt{T})$ regret and constraint violation. However, the performance of the proposed algorithm is compared to the best fixed decision in hindsight that satisfies every time-varying constraint. This is in stark contrast to our work where the feasible set changes over time depending on the perturbations. Furthermore, our feasible set always contains the feasible set in [NY17] (see Fig. 1 and Sec. 2), which means that we compare the cost of our algorithm with a larger set of (possibly better) fixed decisions in hindsight. Hence, the cost of the best fixed decision in hindsight in our work can be smaller than in [NY17]. Finally, [YNW17] considers online constraints that are i.i.d. generated where the feasible set is defined in expectation. The proposed algorithm obtains $O(\sqrt{T})$ regret and constraint violation in expectation, and $O(\sqrt{T}\log(T)\log(\frac{1}{\delta}))$ regret and $O(\sqrt{T}\log(T)\log^{3/2}(\frac{1}{\delta}))$ constraint violation bounds that hold for every sample path with probability $1-\delta$, $\delta\in(0,1)$. The algorithms in [NY17, YNW17] are based on Lyapunov techniques from stochastic network optimization [Mey08, GNT06]; the learning rate is selected based on the time the algorithm will run; and the iterations have the same complexity than the works in [MJY12, JHA16]. Hence, [NY17, YNW17] generalize and improve the bounds of previous OCO works with static long-term constraints that were motivated by computational complexity of the projections. However, unlike [JHA16], the learning rate is not adaptive.

The rest of the paper is organized as follows. Sec. 2 presents the problem model and Sec.  3 the main technical results. In Sec. 4, we present a numerical example where we show the performance of the proposed algorithm depending on $\epsilon$, and compare it to the algorithm in [NY17]. The proof of the main result, Theorem 1, is in the supplementary material.

2 Problem Model

The standard OCO framework can be extended to encompass long-term constraints with additive perturbations as follows. Let $C$ be a convex set that contains the admissible or implementable actions, and $g^{(j)}(x):{\rm I\!R}^{n}\to{\rm I\!R}$, $j\in\{1,\dots,m\}$ be a collection of convex constraints that need to be satisfied on average. Each constraint $g^{(j)}$ has an associated perturbation $b^{(j)}_{t}$ that varies with time. There is no need for the perturbations to be i.i.d., have zero mean, or any other statistical property. The only assumption we will make is that they satisfy a mild Slater condition (see Sec. 3.2). In each round $t\in{\rm I\!N}$, an agent must select an action $x_{t}\in C$ without knowledge of the cost function $f_{t}$ or the perturbation $b_{t}$.

We proceed to define the feasible set, the regret, and the constraint violation measure in our problem. To keep notation short, we let $g:=(g^{(1)},\dots,g^{(m)})$ and $b_{t}:=(b^{(1)}_{t},\dots,b^{(m)}_{t})$. We define the time-varying feasible set (i.e., the set from which we select the best fixed decision in hindsight) as

[TABLE]

with $\bar{b}^{(j)}_{T}:=\max\{b_{t}^{(j)},t\in{\rm I\!N}\}$, $j\in\{1,\dots,m\}$ and $\underline{b}_{T}:=\frac{1}{T}\sum_{t=1}^{T}b_{t}$. This is a key difference with previous work where the feasible set is fixed [MJY12] (i.e., $X_{T}^{\text{fixed}}:=\{x\in C\mid g(x)+b\preceq 0\text{ for some }b\in{\rm I\!R}^{m}$) or satisfies all the constraints [NY17] (i.e., $X_{T}^{\text{min}}:=\{x\in C\mid g(x)+\bar{b}_{T}\preceq 0\}$). Note that the by construction we have that

[TABLE]

where $X_{T}^{\text{max}}:=\{x\in C\mid g(x)+\underline{b}_{T}\preceq 0\}$ is the set of all the fixed decision that satisfy the constraints on average at time $T$. The exact value $b_{T}$ used in the definition of $X_{T}$ will be specified in Theorem 1, and will depend on the sequence of perturbations $\{b_{t}\}$.

To avoid confusion with the definition of the regret where the feasible set does not change, we define

[TABLE]

Each action $x_{t}\in C$ contributes to the aggregate constraint violation

[TABLE]

where $[z]^{+}:=(\max\{0,z^{(1)}\},\dots,\max\{0,z^{(m)}\})$ is the projection of each of the components of vector $z\in{\rm I\!R}^{m}$ onto the non-negative orthant. Similar to the regret, we would like that $V(T)$ grows at most sublinearly with $T$ so that $\lim_{T\to\infty}V(T)/T=0$. There is no requirement that $V(T)=0$ for any particular $T\in{\rm I\!N}$ or on the rate at which $V(T)$ can grow. Finally, note that if the sum of the penalties inflicted by a constraint $j\in\{1,\dots,m\}$ is non-positive (i.e., $\sum_{t=1}^{T}g^{(j)}(x_{t})+b^{(j)}_{t}\preceq 0$), then that constraint does not contribute to the aggregate constraint violation $V(T)$.

3 Main Results

3.1 Proposed algorithm and interpretation

The main technical contribution of the paper is Algorithm 1, which allows us to solve the problem presented in Sec. 2 with $O(T^{\epsilon}\vee T^{1-\epsilon})$ regret and $O(T^{\epsilon})$ constraint violation. In short, to handle long-term constraints, we define a Lagrangian-type function

[TABLE]

where $f^{\prime}_{t}(x_{t})$ is a (sub)gradient of the cost function in the previous round, and $y\in{\rm I\!R}^{m}_{+}$ a vector of dual variables. To streamline exposition, in the rest of the paper we will refer to $\mathcal{L}_{t}(x,y)$ simply as Lagrangian. Note that the Lagrangian is convex in $x$, concave in $y$, and that it depends on $t$ as the objective function and constraints change in each round. The second term of the Lagrangian can be regarded as a penalty or as an adaptive regularizer that allows us to steer the decisions towards the feasible set $X_{T}$.

Algorithm 1 is based on a regularized primal-dual proximal gradient method, where we use the general Bregman divergence as the proximal term instead of the usual squared Euclidean distance; see, for example, [BT03]. Recall the Bregman divergence associated with function $\psi$ is defined as

[TABLE]

where $\psi$ is usually assumed to be $\sigma_{\psi}$-strongly convex function and differentiable. The primal update $(\circ)$ is equivalent to carrying out an (unconstrained) proximal gradient update with the regularization term $\langle y,g(x)+b_{t}\rangle$. The regularization or penalty term is updated via the dual update ($\bullet$), which can be regarded as applying a standard proximal gradient ascent since $(g(x_{t+1})+b_{t+1})\in\partial_{y}\mathcal{L}_{t+1}(x_{t+1},y)$ for a fixed vector $x_{t+1}$. Interestingly, observe that

[TABLE]

and therefore the primal update is oblivious to perturbation $b_{t}$. Hence, the perturbation is only relevant in our algorithm in the update of the dual variables.555This is due to the fact that $y$ is the dual variable of the additive perturbation on the constraints. See Sec. D in the supplementary material for more details.

Finally, observe that we use step size $\rho$ equal to $t^{-\epsilon}$ with $\epsilon\in[0,1)$ for both updates, so there is no need to fix in advance the time horizon the algorithm will run. Note that when $\epsilon=0$, then the algorithm corresponds to using constant step size $\rho=1$. The algorithm’s complexity depends on the structure of the constraints and the functions associated with the Bregman divergence terms in the primal and dual updates. When $g(x)$ is linear (e.g., $g(x)=Ax$) and $\psi,\varphi$ equal to $\frac{1}{2}\|\cdot\|_{2}^{2}$, Algorithm 1 has the same complexity than previous work on OCO with long-term constraints [MJY12, JHA16, YNW17]. In particular, the primal and dual updates can be written as $x_{t+1}=\mathcal{P}_{C}(x_{t}-\rho\langle A^{T},y_{t}\rangle)$ and $y_{t+1}=[y_{t}+\rho(g(x_{t+1})+b_{t+1})]^{+}$.

3.2 Assumptions

The following are the necessary assumptions to establish the convergence of the proposed algorithm.

Bounded set.

Set $C$ is bounded. There exists a constant $D$ such that $\|u-v\|\leq D,\ \forall u,v\in C$.

Bounded perturbation.

$\|b_{t}\|<\infty$ for all $t\in{\rm I\!N}$.

Bounded subgradients.

Fix a norm $\|\cdot\|$ and let $\|\cdot\|_{*}$ denote its dual. There exist constants $F_{*}$, $G_{*}$, $G$ such that $\|f^{\prime}_{t}(x)\|_{*}\leq F_{*}$, $\|g(x)+b_{t}\|_{*}\leq G_{*}$, $\|g(x)+b_{t}\|\leq G$ for all $x\in C$, $t\in{\rm I\!N}$.

Slater condition.

There exists a $\eta>0$ such that $g(\hat{x})+b_{t}+\eta\mathds{1}\preceq 0$ for an $\hat{x}\in C$ and all $t\in{\rm I\!N}$.

Bregman functions.

$\psi$ and $\varphi$ are $\sigma_{\psi},\sigma_{\varphi}$-strongly convex and $L_{\psi}$, $L_{\varphi}$-smooth. Also, $\varphi$ is strictly increasing.

The first assumption is standard in OCO. The second and third assumptions are also standard in OCO and ensure that the subgradients of the of primal and dual updates are bounded. The Slater condition says that there is a set of actions that satisfy the constraints $g(x)+b_{t}$ strictly for all $t\in\{1,\dots,T\}$, and is key to ensure that the constraint violation $V(T)$ is sublinear. Importantly, the Slater condition assumption is mild in many problems. For example, when the perturbation $b_{t}$ represents the budget available at time $t$, that budget has to be always positive—independently of whether we decide to spend more (i.e., violate the constraint). Finally, the assumption that function $\psi$ and $\varphi$ are strongly convex is also standard in the definition of the Bregman divergence. The additional assumption that $\psi$ and $\varphi$ are smooth (hence, $\psi$ and $\varphi$ are upper and lower bounded by a quadratic function) is to streamline exposition in the proofs.666Technically, all we need is that $B_{\psi}(u,v)$ is uniformly upper bounded for all $u,v\in C$. Such assumption is also made in previous work and elsewhere to streamline exposition; see, e.g., [MJY12, Lemma 10], [DHS11]. The assumption that $\varphi$ is strictly increasing is necessary to obtain the faster rates on the constraint violation when $\epsilon\in[0,\frac{1}{2})$; note that this is satisfied, for example, by the squared Euclidean distance.

3.3 Bounds and discussion

Theorem 1**.**

Consider Algorithm 1 and suppose the assumptions in Sec. 3.2 are satisfied. For any $X_{T}:=\{x\in C\mid g(x)+b_{T}\preceq 0\}$ with $b_{T}\in\{{w\in[\underline{b}_{T},\bar{b}_{T}]}\mid\sum_{t=1}^{T}\langle y_{t},b_{t+1}-w\rangle\leq 0\}$, we have

[TABLE]

where $E$ is a constant that does not depend on $T$ and captures the diameter of the set in which the dual variables are contained. Specifically, $E:=\sqrt{\frac{L_{\varphi}}{\sigma_{\varphi}}\left(\frac{2\chi}{\eta}\right)^{2}+\frac{2}{\sigma_{\varphi}}\chi}$ where $\chi:=\frac{6G^{2}_{*}}{\sigma_{\varphi}}+3F_{*}D+\frac{L_{\psi}D^{2}}{2}$.

Feasible set: The parameter $w\in[\underline{b}_{T},\bar{b}_{T}]$ determines the feasible set $X_{T}$ used in the definition of the regret in Eq. (4). Observe that the condition $\sum_{t=1}^{T}\langle y_{t},b_{t+1}-w\rangle\leq 0$ always holds for $w=\bar{b}_{T}$, since then $b_{t+1}\preceq\bar{b}_{T}:=\max\{b_{t},t\in{\rm I\!N}\}$ and $y_{t}\succeq 0$ for all $t\in{\rm I\!N}$ (this case corresponds to $X_{T}=X_{T}^{\text{min}}$ as in [NY17]). If $b_{t+1}$ did not vary over time ($b_{t}=b$ for all $t\in{\rm I\!N}$), then $b_{t+1}=\underline{b}_{T}$ and therefore $\sum_{t=1}^{T}\langle y_{t},b_{t+1}-w\rangle=0$ (i.e., $X_{T}=X^{\text{max}}_{T}$ for all $T\in{\rm I\!N}$). Similarly, if $b_{t}$ were an i.i.d. random variable with expected value $b$, then $\mathbf{E}(\underline{b}_{T})=b$ and therefore $\mathbf{E}(\sum_{t=1}^{T}\langle y_{t},b_{t+1}-w\rangle)\leq 0$ for any $w\in[\underline{b}_{T},\bar{b}_{T}]$. It is difficult to characterize $X_{T}$ for other cases as this does not depend only on the sequences of perturbations $\{b_{t}\}$ but also on the learning rates (through the dual variables $\{y_{t}\}$).

Interpretation of the regret bound: The bound has the same structure than the usual OCO bounds.777See, for example, [Zin03, Theorem 1]. Or more specifically, the regret bound in the second column on page 4, i.e., $R(T)\leq\|F\|^{2}\frac{1}{2\eta_{T}}+\frac{\|\nabla c\|^{2}}{2}\sum_{t=1}^{T}\eta_{t}$. For example, for the case where $\psi$ and $\varphi$ are the squared Euclidean distance (i.e., $L_{\psi},L_{\varphi},\sigma_{\psi},\sigma_{\varphi}=2$) we have $\tilde{R}(T)\leq\frac{1}{\rho_{T}}\left(D^{2}+E^{2}\right)+(F_{*}^{2}+G_{*}^{2})\sum_{t=1}^{T}\rho_{t}$. The first term is related to the size of the sets where the primal and dual variables are contained (i.e., $D$ and $E$ respectively) and is inversely proportional to the learning rate at time $T$ (i.e., $\rho^{-1}_{T}=T^{\epsilon}$). The second term consists of the bounds on the subgradients of the cost functions ($F_{*}$) and the constraints ($G_{*}$) multiplied by $\sum_{t=1}^{T}\rho_{t}\leq 1+\int_{1}^{T}t^{-\epsilon}dt\leq 1+\int_{0}^{T}t^{-\epsilon}dt\leq 1+\frac{T^{1-\epsilon}}{1-\epsilon}$.888We write $O(T^{\epsilon}\vee T^{1-\epsilon})$ instead of $O(T^{\epsilon}\vee\frac{T^{1-\epsilon}}{1-\epsilon})$ in Theorem 1 as the interesting range is when $\epsilon\in[0,\frac{1}{2}]$. The bounds in Theorem 1 are of course useful if the constants are bounded, which is the case for $D$, $F_{*}$ and $G_{*}$ by standard OCO assumptions (see Sec. 3.2). However, for constant $E$ we need more work. To show that this constant exists is the main technical challenge of the paper; we will this discuss it in detail later in the section.

Our analysis also allow us to recover the standard OCO bound when the constraints are always satisfied. We have the following corollary to Theorem 1.

Corollary 1**.**

Suppose that $X_{T}=C$ (i.e., $g(x)+b_{t}\preceq 0$ for all $x\in C$ and $t\in{\rm I\!N}$). The bound on the regret becomes $R(T)\leq\frac{1}{\rho_{T}}\left(\frac{L_{\psi}D^{2}}{2}\right)+\left(\frac{2F_{*}^{2}}{\sigma_{\psi}}\right)\sum_{t=1}^{T}\rho_{t}=O(T^{\epsilon}\vee T^{1-\epsilon})$.

That is, when the constraints are always satisfied the dual variables will always be equal to zero and therefore $E=0$ and $G_{*}=0$.999The fact that $G_{*}=0$ follows by adding a slack variable $s_{t}$ to change the inequality constraint to equality, i.e., $g(x)+b_{t}+s_{t}=0$. Hence, by considering perturbed constraints in the learning problem we are adding $(2\rho_{T})^{-1}L_{\varphi}E^{2}+{2G_{*}^{2}}{\sigma_{\varphi}}^{-1}\sum_{t=1}^{T}\rho_{t}$ to the bound of the standard regret in Corollary 1. Such symmetry is not available in previous works [MJY12, JHA16, NY17, YNW17], and it appears in our work as Algorithm 1 can be regarded, informally, as applying OGD twice (see Lemma 3 in the supplementary material for the technical details). As a result, the constants in the usual OCO bound appear “duplicated”.

Interpretation of the constraint violation bound: The bound on the accumulated constraint violation $V(T)$ consists of two terms. The first term is a constant related to the constraints, and the second term depends on constants $E$ and $L_{\varphi}$, and are divided by the learning rate at time $T$. Hence, if $\epsilon=0$ we have that $V(T)\leq O(1)$; however, constant constraint violation comes at the price of the regret not being sublinear. Also, observe that for any $\epsilon$ in the range $[0,\frac{1}{2})$, the constraint violation has better rate than the regret.

Constant $E$: This constant is analogous to constant $D$, which measures the maximum distance between any two vectors in the bounded set $C$ of primal variables; see Sec. 3.2. However, we cannot define $E$ in the same way as the dual variables exist in the nonnegative orthant (which is an unbounded set). Instead, we show that the difference between the vectors generated by the dual update in Algorithm 1 is bounded (not any two vectors in ${\rm I\!R}^{m}_{+}$). Or equivalently, that the sequence of dual variables obtained with Algorithm 1 remains bounded for all $t\in{\rm I\!N}$; see Lemma 7 in the supplementary material.

To ensure that $\|y_{t}\|$ is bounded for any $t\in{\rm I\!N}$, we rely on the Slater condition. In brief, this condition requires that there exists an $x\in C$ such that $g(x)+b_{t}+\eta\mathds{1}\preceq 0$ for some scalar $\eta>0$, and ensures that the dual variables in Algorithm 1 are attracted to a bounded set within ${\rm I\!R}^{m}_{+}$.101010This type of behavior is typical in dual subgradient methods. See, for example, Figure 8.2.6. in (Bertsekas et al. 2003). This is also discussed in detail in [NO09]; see Lemma 1 in [NO09]. And since $y_{t}$ at $t=1$ is bounded, the sequence of dual variables will remain bounded. The technical challenge is to characterize the diameter of the set to which these dual variables are attracted since unlike standard optimization with a fixed objective function, in OCO the cost functions vary over time and, indirectly, the (bounded) sets to which the dual variables are attracted. See Proposition 2 and discussion in Section B.2 in the supplementary material.

Finally, we note that $E=O(D^{2})$. Observe that when $\psi$ and $\varphi$ are the squared Euclidean distance, $E$ gets simplified to $\sqrt{{2\chi^{2}}{\eta^{-2}}+\chi}$ where $\chi:=\frac{6G^{2}_{*}}{\sigma_{\varphi}}+3F_{*}D+\frac{L_{\psi}D^{2}}{2}$. This last observation implies that $\tilde{R}(T)\leq O(D^{4}(T^{\epsilon}\vee T^{1-\epsilon}))$.

Constrained convex optimization: Our results can also be applied to constrained optimization problems. The following corollary to Theorem 1 establishes the convergence of a constrained convex program with relaxed constraints and primal averaging.

Corollary 2**.**

Consider the setup of Theorem 1 where the objective function and constraints are constant (i.e., $f_{t}=f$ and $b_{t}=b$ for all $t\in{\rm I\!N}$) and step size $\rho_{t}=\alpha t^{-\epsilon}$ with $\alpha>0$. We have that

[TABLE]

where $f^{\star}:=\min_{x\in X}f(x)$ with $X=\{x\in C\mid g(x)+b\preceq 0\}$ and $\bar{x}_{T}:=\frac{1}{T}\sum_{t=1}^{T}x_{t}$.

The result recovers the upper bound on the objective and constraint violation in Proposition 1 in [NO09] when $\epsilon=0$ (fixed step size), but also ensures that $f(\bar{x}_{T})\to f^{\star}$ and $\bar{x}_{T}$ converges to a vector in $X$ asymptotically as $T\to\infty$ for any $\epsilon\in(0,1)$.

4 Numerical Example

We present a variation of the example in [YNW17, Sec. 5] where the actions made by the decision maker affect the constraints. In short, consider a geo-distributed datacenter that consists of a front-end router and $n$ clusters distributed in different geographical zones. Jobs arrive in the front-end router and must be scheduled to one of the clusters. The cost of executing jobs depends on the electricity cost of running each of the clusters—which varies across sites as each cluster buys power from its local market. The goal is to schedule jobs to clusters to minimize the total electricity cost while ensuring that all the jobs are served. Importantly, the cost of electricity is not known at the time to schedule jobs. We model the problem above as an OCO as follows. Divide time in slots of equal duration and let $x_{t}\in[0,1]^{n}$ be the fraction that each of the clusters is utilized at time $t\in{\rm I\!N}$. The cost functions are assumed to be linear (i.e., $f_{t}(x_{t})=\langle l_{t},x_{t}\rangle$ with $l_{t}\in{\rm I\!R}^{n}_{+}$) and capture the price of electricity. The constraints are given by $b_{t}\leq\langle a,x_{t}\rangle$, where $a\in{\rm I\!R}^{n}_{+}$ captures the efficiency of each cluster (i.e., the number of jobs it can serve per time slot) and $b_{t}$ the jobs that arrive in the front-end router at time $t$.

We run a simulation with $n=10$ clusters, $\{l_{t}\}\in[0,1]^{n}$, and $b_{t}=\frac{1}{2}\langle\mathds{1},a\rangle e^{-l_{t-1}(x_{t-1})}$. Hence, the jobs that arrive in a time slot depend on the cost in the previous iteration. The simulation results are shown in Fig. 2, where we evaluate the performance of our algorithm with $\epsilon\in\{0,\frac{1}{4},\frac{1}{2},\frac{3}{4}\}$ and compare it with the algorithm in [NY17] (indicated in blue). First, observe that by selecting $\epsilon\in\{0,\frac{1}{4},\frac{1}{2}\}$, we can trade regret for constraint violation. In particular, with small $\epsilon$, we are sacrificing regret for lower constraint violation—in this example, lower constraint violation means that the jobs have to wait less time at the front-end router before they can be served. When $\epsilon$ is larger than $1/2$, we obtain that our algorithm has a surprising behavior that was not observed in previous work: the regret and constraint violation have a sine-wave form where the period of the wave increases with time. The latter is shown in the Fig. 2 with $\epsilon=3/4$, but we observe the same behavior for other values of $\epsilon$ larger than $1/2$. Another interesting observation is the temporal tradeoff between the regret and constraint violation when $\epsilon=3/4$; observe from the figure that the peaks of the regret correspond to the lowest values of the constraint violation and vice versa. The lines in light blue show the performance of the algorithm in [NY17], which compares to our algorithm when $\epsilon=1/2$ since then both algorithms have $O(\sqrt{T})$ regret and constraint violation.111111To compare the results fairly, we select the best policy in hindsight from $X_{T}$ (the time-varying feasible set in Eq. (3)) instead of the fixed policy in hindsight that satisfies each constraint individually, i.e., $X_{T}^{\text{min}}$. Observe from the figure that our algorithm has significant smaller regret and constraint violation. This result matches the improvements observed in [JHA16] with respect to previous approaches that used a constant learning rate [MYJ*+*12]. Finally, in Fig. 2c, we show the cost in hindsight depending on whether the best fixed decision is selected from $X_{T}^{\text{min}}$, $X_{T}$, or $X_{T}^{\text{max}}$ with $\epsilon\in\{\frac{1}{2},0.99\}$. Recall that $X_{T}^{\text{min}}\subseteq X_{T}\subseteq X_{T}^{\text{max}}$ and therefore $\min_{x\in X_{T}^{\text{min}}}\sum_{t=1}^{T}f_{t}(x)\geq\min_{x\in X_{T}^{\text{}}}\sum_{t=1}^{T}f_{t}(x)\geq\min_{x\in X_{T}^{\text{max}}}\sum_{t=1}^{T}f_{t}(x).$ Observe from the figure that when $\epsilon=1/2$, we obtain that the costs with $X_{T}$ and $X_{T}^{\text{max}}$ are exactly the same (i.e., the pink dots are exactly on top of the yellow line), whereas when $\epsilon=0.99$, the costs do not coincide exactly ($t\geq 500$). Importantly, notice the large difference between the costs when the best fixed decisions in hindsight are selected from $X_{T}^{\text{min}}$ and $X_{T}$. Finally, we note that the sets change over time and are affected by the actions made the decision maker. Observe that when $t=200$, the cost of the best fixed decision in hindsight is larger with $\epsilon=0.5$ than with $\epsilon=0.99$ (i.e., $X_{T}(0.5)\subset X_{T}(0.99)$); however, we have the opposite when $t=700$ (i.e., $X_{T}(0.99)\subset X_{T}(0.5)$).

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 795244.

Appendices

The supplementary material is divided into four sections. Sec. A contains the preliminaries and background material. In Sec. B, we present the proof of Theorem 1, and in Sec. C the proofs of the lemmas and propositions in Sec. A and B. Finally, in Sec. D, we provide additional background material on Lagrange duality and explain how this connects to our approach.

Appendix A Preliminaries

A.1 Notation

We use ${\rm I\!N}$, ${\rm I\!R}_{+}$ and ${\rm I\!R}^{n}$ to denote the set of natural numbers, nonnegative real numbers and $n$-dimensional real vectors. The symbol $\mathds{1}$ indicates the all ones column vector—the dimension of the vector will be implied by the context. We use $f^{\prime}(x)\in\partial f(x)$ to indicate a subgadient in the subdifferential of $f$ at $x$, and $x_{t}$ to indicate the $t$’th element in a sequence $\{x_{t}\}$. For two vectors $u,v\in{\rm I\!R}^{n}$, we write $u\succeq v$ to indicate that $u$ is element-wise larger than or equal to $v$. The inner product between vectors $u,v$ is indicated with $\langle u,v\rangle$.

A.2 Bregman divergence

Let $B_{\psi}(a,b)$ be the Bregman divergence as defined in Eq. (7). We recall the following well-known identity [CT93],

[TABLE]

which is a generalization of the quadratic identify for the Euclidean norm. It is easy to derive Eq. (8) from the definition of the Bregman divergence in Eq. (7). Observe that

[TABLE]

A.3 Online proximal gradient method

Let $\phi:{\rm I\!R}^{n}\to{\rm I\!R}$ be a convex function. The following update

[TABLE]

corresponds to the standard proximal method where we have replaced the squared Euclidean distance with the Bregman distance. The following two lemmas are variations of well-known results and correspond to the proximal method and proximal gradient method (or, mirror-descent); see [Van16]. We state them to measure the progress in one iteration.

Lemma 1** (One iteration proximal method).**

Consider the proximal update in Eq. (9) where $\phi:{\rm I\!R}^{n}\to{\rm I\!R}$ is a convex function, $C\subset{\rm I\!R}^{n}$ a convex set (not necessarily bounded), and $\rho>0$. For any $z\in C$, the following bound holds

[TABLE]

Obviously, since the bound holds for any $z\in C$, it also holds for the $z$ that minimizes $\phi(z)$. Next, we apply the result in Lemma 1 to the online setting. Let $f$ and $\theta$ be two convex functions from ${\rm I\!R}^{n}\to{\rm I\!R}$ and let $\phi(u)=\langle f^{\prime}(x),u\rangle+\theta(u)$, where $f^{\prime}(x)\in\partial f(x)$. Function $\theta$ can be regarded as a penalty function or regularizer, and we can ignore it if $\theta(u)=0$ for all $u\in C$. We will relate $\theta$ with the second term of the Lagrangian in Eq. (6) in the next section.

Lemma 2** (One iteration proximal gradient method).**

Consider the setup of Lemma 1 where $\phi(u)=\langle f^{\prime}(x),u\rangle+\theta(u)$ in Eq. (9). For any $z\in C$, the following bound holds

[TABLE]

The key point from the last lemma is that by only using a subgradient of $f$ we can recover a bound on $f$ itself. Also, observe that if we let $\theta(x)=0$ for all $x\in C$; index the objective and the step size with $t$ (i.e., let $f=f_{t}$ and $\rho=\rho_{t}$); and sum from $t=1,\dots,T$, we can follow the rationale of the proof of Theorem 1 in [Zin03]121212Shown also in the proof of Lemma 3. to recover the standard OCO bound (bound given in Corollary 1).

Appendix B Proof of Theorem 1

This section contains the technical results that support the claims in Sec.  3. It is divided into two parts. In Sec. B.1, we prove a bound on the regret and constraint violation and show how these depend on the boundedness of the dual variables. Sec. B.2 shows that the dual variables remain uniformly bounded with Algorithm 1, which is the main technical challenge of the paper.

B.1 Regret and constraint violation

As explained in Sec. 3, the update

[TABLE]

is a generalization of Zinkevich’s online gradient descent, and it can be regarded as a proximal gradient update (or, mirror descent) with a regularizer. We can use Lemma 2 to obtain the following result.

Lemma 3**.**

Consider the update in Eq. (11) and let $\{y_{t}\}_{t=1}^{T}$ be an arbitrary sequence of vectors from ${\rm I\!R}^{m}_{+}$. The following bound holds

[TABLE]

From Lemma 3, we obtain the usual bound on the regret131313See the discussion after Theorem 1. with the additional term $-\sum_{t=1}^{T}\langle y_{t},g(x_{t+1})+b_{t+1}\rangle$ due to the regularizer we have added in the update. To make the term vanish, we can apply a proximal gradient update

[TABLE]

since $\mathcal{L}_{t}(x_{t+1},y)$ is concave in $y$ for a fixed $x_{t+1}$. The following result is also an application of Lemma 2.

Lemma 4**.**

Consider the update in Eq. (12) and suppose there exists a constant $E$ such that $\|y_{t}\|\leq E$ for all $t\in{\rm I\!N}$. The following bound holds

[TABLE]

Combining Lemmas 3 and 4, we obtain the bound on the regret in Theorem 1. It only remains to show that constant $E$ exists and does not depend on $t$. Before we proceed to do that, we establish the bound on the constraint violation.

Proposition 1** (Constraint violation).**

Select $x_{1}\in C$. The updates in Eq. (11) and (12) have constraint violation

[TABLE]

where $\|g(x)+b_{t}\|\leq G$ for all $x\in C$ and $t\in{\rm I\!N}$.

The proof of the proposition is based as well on OGD arguments. Observe from Eq. (13) that if $\|y_{t}\|\leq E$ for all $t\in{\rm I\!N}$ (as assumed to obtain the result in Lemma 4), then we obtain the claimed bound on the constraint violation in Theorem 1. We show that constant $E$ exists in the next section.

B.2 Bounded dual variables

We take as starting point a classic result from Lagrange duality in constrained convex optimization, which says that the set of optimal dual variables is bounded when the Slater condition holds [Uza58]. This result is important because when we solve the dual problem with an iterative method, such as the subgradient method,141414See Sec. 8.2 in [BNO03] for a detailed explanation of the convergence of the dual subgradient method. See also Lemma 1 and 3 in [NO09]. we obtain a sequence of dual variables that is attracted to a bounded set. Hence, the dual variables remain bounded for all $t\in{\rm I\!N}$.

In our problem, since the Slater condition holds for every constraint (see Sec. 3.2), we could in principle use the same methodology than in offline constrained convex optimization by defining the time-varying Lagrange dual function

[TABLE]

However, that is not possible because each dual function depends on the previous one (through the objective $\langle f^{\prime}_{t}(x_{t}),u\rangle+{\rho_{t}}^{-1}B_{\psi}(u,x_{t})$), which correlates the set of dual solutions. As a result, we cannot establish that each set $\arg\max_{y\in{\rm I\!R}^{m}_{+}}\Psi_{t}(y)$, $t\in{\rm I\!N}$ is uniformly bounded. We show this formally in the following proposition.

Proposition 2** (Bounded optimal dual variables).**

For every $y_{t}^{*}\in\arg\max_{y\in{\rm I\!R}^{m}_{+}}\Psi_{t}(y)$ and any $\hat{x}\in C$, we have

[TABLE]

Proposition 2 gives an upper bound on the set of optimal dual variables for each $t$. The bound consists of two parts: a fixed part $\frac{1}{\eta}F_{*}D$ and a variable part $\frac{1}{\eta\rho_{t}}(B_{\psi}(\hat{x},x_{t})-B_{\psi}(\hat{x},x_{t+1}))$ that depends on the previous primal point $x_{t}$ and step size $\rho_{t}$. The bound is illustrated in Fig. 3 schematically. Note that if we upper bound the term $B_{\psi}(\hat{x},x_{t})-B_{\psi}(\hat{x},x_{t+1})$ in Proposition 2 by a constant, then the variable term in the upper bound increases with $t$ since $\rho_{t}\to 0$ as $t\to\infty$ (unless $\epsilon=0$ and so $\rho_{t}=1$). Hence, we cannot claim that each the set of optimal dual variables is uniformly bounded and, therefore, ensure that the sequence of dual variables $\{y_{t}\}$ is attracted to a bounded set—which is key to ensure that constant $E$ exists.

To deal with the issue mentioned above, we adopt another strategy and show that the dual variables remain bounded over a span of iterations. Namely, we do not measure the behavior (or, progress) of the dual variables in one step, but over multiple steps. The intuition behind our strategy is that the variable terms in Proposition 2 “cancel out” when we consider the average set of optimal dual variables. The length of the interval we consider is proportional to the step size and given by

[TABLE]

where $\lceil\cdot\rceil$ is the ceiling function.

We proceed to present a lemma that measures the difference of the dual variables according to the Bregman distance over a span of $\lceil t^{\epsilon}\rceil$ iterations. We present first the following preliminary lemma.

Lemma 5** (Bounded subsequences).**

Let $\rho_{t}=t^{-\epsilon}$ and $S:=\{t,\dots,t+\lceil t^{\epsilon}\rceil\}$, $t\in{\rm I\!N}$. The following bounds hold

[TABLE]

Lemma 6**.**

Consider the setup of Theorem 1. For any $t\in{\rm I\!N}$ and $k\in\{1,\dots,\lceil t^{\epsilon}\rceil\}$, the following bound holds

[TABLE]

where $\chi:=\frac{6G^{2}_{*}}{\sigma_{\varphi}}+3F_{*}D+\frac{L_{\psi}D^{2}}{2}$ is a constant that does not depend on $t$.

There are two important observations from the last lemma.

• •

Observation 1: When $\eta\sum_{t\in S}\|y_{t}\|$ is larger than $\chi$, then the LHS of Eq. (14) is negative. Hence, $y_{t+\lceil t^{\epsilon}\rceil}$ is closer to the origin than $y_{t}$ with respect to the Bregman divergence “metric”. Importantly, $\chi$ does not depend on $t$.

• •

Observation 2: For any $t\in{\rm I\!N}$, the maximum increment of the dual variables in $\lceil t^{\epsilon}\rceil$ iterations is $\chi$ . That is, $B_{\varphi}(0,y_{t+\lceil t^{\epsilon}\rceil})-B_{\varphi}(0,y_{t})\leq\chi$ for any $t\in{\rm I\!N}$ since the second term in the RHS of Eq. (14) is nonnegative.

Using these two observations, we can establish an upper bound on the dual variables for all $t\in{\rm I\!N}$. We have the following lemma.

Lemma 7** (Bounded dual set).**

Consider the setup of Theorem 1 and $\chi$ as defined in Lemma 6. For all $t=1,2,\dots$ we have

[TABLE]

Appendix C Proofs

C.1 Proof of Lemma 1

The update in Eq. (9) is equivalent to $x^{+}=\arg\min_{u\in{\rm I\!R}^{n}}\{\phi(u)+\rho^{-1}B_{\psi}(u,x)+I_{C}(u)\}$ where $I_{C}(u)$ is the indicator function. That is, $I_{C}(u)=0$ if $u\in C$ and $I_{C}(u)=+\infty$ if $u\notin C$. From the optimality condition, we have

[TABLE]

where $\phi^{\prime}(x^{+})\in\partial\phi(x^{+})$, and $s$ is a vector in the normal cone of $C$ at $x^{+}$, i.e., $s\in N_{C}(x^{+}):=\{s\in{\rm I\!R}^{n}\mid\langle s,z-x^{+}\rangle\leq 0,\forall z\in C\}$. Hence, if we multiply the last equation across by $(z-x^{+})$ we obtain

[TABLE]

Next, rearranging terms and using the fact that $\phi(x^{+})-\phi(z)\leq\langle\phi^{\prime}(x^{+}),x^{+}-z\rangle$ (since $\phi$ is convex) yields

[TABLE]

Finally, since $-\nabla B_{\psi}(x^{+},x)=\nabla\psi(x)-\nabla\psi(x^{+})$, we can use the identity in Eq. (8) with $a=x^{+}$, $b=x$, $c=z$ to obtain $-\langle\nabla\psi(x)-\nabla\psi(x^{+}),z-x^{+}\rangle=B_{\psi}(z,x)-B_{\psi}(z,x^{+})-B_{\psi}(x^{+},x)$, which concludes the proof.

C.2 Proof of Lemma 2

From Lemma 1, we have

[TABLE]

Add $\langle f^{\prime}(x),x-x^{+}\rangle$ to both sides of Eq. (15) and use the fact that $-B_{\psi}(x^{+},x)\leq-(\sigma_{\psi}/2)\|x^{+}-x\|^{2}$ (since $\psi$ is strongly convex) to obtain

[TABLE]

where $\Delta:=\langle f^{\prime}(x),x-x^{+}\rangle-{\sigma_{\psi}}({2\rho})^{-1}\|x^{+}-x\|^{2}$. Next, observe that

[TABLE]

(since the convex conjugate of $\lambda\phi(u)$ for $\lambda>0$ is $\lambda\phi^{*}(u^{*}/\lambda)$; see [RW98, pp. 475]). Finally, since $f(x)-f(z)\leq\langle f^{\prime}(x),x-z\rangle$, we obtain the stated result.

C.3 Proof of Lemma 3

Let $\phi(u)=\langle f^{\prime}_{t}(x_{t}),u\rangle+\langle y,g(u)+b_{t+1}\rangle$ in Lemma 2. Note that we use $b_{t+1}$ instead of $b_{t}$ as the perturbation does not affect the primal variables update (see the second paragraph in Sec. 3). We have

[TABLE]

where we have used the fact that $\|f_{t}^{\prime}(x)\|_{*}\leq F_{*}$ by assumption (see Sec. 3.2). Next, let $x^{+}=x_{t+1}$, $x=x_{t}$, $y=y_{t}$, $\rho=\rho_{t}$ in Eq. (17). Summing from $t=1,\dots,T$ and rearranging terms yields

[TABLE]

Now, select $z\in X_{T}$ and observe that

[TABLE]

where the last equation follows since $g(z)+b_{T}\preceq 0$ for any $z\in X_{T}\subseteq X^{\text{max}}_{T}$ (by construction); and $\sum_{t=1}^{T}\langle y_{t},b_{t+1}-b_{T}\rangle\leq 0$ by the choice of $b_{T}$ (see Theorem 1). Hence,

[TABLE]

Finally, we can upper bound the first term in the RHS of the last equation as follows

[TABLE]

where (a) follows from the smoothness of $\psi$, and (b) from the assumption that $\max_{u,v\in C}\|u-v\|\leq D$ (see Sec. 3.2).

C.4 Proof of Lemma 4

Let $\phi(y)=\langle y,-(g(x_{t+1})+b_{t+1})\rangle$ in Lemma 2 and fix $y^{+}=y_{t+1}$ and $y=y_{t}$. Summing from $t=1,\dots,T$ we have

[TABLE]

where we have used the fact that $\|g(x)+b_{t}\|_{*}\leq G_{*}$ for all $x\in C$, $t\in{\rm I\!N}$ by assumption; see Sec. 3.2. Next, let $z=0$ and rearrange terms to obtain

[TABLE]

Dropping the second term in the RHS of the last equation and using the fact that $B_{\varphi}(0,y_{t})\leq 2^{-1}L_{\varphi}\|y_{t}\|^{2}$ (by the smoothness of $\varphi$) and $\|y_{t}\|\leq E$ for all $t\in{\rm I\!N}$ (by assumption), we obtain

[TABLE]

which concludes the proof.

C.5 Proof of Proposition 1

From the optimality condition of the update in Eq. (12) we have

[TABLE]

where $s$ is a vector in the normal cone of ${\rm I\!R}^{m}_{+}$ at $y^{+}$, i.e., $s\in N_{{\rm I\!R}^{m}_{+}}(y^{+})$. Rearranging terms and using the fact that $N_{{\rm I\!R}^{m}_{+}}(y^{+})\subseteq{\rm I\!R}^{m}_{-}$ (i.e., $s\preceq 0$) yields

[TABLE]

Next, let $x^{+}=x_{t+1}$, $y^{+}=y_{t+1}$, $y=y_{t}$, $\rho=\rho_{t}$ and sum the last equation from $t=1,\dots,T-1$ to obtain

[TABLE]

where (a) follows by rearranging terms and (b) by dropping the second and third terms in the RHS of (a) since they are nonpositive (note that $\nabla\varphi(y)\succeq 0$ since $\varphi$ is a strictly increasing function and $\rho_{t+1}\leq\rho_{t}$ for all $t\in{\rm I\!N}$). Furthermore, observe that we can write

[TABLE]

Adding $g(x_{1})+b_{1}$ to both sides

[TABLE]

and therefore

[TABLE]

Finally, if we use the fact that $\|\nabla\varphi(y_{T})\|\leq 2^{-1}L_{\varphi}\|y_{T}\|$ since $\varphi$ is $L_{\varphi}$-smooth (by assumption; see Sec. 3.2) and $\|g(x)+b_{t}\|\leq G$ for all $t\in{\rm I\!N}$, we obtain the stated result.

C.6 Proof of Lemma 5

We start with claim (i). From the integral test, we have $\delta\leq\sum_{t\in S}\rho_{t}\leq 1+\delta$ where

[TABLE]

Hence, we need to upper and lower bound $\delta$. For the upper bound, observe that

[TABLE]

The equation in the RHS is decreasing in $t$ for a fixed $\epsilon$, but also decreasing in $\epsilon$ for a fixed $t$. Thus, the maximum is attained when $t=1$ and $\epsilon=0$. Hence, $\delta\leq 2$.

We continue with the lower bound. Observe that

[TABLE]

For any $t\in{\rm I\!N}$, the minimum is attained when $\epsilon=1$ and equal to

[TABLE]

Hence, $\log(2)\leq\sum_{t\in S}\rho_{t}\leq 3$ as claimed.

We proceed to show claim (ii). Using again the fact that $\sum_{t\in S}\rho^{2}_{t}\leq 1+\delta$, we can write

[TABLE]

Like in the first case, the maximum is attained when $t=1$ and $\epsilon=0$ and therefore $\sum_{t\in S}\rho^{2}_{t}\leq 3$ as claimed.

C.7 Proof of Lemma 6

From Lemma 2 with $\phi(y)=\langle y,-(g(x_{t+1})+b_{t+1})\rangle$ and $z=0$ we have

[TABLE]

Now observe that

[TABLE]

where (a) follows from Lemma 1 for any $z\in C$; (b) since $\langle f_{t}^{\prime}(x_{t}),z-x_{t+1}\rangle\leq\|f^{\prime}_{t}(x_{t})\|_{*}\|z-x_{t+1}\|\leq F_{*}D$ by Hölder’s inequality and by dropping $-\rho^{-1}B_{\psi}(x_{t+1},x_{t})$; and (c) by letting $z=\hat{x}$ (a Slater point that satisfies all the constraints; see Sec. 3.2) and the fact that

[TABLE]

for some $\eta>0$. Hence, by multiplying Eq. (21) across by $\rho$ we have

[TABLE]

Next, let $y^{+}=y_{t+1}$, $y=y_{t}$, $\rho=\rho_{t}$ and sum from $t,\dots,t+k$ with $k\in{\rm I\!N}$

[TABLE]

where in the last equation we have used the fact that $\|g(x)+b_{t}\|_{*}\leq G_{*}$ for all $x\in C$, $t\in{\rm I\!N}$ and that $B_{\psi}(z,x)\leq 2^{-1}L_{\psi}D^{2}$ for all $x,z\in C$. Finally, by using the bounds in Lemma 5, we obtain that

[TABLE]

for any $k\in\{1,\dots,\lceil t^{\epsilon}\rceil\}$.

C.8 Proof of Lemma 7

Define set

[TABLE]

and consider the following two observations from Lemma 6.

• •

Observation (i). For every $y_{t}\in{\rm I\!R}^{m}_{+}$ and $k\in\{1,\dots,\lceil t^{\epsilon}\rceil\}$, we have

[TABLE]

• •

Observation (ii). For every $y_{t}\in{\rm I\!R}^{m}_{+}$, if $y_{t+k}\notin\Omega$ for all $k\in\{1,\dots,\lceil t^{\epsilon}\rceil\}$, then

[TABLE]

where (a) follows since $-\|y\|>-2\chi/\eta$ if $y\notin\Omega$, and (b) because $\sum_{i\in S}\rho_{i}\geq\log(2)>1/2$ by Lemma 5. The last bound also holds if $y_{t+(i-1)\lceil t^{\epsilon}\rceil+k}\notin\Omega$ for all $i=1,\dots,M$ with $M\in\{1,2,\dots,\}$, $k\in\{1,\dots,\lceil t^{\epsilon}\rceil\}$. That is,

[TABLE]

Now, consider the case where $y_{t}\in\Omega$ but $y_{t+1}\notin\Omega$.151515Note that $y_{1}\in\Omega$. Fix $t^{\prime}={t+M\lceil t^{\epsilon}\rceil}+k$ for some $M\in\{0,1,\dots\}$ and $k\in\{1,\dots,\lceil t^{\epsilon}\rceil\}$. Combining the two observations, we can write

[TABLE]

Next, observe that if we use the fact that $\frac{\sigma_{\varphi}}{2}\|y\|^{2}\leq B_{\varphi}(0,y)\leq\frac{L_{\varphi}}{2}\|y\|^{2}$ (by the strong convexity and the smoothness of $\varphi$), we can write

[TABLE]

where (a) follows since $y_{t}\in\Omega$. The last equation concludes the proof since $\max_{y\in\Omega}\|y\|\leq E$.

C.9 Proof of Proposition 2

Let $\phi(z)=\langle f^{\prime}_{t}(x_{t}),z\rangle+\langle y^{*}_{t},g(z)+b_{t+1}\rangle$ in Lemma 1 with $z\in C$ and $y^{*}_{t}\in\arg\max_{y\in{\rm I\!R}^{m}_{+}}\{\min_{u\in C}\mathcal{L}_{t}(u,y)\}$. We have that

[TABLE]

Rearranging terms and dropping $-\rho^{-1}B_{\psi}(x^{+},x_{t})$ yields

[TABLE]

Next, observe that

[TABLE]

where $x^{*}\in\arg\min_{u\in C}\{\langle f^{\prime}_{t}(x_{t}),u\rangle+\langle y^{*}_{t},g(u)+b_{t+1}\rangle\}$ and Eq. (25) follows by complementary slackness [BV04, Sec. 5.5.2], i.e.,

[TABLE]

Eq. (26) follows since $\langle f^{\prime}_{t}(x_{t}),z-x^{*}\rangle\leq\|f^{\prime}_{t}(x_{t})\|_{*}\|z-x^{*}\|\leq F_{*}D$ by Hölder’s inequality and the fact that set $C$ is bounded (see Sec. 3.2). Hence, we can upper bound the first two terms in Eq. (24) and obtain

[TABLE]

Finally, let $z=\hat{x}$ be a vector that satisfies the Slater condition (see Sec. 3.2), and note that

[TABLE]

Using the last bound; diving across by $\eta$; and letting $\rho=\rho_{t}$, we obtain the stated result.

Appendix D Lagrange Duality

This part is not directly related to the online problem in the main part of the paper, however, we think it may be useful as support material for the readers that are not familiar with Lagrange duality. To streamline exposition, we use standard convex optimization notation (e.g., [Roc70, BV04]).

Let $f_{0}:C\to{\rm I\!R}$ be a convex function, $f_{i}:C\to{\rm I\!R}$ a collection of $m$ inequality convex constraints161616An equality constraint can be written with two inequality constraints., and $C\subseteq{\rm I\!R}^{n}$ a convex set. Define

[TABLE]

where $\mathbf{I}$ is the indicator function, i.e., $\mathbf{I}(f_{1}(x),\dots,f_{m}(x))=0$ if $f_{i}(x)\leq 0$ for all $i=1,\dots,m$, and $\mathbf{I}(f_{1}(x),\dots,f_{m}(x))=\infty$ if $f_{i}(x)>0$ for some $i=1,\dots,m$. Clearly, finding the $x\in C$ that minimizes $l$ is equivalent to finding the $x\in C$ that minimizes $f$ such that every constraint $f_{i},i=1,\dots,m$ is satisfied (i.e., the value of every constraint is less than or equal to zero).

Now, consider the perturbed function

[TABLE]

where $u=(u_{1},\dots,u_{m})$ is a vector from ${\rm I\!R}^{m}$. Note that $l(x)=l(x,0)$, and that $l(x,u)$ is convex in $u$ for a fixed $x\in C$. Hence, we can write the convex conjugate [Roc70, Sec. 12] of $l(x,u)$ with respect to the perturbation $u$ for a fixed $x\in C$ as follows

[TABLE]

where the last equation follows since if $f_{i}(x)+u_{i}>0$ for some $i=1,\dots,m$, then $l^{*}(x,u^{*})=-\infty$ — which will never be the case since we can always select $u_{i}\leq-f_{i}(x)$ as $u_{i}\in{\rm I\!R}$. Next, observe that $\sup_{u\in{\rm I\!R}^{m},u_{i}\leq-f_{i}(x),\ \forall i=1,\dots,m}\langle u,u^{*}\rangle$ can be written as the linear program

[TABLE]

It is easy to see that if $u^{*}_{i}<0$ for some $i=1,\dots,m$, then the problem’s solution is unbounded above (i.e., we can always select a $u_{i}$ as negative as we want), and otherwise equal to $-\sum_{i=1}^{m}u^{*}_{i}f_{i}(x)$. Hence,

[TABLE]

Finally, observe that if we let $y=u^{*}$, we obtain

[TABLE]

which is the classic definition of the Lagrangian for $y\succeq 0$. It is well-known that when the Slater condition holds [BV04, Ch. 5], then

[TABLE]

where $f^{\star}=\inf l(x,0)$, i.e., the solution to the “unperturbed” problem. Note as well that $\sup_{y\succeq 0}\mathcal{L}(x,y)$ is indeed like $l(x,0)$; if $x\notin C$ then $\sup_{y\succeq 0}\mathcal{L}(x,y)$ is equal to $+\infty$, and otherwise, when $x\in C$, $\sup_{y\succeq 0}\mathcal{L}(x,y)$ is equal to $f(x)$ .

Intuition behind how Lagrange duality fits into our problem. The perturbed constraints in the main body of the paper can be regarded as if we had the static constraint $f_{i}(x)=g_{i}(x)+\bar{b}_{i}$ where $\bar{b}_{i}=\frac{1}{T}\sum_{t=1}^{T}b_{i}^{(t)}$ is the average of the “perturbations” for any horizon $T\in{\rm I\!N}$. The issue is that the average $\bar{b}_{i}$ is not known a priori and is only revealed as we keep playing actions in each round. Hence, we can not use an approach with “hard” constraints such as minimizing $l(x,0)$. Instead, we relax the constraints and formulate the Lagrange dual problem. Specifically, we let

[TABLE]

and aim to maximize $h$ by carrying out the (sub)gradient ascent update

[TABLE]

where $\alpha^{(t)}>0$ is a step size and $h^{\prime}(y)$ a subgradient of $h$ at $y$. Note that the projection of the dual variables onto the nonnegative orthant is because $h(y)=-\infty$ if $y_{i}<0$ form some $i=1,\dots,m$. The crucial part is that $h^{\prime}(y)$ is given by $g_{i}(x^{*}(y))+\bar{b}_{i}$ where

[TABLE]

That is, $x^{*}(y)$ does not depend on the average $\bar{b}_{i}$.

Now observe that we can write the “noisy” version of update $y^{(t+1)}=[y^{(t)}+\alpha^{(t)}h^{\prime}(y)]^{+}$ as follows

[TABLE]

where $u_{i}=b_{i}-\bar{b}_{i}$ is a “noise” vector. In words, the update can be regarded as a “stochastic” dual subgradient ascent since $\sum_{t=1}^{T}u_{i}^{(t)}=0$ for all $i=1,\dots,m$ for any horizon $T$. However, note that as the average $\bar{b}_{i}$ changes, we are changing the set of feasible solutions (or equivalently, the optimization problem itself). This corresponds to the time-varying feasible set $X_{T}^{\text{max}}$ in the main body of the paper. Recall also that since we do not add any statistical properties to the sequence $\{b_{t}\}$ of perturbations, we are restricted to comparing our solutions with the more restrictive set $X_{T}\subseteq X_{T}^{\text{max}}$.

The difficulties of applying the approach presented in this section to the online setting are that (i) the cost function varies over time and (ii) that this is not known in advance (i.e., it is learnt after the action has been played). To deal with these issues, we replace $f_{t}$ for its gradient (as explain in Sec. 3) and use a primal-dual proximal gradient approach [Van16, Lecture 12] as explained in the Sec. A.