A Convex Optimization Approach to Discrete Optimal Control

TL;DR

This paper introduces a convex optimization framework for discrete optimal control that incorporates myopic and discrete actions using stochastic subgradients, enabling flexible control policies without altering convex updates.

Contribution

It extends convex optimization methods to handle discrete and myopic control actions via stochastic and epsilon-subgradients, allowing flexible policy design.

Findings

Decouples control action choice from subgradient selection.

Provides two classes of discrete control policies, including block-based methods.

Demonstrates the approach's applicability to systems with action order constraints.

Abstract

In this paper, we bring the celebrated max-weight features (myopic and discrete actions) to mainstream convex optimization. Myopic actions are important in control because decisions need to be made in an online manner and without knowledge of future events, and discrete actions because many systems have a finite (so non-convex) number of control decisions. For example, whether to transmit a packet or not in communication networks. Our results show that these two features can be encompassed in the subgradient method for the Lagrange dual problem by the use of stochastic and $\epsilon$-subgradients. One of the appealing features of our approach is that it decouples the choice of a control action from a specific choice of subgradient, which allows us to design control policies without changing the underlying convex updates. Two classes of discrete control policies are presented: one that can make discrete actions by looking only at the system's current state, and another that selects actions using blocks. The latter class is useful for handling systems that have constraints on the order in which actions are selected.

Tables

Table 1. TABLE I: Summary of the convergence of the bound in Lemma 1 depending on the assumptions made on the perturbations ( 𝐄 ​ ( δ k ) = δ 𝐄 subscript 𝛿 𝑘 𝛿 \mathbf{E}(\delta_{k})=\delta for all k 𝑘 k , σ δ 2 superscript subscript 𝜎 𝛿 2 \sigma_{\delta}^{2} is the variance of δ k subscript 𝛿 𝑘 \delta_{k} , and σ ϵ 2 superscript subscript 𝜎 italic-ϵ 2 \sigma_{\epsilon}^{2} the variance of ϵ k subscript italic-ϵ 𝑘 \epsilon_{k} ).

	${\sigma }_{\delta }^2=0$		${\sigma }_{\delta }^2=\infty$
	deterministic	w.h.p.	-
	w.h.p.	w.h.p.	-
${\sigma }_{ϵ}^2=\infty$	$k\to \infty$	$k\to \infty$	-

Equations

$$\displaystyle\begin{array}[]{ll}\underset{x\in X}{\text{minimize}}&f(x)\\ \mbox{\text{subject to}}&g_{j}(x)\leq 0\qquad j=1,\dots,m\end{array}$$

$$h(\lambda )=\underset{x\in X}{\inf }L(x,\lambda )=\underset{x\in X}{\inf }\left\{f(x)+{\lambda }^{T}g(x)\right\},$$

$$\underset{\lambda ⪰0}{\mathrm{maximize}}h(\lambda )$$

$${\lambda }_{k+1}=[{\lambda }_k+\alpha \partial h({\lambda }_k){]}^{+}k=1,2,\dots$$

$$\Vert \partial h(\lambda ){\Vert }_2\le \underset{x\in X}{\max }\Vert g(x){\Vert }_2:={\sigma }_g,$$

$$|h({\lambda }_k)-h({\lambda }^{\star })|\le \Vert {\lambda }_k-{\lambda }^{\star }{\Vert }_2{\sigma }_g$$

$$\displaystyle\begin{array}[]{ll}\underset{x\in X}{\text{minimize}}&f(x)\\ \mbox{\text{subject to}}&g(x)+\delta\preceq 0\end{array}$$

$$\displaystyle\begin{array}[]{ll}\underset{\lambda\succeq 0}{\text{maximize}}&h(\lambda,\delta)\end{array}$$

$${\lambda }_{k+1}$$

$$=[{\lambda }_k+\alpha (g(x_k)+{\delta }_k){]}^{+}$$

$$x_k\in \arg \underset{x\in X}{\min }L(x,{\mu }_k,{\delta }_k),$$

$$\arg \underset{x\in X}{\min }L(x,{\mu }_k,\delta )$$

$$=\arg \underset{x\in X}{\min }\{f(x)+{\mu }_k^{T}g(x)\}$$

$$-\frac{\Vert {\lambda }_1-\theta {\Vert }_2^2}{2\alpha k}-\Gamma \le \frac{1}{k}\sum _{i=1}^{k}h({\lambda }_i,\delta )-h(\theta ,\delta )$$

$${\Gamma }_a$$

$${\Gamma }_c$$

$${\Gamma }_d$$

$${\Gamma }_e$$

$$\frac{\Vert {\lambda }_1-{\lambda }^{\star }(\delta ){\Vert }_2^2}{2\alpha k}$$

$${\lambda }_{k+1}$$

$$\mathbf{E}\left(f({\bar{x}}_k)-f^{\star }(\delta )\right)\le \frac{\alpha \Theta }{2}+\frac{\Vert {\lambda }_1{\Vert }_2^2}{2\alpha k}+\frac{2}{k}\sum _{i=1}^k\Vert {ϵ}_i{\Vert }_2{\sigma }_g$$

$$\mathbf{E}\left(f({\bar{x}}_k)-f^{\star }(\delta )\right)\ge -\frac{\Omega +\Vert \mathbf{E}({\bar{\lambda }}_k){\Vert }_2(\Vert {\lambda }^{\star }(\delta ){\Vert }_2+\sqrt{\Omega })}{\alpha k}$$

$${\Vert \mathbf{E}\left(g({\bar{x}}_k)+\delta \right)\Vert }_2\le \frac{1}{\alpha k}\left(\Vert {\lambda }^{\star }(\delta ){\Vert }_2+\sqrt{\Omega }\right)$$

$${\Vert \mathbf{E}\left({\bar{\lambda }}_k\right)\Vert }_2\le \frac{1}{\upsilon }\left(f(\hat{x})-h({\lambda }^{\star }(\delta ))+\frac{\Omega }{\alpha k}\right)$$

$$\Omega =\Vert {\lambda }_1-{\lambda }^{\star }(\delta ){\Vert }_2^2+{\alpha }^2\Theta k+2\alpha \sum _{i=1}^k\Vert {ϵ}_i{\Vert }_2{\sigma }_g.$$

$${\lambda }_{k+1}$$

$${\mu }_{k+1}$$

$$\Vert {\lambda }_k-{\mu }_k{\Vert }_2\le ϵ:=2\alpha \Vert A{\Vert }_2\psi k=1,2,\dots$$

$$E:=$$

$$U:=$$

$${\Vert \sum _{i=1}^k{x}_i-y_i\Vert }_2={\Vert \sum _{i=1}^{k}W(u_i-e_i)\Vert }_2\le \Vert W{\Vert }_2{\Vert \sum _{i=1}^k{u}_i-e_i\Vert }_2$$

$$\left(\delta +z-z^{\prime }\right)\in D,$$

$$(((\cdots ((z_1-z_1^{\prime })+z_2-z_2^{\prime })+\cdots +z_{K-1}-z_{K-1}^{\prime })$$

$$+z_K-z_K^{\prime }))=\left(\sum _{\tau =1}^K{z}_{\tau }-z_{\tau }^{\prime }\right)\in D,$$

$$e_i\in \arg \underset{e\in E}{\min }\Vert (\delta +z-\sum _{\kappa =1}^{i-1}{e}_{\kappa })-e{\Vert }_{\infty },i=1,\dots ,|Y|$$

$$e_i^{(\tau )}\in \arg \underset{e\in E}{\min }{\Vert \left(z_{\tau }-\sum _{\kappa =1}^{i-1}{e}_{\kappa }^{(\tau )}\right)-e\Vert }_{\infty }$$

Full text

I Introduction

Convexity plays a central role in mathematical optimization from both a theoretical and practical point of view. Some of the advantages of formulating a problem as a convex optimization are that there exist numerical methods that can solve the optimization problem in a reliable and efficient manner, and that a solution is always a global optimum. When an optimization problem is not of the convex type, then one enters into the realm of non-convex optimization where the specific structure of a problem must be exploited to obtain a solution, often not necessarily the optimal one.

In some special cases there exist algorithms that can find optimal solutions to non-convex problems. One is the case of max-weight scheduling: an algorithm initially devised for scheduling packets in queueing networks which has received much attention in the networking and control communities in recent years. In short, max-weight was proposed by Tassiulas and Ephremides in their seminal paper [1]. It considers a network of interconnected queues in a slotted time system where packets arrive in each time slot and a network controller has to make a discrete (so non-convex) scheduling decision as to which packets to serve from each of the queues. Appealing features of max-weight are that the action set matches the actual decision variables (namely, do we transmit or not); that scheduling decisions are made without previous knowledge of the mean packet arrival rate in the system (myopic actions); and that it can stabilize the system (maximize the throughput) whenever there exists a policy or algorithm that would do so. These features have made max-weight well-liked in the community and have fostered the design of extensions that consider (convex) utility functions, systems with time-varying channel capacity and connectivity, heavy-tailed traffic, etc. Similarly, max-weight has been brought to other areas beyond communication networks including traffic signal control [2], cloud computing [3], economics [4], etc., and has become a key tool for discrete decision making in queueing systems.

However, there are some downsides. The success of max-weight has produced so many variants of the algorithm that the state of the art is becoming not only increasingly sophisticated but also increasingly complex. Furthermore, it is often not clear how to combine the different variants (e.g., utility function minimization with heavy-tailed traffic) since the design of a new control or scheduling policy usually involves employing a new proof or exploiting the special structure of a problem. There is a need for abstraction and to put concepts into a unified theoretical framework. While this has been attempted in previous works by means of establishing a connection between max-weight and dual subgradient methods in convex optimization, most of the works [5, 6, 7, 8, 9] have focused on specific congestion control applications and obtained discrete control actions as a result of exploiting the special structure of the problem. In particular, discrete actions are possible because the primal problem allows decomposition and a partial subgradient (or schedule) can be obtained as a result of minimizing a linear program.111A solution of a linear program always lies in an extreme point of a polytope, and that point is matched with a discrete (scheduling) decision. Further, the aforementioned works are deeply rooted in Lyapunov or fluid limit techniques and convergence/stability is only guaranteed asymptotically. That is, they do not make quantitative statements about the system state in finite time; for instance, provide upper and lower bounds on the optimality of the objective function, or a bound on the amount of constraint violation. Because of all the above, the body of work on max-weight approaches is still largely separate from the mainstream literature on convex optimization.

In this paper, we abstract the celebrated max-weight features (myopic and discrete actions) and make them available in standard convex optimization. In particular, our approach consists of formulating the Lagrange dual problem and equipping the subgradient method with a perturbation scheme that can be regarded as using stochastic and $\epsilon$-subgradients. Stochastic subgradients are useful to capture the randomness in the system, while $\epsilon$-subgradients allow us to decouple the choice of subgradient from the selection of a control action, i.e., they provide us with flexibility as to how to select control actions. The latter is important because (i) we can design a new control policy without having to prove again the convergence of the whole algorithm, and (ii) it eases the design of policies that model the characteristics of more complex systems. For example, policies that select actions from a finite set or that have constraints associated with selecting certain subsets of actions. The finiteness of the action set is of particular significance from a theoretical point of view because we are allowing convex optimization to make non-convex updates, and from a practical point of view because many systems, such as computers, make decisions in a discrete-like manner.

The main contributions of the paper are summarized in the following:

• (i)

Unifying Framework: Our analysis brings the celebrated max-weight features to mainstream convex optimization. In particular, they can be encompassed in the subgradient method for the Lagrange dual problem by using $\delta_{k}$ and $\epsilon_{k}$ perturbations. The analysis is presented in a general form and provides different types of convergence depending on the statistical properties of the perturbations, including bounds that are not asymptotic.

• (ii)

General Control Policies: Our analysis clearly separates the selection of a subgradient from a particular choice of control action and establishes the fundamental properties that a control policy should satisfy for it to be optimal.

• (iii)

Discrete Control Policies: We develop two classes of control policies that allow us to use action sets with a finite number of actions (i.e., the action sets are not convex). One that is able to make discrete actions by looking only at the system’s current state, and another that selects actions using blocks. The latter class is useful for handling systems that have constraints on how actions can be selected.

The rest of the paper is organized as follows. We start with the preliminaries, which cover the notation and some background material. In Section III, we study the convergence of the dual subgradient method under a ($\delta_{k},\epsilon_{k}$) perturbation scheme, and in Section IV how the $\epsilon_{k}$ perturbations can be used to equip the dual subgradient method with discrete actions. Section V provides some remarks and discussion, and Section VI illustrates the results with an example that considers discrete scheduling decisions with constraints. Finally, in Section VII we provide an overview of the state of the art, and compare it with our work. All the proofs are in the appendices.

II Preliminaries

We start by introducing the notation, the standard convex optimization problem setup, and the subgradient method for the Lagrange dual problem.

II-A Notation

The sets of natural, integers and real numbers are denoted by $\mathbf{N}$, $\mathbf{Z}$ and $\mathbf{R}$. We use $\mathbf{R}_{+}$ and $\mathbf{R}^{n}$ to denote the set of nonnegative real numbers and $n$-dimensional real vectors. Similarly, we use $\mathbf{R}^{m\times n}$ to denote the set of $m\times n$ real matrices. Vectors and matrices are usually written, respectively, in lower and upper case, and all vectors are in column form. The transpose of a vector $x\in\mathbf{R}^{n}$ is indicated with $x^{T}$, and we use $\boldsymbol{1}$ to indicate the all ones vector. The Euclidean, $\ell_{1}$ and $\ell_{\infty}$ norms of a vector $x\in\mathbf{R}^{n}$ are indicated, respectively, with $\|x\|_{2}$, $\|x\|_{1}$ and $\|x\|_{\infty}$.

Since we usually work with sequences we will use a subscript to indicate an element in a sequence, and parenthesis to indicate an element in a vector. For example, for a sequence $\{x_{k}\}$ of vectors from $\mathbf{R}^{n}$ we have that $x_{k}=[x_{k}(1),\dots,x_{k}(n)]^{T}$ where $x_{k}(j),\ j=1,\dots,n$ is the $j$’th component of the $k$’th vector in the sequence. For two vectors $x,y\in\mathbf{R}^{n}$ we write $x\succ y$ when $x{(j)}>y{(j)}$ for all $j=1,\dots,n$, and $x\succeq y$ when $x(j)\geq y(j)$. We use $[\cdot]^{+}$ to denote the projection of a vector $x\in\mathbf{R}^{n}$ onto the nonnegative orthant, i.e., $[x]^{+}=[\max\{x(1),0\},\dots,\max\{x(n),0\}]^{T}$.

II-B Convex Optimization Problem

Consider the standard constrained convex optimization problem $\mathcal{P}$

[TABLE]

where $f,g_{j}:X\to{\mathbf{R}}$ are convex functions and $X$ is a convex subset from $\mathbf{R}^{n}$. We will assume that set $X_{0}:=\{x\in X\mid g_{j}(x)\leq 0,\ j=1,\dots,m\}\neq\emptyset$, and so problem $\mathcal{P}$ is feasible. Also, and using standard notation, we define $f^{\star}:=\min_{x\in X_{0}}f(x)$ and $x^{\star}\in\arg\min_{x\in X_{0}}f(x)$.

We can transform problem $\mathcal{P}$ into an unconstrained convex optimization by applying a (Lagrange) relaxation on the constraints. The Lagrange dual function associated to problem $\mathcal{P}$ is given by

[TABLE]

where $g(x)=[g_{1}(x),\dots,g_{m}(x)]^{T}$, and $\lambda\in\mathbf{R}^{m}_{+}$ is a vector of Lagrange multipliers. Function $h$ is concave [10, Chapter 5] and so we can cast the following unconstrained222The Lagrange dual function is equal to $-\infty$ when $\lambda\prec 0$. concave maximization problem $\mathcal{D}$

[TABLE]

where $h(\lambda^{\star})=f^{\star}$ with $\lambda^{\star}\in\arg\max_{\lambda\succeq 0}h(\lambda)$ when strong duality holds. That is, solving problem $\mathcal{P}$ is equivalent to solving problem $\mathcal{D}$. A sufficient condition for strong duality to hold is the following.

Assumption 1** (Slater condition).**

$X_{0}$* is non-empty. There exists a vector $x\in X$ such that $g_{j}(x)<0$ for all $j=1,\dots,m$.*

II-C Classic Subgradient Method

Problem $\mathcal{D}$ can be solved using the subgradient method. One of the motivations for using this iterative method is that it allows the Lagrange dual function to be nondifferentiable, and so it imposes few requirements on the objective function and constraints. Another motivation for using the subgradient method is that when the Lagrangian has favorable structure, then the algorithm can be implemented in a distributed manner, and therefore can be used to solve large-scale problems. Nonetheless, in this work, the primary motivation for using the subgradient method in the Lagrange dual problem is that it allows us to handle perturbations on the constraints. As we will show in Section III, this will be key to handling resource allocation problems where the resources that need to be allocated are not known in advance.

II-C1 Iteration

In short, the subgradient method for the Lagrange dual problem consists of the following update:

[TABLE]

where $\lambda_{1}\in\mathbf{R}^{m}_{+}$, $\partial h(\lambda_{k})$ is a subgradient in the subdifferential of $h$ at point $\lambda_{k}$ and $\alpha>0$ a constant step size. The classic subgradient method can make use of more complex step sizes, but constant step size will play an important role in our analysis, and it is extensively used in practical applications.

II-C2 Computing a Subgradient

A dual subgradient can be obtained by first minimizing $L(\cdot,\lambda_{k})$ and then evaluating an $x_{k}\in\arg\min_{x\in X}L(x,\lambda_{k})$ on the constraints, i.e., $\partial h(\lambda_{k})=g(x_{k})$. Note that minimizing $L(\cdot,\lambda_{k})$ for a fixed $\lambda_{k}\in\mathbf{R}^{m}_{+}$ is an unconstrained convex optimization that can be carried out with a variety of methods, and the choice of using one method or another will depend on the assumptions made on the objective function and constraints. Sometimes it is not possible to exactly minimize the Lagrangian and an approximation is obtained instead, i.e., an $x_{k}\in X$ such that $L(x_{k},\lambda_{k})-h(\lambda_{k})\leq\xi$ where $\xi\geq 0$. This can be equivalently regarded as exactly minimizing the Lagrangian when an approximate Lagrange multiplier is used instead of the true Lagrange multiplier (see Appendix A for a detailed explanation). That is, we obtain an $x_{k}\in\arg\min_{x\in X}L(x,\mu_{k})$ where $\mu_{k}=\lambda_{k}+\epsilon_{k}$ and $\epsilon_{k}\in\mathbf{R}^{m}$ such that $\mu_{k}\succeq 0$ ($\epsilon_{k}$ can be regarded as a perturbation or error in the Lagrange multiplier).

II-C3 Convergence

A standard assumption made to prove the convergence of the subgradient method is that the subdifferential of $h$ is bounded for all $\lambda\succeq 0$. We can ensure this by making the following assumption.

Assumption 2**.**

$X$* is bounded.*

Observe that since we always have that $\partial h(\lambda)=g(x)$ for some $x\in X$, if $g(x)$ is bounded for every $x\in X$ then the subgradients of the Lagrange dual function are also bounded. That is, we have that

[TABLE]

and $\sigma_{g}$ is finite because $g$ is a closed convex function (and so continuous) and $X$ is bounded.

The basic idea behind the convergence of the dual subgradient method with constant step size is that

1. (i)

the Euclidean distance between $\lambda_{k}$ and a vector $\lambda^{\star}\in\Lambda^{\star}:=\arg\max_{\lambda\succeq 0}h(\lambda)$ decreases monotonically when $\lambda_{k}$ is sufficiently “far away” from $\Lambda^{\star}$;333Under the Slater condition $\Lambda^{\star}$ is a bounded subset from $\mathbf{R}^{m}_{+}$ (Lemma 1 in [11]). 2. (ii)

when $\lambda_{k}$ is sufficiently close to $\Lambda^{\star}$, it remains in a ball around it.

Important characteristics of the dual subgradient method are that the size of the ball to which $\lambda_{k}$ converges depends on $\alpha$; that $\lambda_{k}$ converges to an $\alpha$-ball around $\Lambda^{\star}$ in finite time; and that by selecting $\alpha$ sufficiently small we can make the $\alpha$-ball arbitrarily small. It is important to highlight as well that the monotonic convergence of $\lambda_{k}$ to a ball around $\Lambda^{\star}$ does not imply that the value of the Lagrange dual function improves in each iteration.444By monotonic convergence we mean the Euclidean distance between $\lambda_{k}$ and a point in $\Lambda^{\star}$ decreases. Yet, since from Assumption 2 we have that the Lagrange dual function is Lipschitz continuous555For any $\lambda_{1},\lambda_{2}\succeq 0$ we have that $|h(\lambda_{1})-h(\lambda_{2})|\leq\|\lambda_{1}-\lambda_{2}\|_{2}\sigma_{g}$. for $\lambda\succeq 0$, if the RHS in

[TABLE]

decreases, then $h(\lambda_{k})$ will eventually approach $h(\lambda^{\star})$.

III Subgradient Method with Perturbations

In this section, we introduce the framework that will allow us to tackle optimization problems with discrete control actions. We begin by considering the following convex optimization problem $\mathcal{P}(\delta)$

[TABLE]

where $\delta\in\mathbf{R}^{m}$. If perturbation $\delta$ were known we could use an interior point method or similar to solve the problem. However, we will assume that $\delta$ is not known in advance and tackle the problem using a Lagrange relaxation on the constraints. The interpretation of perturbation $\delta$ will depend on the details of the problem being considered, for example, in a packet switched network $\delta$ may be the (unknown) mean packet arrival rate.

Because the solution of problem $\mathcal{P}(\delta)$ depends on $\delta$, it will be convenient to define $X_{0}(\delta):=\{x\in X\mid g(x)+\delta\preceq 0\}$, $f^{\star}(\delta):=\min_{x\in X_{0}(\delta)}f(x)$, and $x^{\star}(\delta)$ to be a solution of problem $\mathcal{P}(\delta)$. Similarly, we also parameterize the Lagrangian $L(x,\lambda,\delta)=f(x)+\lambda^{T}(g(x)+\delta)$, the Lagrange dual function $h(\lambda,\delta):=\inf_{x\in X}L(x,\lambda,\delta)$, and define the Lagrange dual problem $\mathcal{D}(\delta)$

[TABLE]

where $\lambda^{\star}(\delta)$ is a vector in the set of dual optima $\Lambda^{\star}(\delta):=\arg\max_{\lambda\succeq 0}h(\lambda,\delta)$.

III-A Subgradient Method with Perturbations

The general version of the subgradient method we consider is the following

[TABLE]

for $k=1,2,\dots$ with $\lambda_{1}\in\mathbf{R}^{m}_{+}$ and where

[TABLE]

$\delta_{k}\in\mathbf{R}^{m}$ and $\mu_{k}=\lambda_{k}+\epsilon_{k}$ with $\epsilon_{k}\in\mathbf{R}^{m}$. We will refer to $\delta_{k}$ and $\epsilon_{k}$ as perturbations. Since parameter $\delta$ is not known in the optimization we have replaced it with a surrogate $\delta_{k}$, which can be regarded as an approximation or perturbed version of parameter $\delta$. Later, we will add assumptions on the properties that $\delta_{k}$ must have in order that update (1) solves problem $\mathcal{P}(\delta)$. An important observation is that for any $\mu_{k}\in\mathbf{R}^{m}_{+}$

[TABLE]

and so $g(x_{k})$ or the “partial” subgradient of $h(\lambda_{k},\delta)$ can be obtained independently of perturbation $\delta$.

We are now in the position to present the following lemma.

Lemma 1** (Dual Subgradient Method).**

Consider the optimization problem $\mathcal{P}(\delta)$ and update (1) with $\mu_{k}\succeq 0$ for all $k$. Suppose $\{\delta_{k}\}$ is a sequence of points from $\mathbf{R}^{m}$ such that $\lim_{k\to\infty}k^{-1}\sum_{i=1}^{k}\delta_{i}=\delta$. Then,

[TABLE]

where $\theta$ is any vector from $\mathbf{R}^{m}_{+}$, $\Gamma:=\alpha(\Gamma_{a}+\Gamma_{b}+\Gamma_{c})+\Gamma_{d}+\Gamma_{e}$ and

[TABLE]

Proof:

See Appendix B. ∎

Lemma 1 is expressed in a general form and establishes a lower bound on $k^{-1}\sum_{i=1}^{k}h(\lambda_{i},\delta)-h(\theta,\delta)$, where $\theta$ is any vector from $\mathbf{R}^{m}_{+}$. When $\theta=\lambda^{\star}(\delta)$ we can upper bound (2) by zero since $h(\lambda,\delta)\leq h(\lambda^{\star}(\delta),\delta)$ for all $\lambda\in\mathbf{R}^{m}_{+}$. The bound on the left-hand side of (2) depends on the properties of perturbations $\delta_{k}$ and $\epsilon_{k}$. Next, we provide a detailed analysis of the lower bound and how the different assumptions on the properties of the perturbations result in different types of convergence. A summary of the results can be found in Section III-A2.

III-A1 Analysis of Perturbations

Firstly, by Assumption 1 and Lemma 1 in [11] we have that $\lambda^{\star}(\delta)$ is a bounded vector from $\mathbf{R}^{m}_{+}$ and therefore the first term on the LHS of (2)

[TABLE]

is bounded and goes to zero as $k\to\infty$. Since (3) is divided by $\alpha$, the convergence rate is inversely proportional to the step size used.

Turning now to $\alpha(\Gamma_{a}+\Gamma_{b}+\Gamma_{c})$, when $\Gamma_{a}$, $\Gamma_{b}$ and $\Gamma_{c}$ are bounded above then this expression can be made arbitrarily small by selecting $\alpha$ sufficiently small. Term $\Gamma_{a}$ is the sum of the dual subgradients. If we assume that $X$ is bounded (c.f., Assumption 2) we have that $\|g(x_{i})+\delta\|_{2}$ is upper bounded by some constant $\sigma_{g}$,666Note that $g(x_{i})+\delta$ is a subgradient of $h(\lambda_{k},\delta)$. and so $\Gamma_{a}$ is bounded by $\sigma_{g}^{2}/2$ for all $i=1,\dots,k$. The bounds on terms $\Gamma_{b}$ and $\Gamma_{c}$ depend on the characteristics of sequence $\{\delta_{k}\}$. We consider two cases. Case (i) $\delta_{k}$, $k=1,2,\dots$ are uniformly bounded. Then, $\Gamma_{b}$ is trivially uniformly upper bounded for all $k$; and since $(\delta_{k}-\delta)^{T}(g(x_{k})+\delta)\leq\|\delta_{k}-\delta\|_{2}\|g(x_{k})+\delta\|_{2}\leq\|\delta_{k}-\delta\|_{2}\sigma_{g}$ by Cauchy-Schwarz, we have that $\Gamma_{c}$ is also uniformly upper bounded. Case (ii) $\delta_{k}$, $k=1,2,\dots$ is a realization of independent random variables with finite variance and kurtosis, but they do not necessarily have to have bounded support. In this case, we can upper bound $\Gamma_{b}$ and $\Gamma_{c}$ with probability one asymptotically as $k\to\infty$ using Hoeffding’s inequality [12]. Hoeffding’s bound can be applied to $\Gamma_{b}$ directly, and for $\Gamma_{c}$ it is sufficient to note that $-k^{-1}\sum_{i=1}^{k}\sum_{j=1}^{m}(\delta_{i}(j)-\delta(j))(g_{l}(x_{k})+\delta(j))\geq-\sum_{j=1}^{m}|k^{-1}\sum_{i=1}^{k}(\delta_{i}(j)-\delta(j))|\sigma_{g}$ where $\delta(j)$ is the $j$’th component of vector $\delta\in\mathbf{R}^{m}$, and $l\in\{1,\dots,m\}$.

Finally, we have the terms $\Gamma_{d}$ and $\Gamma_{e}$ which are not scaled by $\alpha$ in the bound. Since $\Gamma_{d}$ depends on sequence $\epsilon_{k}$, the boundedness of the term will depend on the assumptions we make on this perturbation. We consider three cases. Case (i) $\|\epsilon_{k}\|_{2}\leq\epsilon$ for all $k$ for some $\epsilon>0$. In this case we have that $\Gamma_{d}$ can be uniformly upper bounded by $2\epsilon\sigma_{g}$. Case (ii) $\lim_{k\to\infty}k^{-1}\sum_{i=1}^{k}\|\epsilon_{i}\|_{2}=\epsilon$. We cannot say anything about $\Gamma_{d}$ for finite $k$, but we will have that $\Gamma_{d}$ is upper bounded by $2\epsilon\sigma_{g}$ when $k\to\infty$. An interesting observation is that if $\{\epsilon_{k}\}$ were a stochastic process, it would not need to have finite variance in order for $\lim_{k\to\infty}k^{-1}\sum_{i=1}^{k}\|\epsilon_{i}\|_{2}$ to exist and be finite (note this is in marked contrast to perturbation $\delta_{k}$, which always has to have finite variance). Case (iii) $\{\|\epsilon_{k}\|_{2}\}$ is a realization of independent random variables with finite variance and mean $\epsilon$. In this case we can use Hoeffding’s inequality to give a bound on $\Gamma_{d}$ with probability one asymptotically as $k\to\infty$.

Term $\Gamma_{e}$ is perhaps the term for which the analysis is more delicate. In the deterministic subgradient method we have that $\delta_{k}=\delta$ for all $k$ and so the term is equal to zero for all $k$. Observe that when $\Gamma_{e}$ is nonnegative, then we can ignore the term since this would still leave a lower bound on the LHS of (2). However, since $\lambda^{\star}(\delta)$ is not known (we only know it is finite), it is not possible to determine the sign of $\Gamma_{e}$, and so the term could be unbounded below when $k\to\infty$. Because of all this, we will usually require that $\{\delta_{k}\}$ is an ergodic process (i.e., $\mathbf{E}(\delta_{k})=\delta$ for all $k$) and make use of the fact that $\lambda_{k}$ and $\delta_{k}$ are independent for all $k$; in which case $\mathbf{E}(k^{-1}\sum_{i=1}^{k}(\lambda_{i}-\lambda^{\star}(\delta))^{T}(\delta_{i}-\delta))=k^{-1}\sum_{i=1}^{k}\mathbf{E}(\lambda_{i}-\lambda^{\star}(\delta))^{T}\mathbf{E}(\delta_{i}-\delta)=0$ and the expected value of the lower bound in Lemma 1 does not depend on term $\Gamma_{e}$.

III-A2 Summary of the Different Types of Convergence

Table I provides a summary of the convergence properties of the lower bound obtained in Lemma 1 under the assumption that $\{\delta_{k}\}$ is an ergodic stochastic process777Hence, $\mathbf{E}(\delta_{k})=\delta$ for all $k$ where $\delta$ is the perturbation on the constraints given in problem $\mathcal{P}(\delta)$. and expectation is taken with respect to random variable $\delta_{k}$ (since we need term $\Gamma_{e}$ to vanish). By deterministic convergence we mean that it is possible to obtain a lower bound of $k^{-1}\sum_{i=1}^{k}h(\lambda_{i},\delta)-h(\lambda^{\star}(\delta),\delta)$ for every $k=1,2,\dots$; by w.h.p. that a lower bound can be given with high probability for $k$ large enough; and by $k\to\infty$ that the lower bound will only hold asymptotically, i.e., it is not possible to say anything about the bound for finite $k$.

III-B Recovery of Primal Solutions

We are now in the position to present one of our main theorems, which establishes the convergence of the objective function to a ball around the optimum, and provides bounds on the amount of constraint violation.

Theorem 1** (Convergence).**

Consider problem $\mathcal{P}(\delta)$ and update

[TABLE]

where $\mu_{k}=\lambda_{k}+\epsilon_{k}$ with $\lambda_{1}\in\mathbf{R}^{m}_{+}$ and $\{\epsilon_{k}\}$ a sequence of points from $\mathbf{R}^{m}$ such that $\mu_{k}\succeq 0$ for all $k$. Suppose $X_{0}(\delta)$ has nonempty relative interior (the Slater condition is satisfied) and that $\delta_{k}$ is an ergodic stochastic process with expected value $\delta$ and $\mathbf{E}(\|\delta_{k}-\delta\|_{2}^{2})=\sigma_{\delta}^{2}$ for some finite $\sigma_{\delta}^{2}$. Further, suppose that $\lim_{k\to\infty}k^{-1}\sum_{i=1}^{k}\|\epsilon_{i}\|_{2}=\epsilon$ for some $\epsilon\geq 0$ and that Assumption 2 holds. Then,

[TABLE]

where $\bar{x}_{k}=k^{-1}\sum_{i=1}^{k}x_{i}$, $\bar{\lambda}_{k}:=k^{-1}\sum_{i=1}^{k}\lambda_{i}$, $\Theta:=\sigma^{2}_{g}+\sigma_{\delta}^{2}$, $\hat{x}$ is a Slater point (i.e., $g(\hat{x})\prec 0$), $\upsilon:=\min_{j\in\{1,\dots,m\}}-g_{j}(\hat{x})$, and

[TABLE]

Proof:

See Appendix B. ∎

Claims (i) and (ii) in Theorem 1 establish that $\mathbf{E}(f(\bar{x}_{k}))$ converges to a ball around $f^{\star}(\delta)$, where the type of convergence will depend on the assumptions made on the perturbations, as indicated in Table I. Also, note that choosing $\lambda_{1}=0$ is always a good choice to obtain a sharper upper bound. Claim (iii) provides a bound on the expected value of the constraint violation, and claim (iv) says that the expected value of the running average of the Lagrange multipliers is bounded for all $k$. As we will show in Section V, this will play an important role in establishing the stability of a queueing system. Finally, note that the bounds in claims (ii)-(iv) depend on $\lambda^{\star}(\delta)$, which is usually not known in the optimization. Fortunately, by Lemma 1 in [11], one can obtain an upper bound on $\lambda^{\star}(\delta)$, which is enough. Also, observe that we can use the fact that $-h(\lambda^{\star}(\delta))\leq-h(\lambda)$ for all $\lambda\succeq 0$ and obtain a looser bound in claim (iv).

Theorem 1 establishes the convergence properties of the dual subgradient update (4) under $\delta_{k}$ and $\epsilon_{k}$ perturbations without connecting with any specific application. However, by appropriate definition of these perturbations a wide range of situations can be encompassed. For example, we can use perturbations $\delta_{k}$ to relax the perfect knowledge of the constraints, and perturbations $\epsilon_{k}$ to capture asynchronism in the primal updates (see [13]). However, of particular interest here is that we can use the $\epsilon_{k}$ perturbations in this framework to analyze the use of discrete-valued control actions for solving optimization problem $\mathcal{P}(\delta)$. We consider this in detail in the next section.

IV Discrete Control Actions

In this section, we present the second main contribution of the paper: how to use $\epsilon_{k}$ perturbations to equip the dual subgradient method with discrete control actions. Discrete actions or decisions are crucial in control because many systems are restricted to a finite number of states or choices. For instance, a traffic controller has to decide whether a traffic light should be red or green, or a video application which streaming quality to use e.g., $\{360p,480p,720p,1080p\}$.

We present two classes of discrete control policies. One that selects discrete control actions in an online or myopic manner based only on the current state of the system, and another batch approach that chooses discrete control actions in blocks or groups. The latter class is particularly useful for problems where there are constraints or penalties associated with selecting subsets of actions. For example, in video streaming where the application wants to maximize the quality of the video delivered but at the same time minimize the variability of the quality, and so has constraints on how often it can change the quality of the video stream.

IV-A Problem Formulation

We start by introducing the following definitions:

Definition 1** (Finite Action Set).**

$Y$* is a finite collection of points from $\mathbf{R}^{n}$.*

Definition 2** (Convex Action Set).**

$X\subseteq\mathrm{conv}{(Y)}$* and convex.*

We will regard selecting a point $y$ from finite set $Y$ as taking a discrete control action. The physical action associated with each point in set $Y$ will depend on the context in which the optimization is applied, e.g., the points in $Y$ may correspond to the actions of setting a traffic light to be red and green. Figure 1 shows an example of two sets $Y$ and respective convex sets $X\subseteq\mathrm{conv}{(Y)}$.

We consider problem $\mathcal{P}(\delta)$ from Section III, but now require that the inequality constraints are linear, i.e., $g(x):=Ax$ with $A\in\mathbf{R}^{m\times n}$. The reason for this is the following lemma, which is a restatement of [14, Proposition 3.1.2].

Lemma 2** (Queue Continuity).**

Consider updates

[TABLE]

where $\lambda_{1}=\mu_{1}\geq 0$, $\alpha>0$, $A\in\mathbf{R}^{m\times n}$, $\delta_{k}\in\mathbf{R}^{m}$, and $\{x_{k}\}$ and $\{y_{k}\}$ are, respectively, sequences of points from $X$ and $Y$. Suppose $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}\leq\psi$ for all $k$ and some $\psi\geq 0$. Then,

[TABLE]

Lemma 2 says that when the difference $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}$ is uniformly bounded by some constant $\psi$ then $\mu_{k}$ is an approximate Lagrange multiplier. And by selecting $x_{k}$ according to (1) we can immediately apply Theorem 1 to conclude that sequence $\{y_{k}\}$ of discrete actions (approximately888Note that if $\psi$ is finite we can make $\epsilon$ (and so $\mathbf{E}(f(\bar{x}_{k})-f^{\star}(\delta))$ in Theorem 1) arbitrarily small by selecting step size $\alpha$ sufficiently small. ) solves problem $\mathcal{P}(\delta)$. This is a key observation. Not only does it (i) establish that we can solve $\mathcal{P}(\delta)$ using only discrete actions and (ii) give us a testable condition, $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}\leq\psi$, that the discrete actions must satisfy, but it also (iii) tells us that any sequence $\{y_{k}\}$ of discrete actions satisfying this condition solves $\mathcal{P}(\delta)$. We therefore have considerable flexibility in how we select actions $y_{k}$. In other words, we have the freedom to select from a range of different optimal control policies without changing the underlying convex updates. One way to use this freedom is to select an optimal control policy that satisfies specified constraints, e.g., that does not switch traffic lights between green and red too frequently or which minimizes the use of “costly” actions. Such constraints are often practically important yet are difficult to accommodate within classical control design frameworks.

With this in mind, in the rest of this section we consider methods for constructing sequences $\{y_{k}\}$ of discrete actions that stay close to a sequence $\{x_{k}\}$ of continuous-valued updates in the sense that $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}$ is uniformly bounded. We begin by establishing that for any sequence $\{x_{k}\in X\subseteq\mathrm{conv}(Y)\}$ there always exists discrete-valued sequences $\{y_{k}\in Y\}$ such that $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}$ is uniformly bounded.

IV-B Existence of Discrete Sequences

It will prove convenient to exploit the fact that each point $x\in X$ can be written as a convex combination of points from $Y$. Collect the points in $Y$ as columns in matrix $W$ and define

[TABLE]

where $v_{j}$ is an $|Y|$-dimensional standard basis vector, i.e., all elements of vector $v_{j}$ are equal to [math] except the $j$’th element that is equal to $1$, and $U$ is the $|Y|$-dimensional simplex. Since we can always write a vector $x_{i}\in X$ as the convex combination of points from $Y$ there exists at least one vector $u_{i}\in U$ such that $x_{i}=Wu_{i}$.999A vector $u_{i}$ can be obtained, for example, by solving the optimization problem $\min_{u\in U}\|x_{i}-Wu\|_{2}^{2}$. The non-uniqueness of the solution comes from Carathéodory’s theorem—see, for example, [15]. Similarly, there exists a vector $e_{i}\in E$ such that $y_{i}=We_{i}$. Hence,

[TABLE]

and therefore showing that $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}$ is uniformly bounded is sufficient to establish the boundedness of $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}$.

We have the following useful lemma.

Lemma 3**.**

Let $E$ be a set containing the $|Y|$-dimensional standard basis vectors, $U:=\mathrm{conv}{(E)}$, and $D:=\{\delta\in\mathbf{R}^{|Y|}\mid\delta^{T}\boldsymbol{1}=0,\|\delta\|_{\infty}\leq 1\}$. For any vector $\delta\in D$, and sequence $\{u_{i}\}_{i=1}^{|Y|}$ of points from $U$, there exists at least one sequence $\{e_{i}\}_{i=1}^{|Y|}$ of points from $E$ such that

[TABLE]

where $z:=\sum_{i=1}^{|Y|}u_{i}$ and $z^{\prime}:=\sum_{i=1}^{|Y|}e_{i}$. That is, $\boldsymbol{1}^{T}(\delta+z-z^{\prime})=0$ and $\|\delta+z-z^{\prime}\|_{\infty}\leq 1$.

Proof:

See Appendix C. ∎

Let $\{u^{(1)}_{i}\}_{i=1}^{|Y|}$ be a sequence of points from $U$, and $\{e^{(1)}_{i}\}_{i=1}^{|Y|}$ a sequence of points from $E$ such that $(z_{1}-z^{\prime}_{1})\in D$ where $z_{1}=\sum_{i=1}^{|Y|}u^{(1)}_{i}$ and $z^{\prime}_{1}=\sum_{i=1}^{|Y|}e^{(1)}_{i}$. By Lemma 3 such a sequence $\{e^{(1)}_{i}\}_{i=1}^{|Y|}$ always exists. Similarly, for another sequence $\{u^{(2)}_{i}\}_{i=1}^{|Y|}$ of points from $U$, we can construct a sequence $\{e^{(2)}_{i}\}_{i=1}^{|Y|}$ of points from $E$ such that $(z_{2}-z_{2}^{\prime}+(z_{1}-z^{\prime}_{1}))\in D$ where $z_{2}$ and $z^{\prime}_{2}$ are, respectively, the sum of the elements in sequences $\{u^{(2)}_{i}\}_{i=1}^{|Y|}$ and $\{e^{(2)}_{i}\}_{i=1}^{|Y|}$. Repeating, it follows that for sequences $\{u^{(\tau)}\}_{i=1}^{|Y|}$, $\tau\in\{1,2,\dots,K\}$ we can construct sequences $\{e^{(\tau)}\}_{i=1}^{|Y|}$, $\tau\in\{1,2,\dots,K\}$ such that

[TABLE]

where $z_{\tau}$ and $z^{\prime}_{\tau}$ are the sum of the elements in the respective sequences. It follows that $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}\leq\sqrt{|Y|}$ for $k\in\tau|Y|$, $\tau\in\mathbf{Z}_{+}$ and $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}\leq\sqrt{|Y|(1+2|Y|)^{2}}$ for all $k=1,2,\dots$, since the sequences can diverge element-wise by at most $2|Y|$ over the $|Y|$ steps between $\tau|Y|$ and $(\tau+1)|Y|$. We therefore have the following result.

Theorem 2** (Existence of Discrete Sequences).**

For any sequence $\{u_{k}\}$ of points from $U$ there exists a sequence $\{e_{k}\}$ of points from $E$ such that $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}$ is uniformly bounded for all $k\in\{1,2,\dots\}$.

Note that since we can always permute the entries in sequence $\{e_{k}\}$ while keeping $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{\infty}$ bounded, the existence of one sequence implies the existence of many (indeed exponentially many since the number of permutations grows exponentially with the permutation length).

IV-C Constructing Sequences of Discrete Actions Using Blocks

We now present our first method for constructing sequences of discrete actions, which uses a block-based approach.

Lemma 4**.**

Consider the setup of Lemma 3 and select

[TABLE]

where $z:=\sum_{i=1}^{|Y|}u_{i}$ and $\delta\in D$. Then, $-\boldsymbol{1}\preceq\delta+z-z^{\prime}\preceq\boldsymbol{1}$, with $z^{\prime}=\sum_{i=1}^{|Y|}e_{i}$.

Proof:

See Appendix C. ∎

Partitioning sequence $\{u_{k}\}$, $k\in\{1,2,\dots\}$, of points from $U$ into subsequences $\{u_{i}^{(\tau)}\}_{i=1}^{|Y|}$, $\tau\in\mathbf{Z}_{+}$ with $u_{i}^{(\tau)}=u_{\tau|Y|+i}$ and applying Lemma 4 recursively yields a sequence $\{e_{i}\}$ such that $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}$ is uniformly bounded. That is, we have the following.

Theorem 3**.**

Let $\{u_{k}\}$ be a sequence of points from $U$ partitioned into subsequences $\{u_{i}^{(\tau)}\}_{i=1}^{|Y|}$ with $u_{i}^{(\tau)}=u_{\tau|Y|+i}$, $\tau\in\mathbf{Z}_{+}$. For $i\in\{1,\dots,|Y|\}$, $\tau\in\{0,1,\dots\}$ select

[TABLE]

where $z_{\tau}:=\sum_{i=1}^{|Y|}u_{i}^{(\tau)}$. Then $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}$ is uniformly bounded for all $k=1,2,\dots$, where $e_{\tau|Y|+i}=e_{i}^{(\tau)}$.

Observe that, with this approach, the construction of a subsequence $\{e_{i}^{(\tau)}\}_{i=1}^{|Y|}$ requires that subsequence $\{u_{i}^{(\tau)}\}_{i=1}^{|Y|}$ is known. Hence, we refer to this as a block or batch approach. When sequence $\{u_{k}\}$ is observed in an online manner, then sequence $\{e_{k}\}$ must be constructed with a delay of $|Y|$ elements relative to $\{u_{k}\}$ since in order to construct $\{e_{i}^{(\tau)}\}_{i=1}^{|Y|}$ we must wait for $|Y|$ elements until $\{u_{i}^{(\tau)}\}_{i=1}^{|Y|}$ is observed.

Note that by a similar analysis we can immediately generalize this method of construction to situations where we partition sequence $\{u_{k}\}$, $k\in\{1,2,\dots\}$, into subsequences $\{u_{i}^{(\tau)}\}_{i=1}^{T_{\tau}|Y|}$, $T_{\tau}\in\mathbf{N}$, $\tau\in\mathbf{Z}_{+}$ with $u_{i}^{(\tau)}=u_{\sum_{t=0}^{\tau}T_{t}|Y|+i}$ i.e., where the subsequences can be different lengths so long as they are all some multiple of $|Y|$.

IV-C1 Constrained Control Actions

We can permute sequence $\{e_{i}^{(\tau)}\}_{i=1}^{|Y|}$ arbitrarily while keeping $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}$ uniformly bounded (since the vectors $u_{i}^{(\tau)}$ and $e_{i}^{(\tau)}$ both lie in the unit ball then $\|u_{i}^{(\tau)}-e_{j}^{(\tau)}\|_{\infty}\leq 2$ for all $i$, $j\in\{1,\dots,|Y|\}$). When the sequence of admissible actions is constrained, this flexibility can be used to select a sequence of actions which is admissible. For example, sequence $\{0,1,0,1\}$ might be permuted to $\{0,0,1,1\}$ if the cost of changing the action taken in the previous iteration is high.

IV-D Constructing Sequences of Discrete Actions Myopically

We now consider constructing a discrete valued sequence $\{e_{k}\}$ in a manner which is myopic or “greedy”, i.e., that selects each $e_{k}$, $k=1,2,\dots$ by only looking at $u_{i}$, $i=1,\dots,k$ and $e_{i}$, $i=1,\dots,k-1$. We have the following theorem.

Theorem 4**.**

Let $\{u_{k}\}$ be a sequence of points from $U$. Select

[TABLE]

where $s_{k}=\sum_{i=1}^{k}(u_{i}-e_{i})$. Then, we have that $-\boldsymbol{1}\preceq s_{k}\preceq(|Y|-1)\boldsymbol{1},$ and

[TABLE]

Proof:

See Appendix C. ∎

Theorem 4 guarantees that by using update (7) the difference $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}$ is uniformly bounded. Observe that update (7) selects a vector $e\in E$ that decreases the largest component of vector $s_{k}$, and so it does not provide flexibility to select other actions in $Y$. That is, the benefit of myopic selection comes at the cost of reduced freedom in the choice of discrete control action. Regarding complexity, solving (7) in general involves using exhaustive search. However, it is not necessary to solve (7) for every $u_{k}$, $k=1,2,\dots$ so long as the difference between the steps when (7) is performed is bounded. This is because the elements of $u_{k}$ and $e_{k}$ can diverge by at most $2$ at every step (recall both vectors lie in the unit ball) and so remain bounded between updates. Hence, the cost of (7) can be amortized across steps. We have the following corollary.

Corollary 1**.**

Consider the setup of Theorem 4 and suppose update (7) is performed at steps $k\in\{\tau_{1},\tau_{2},\dots\}:=\mathcal{T}\subseteq\mathbf{N}$; otherwise, $e_{k}$ is selected equal to $e_{k-1}$. Then, we have that $-\bar{\tau}\boldsymbol{1}\preceq s_{k}\preceq\bar{\tau}(|Y|-1)\boldsymbol{1},$ for all $k$ where $\bar{\tau}=\max_{j\in\{1,2,...\}}\{\tau_{j+1}-\tau_{j}\}$ and

[TABLE]

Proof:

See Appendix C. ∎

IV-D1 Constrained Control Actions

As already noted, we are interested in constructing sequences of actions in a flexible way that can be easily adapted to the constraints on the admissible actions. Corollary 1 allows us to accommodate one common class of constraints on the actions, namely that once an action has been initiated it must run to completion. For example, suppose we are scheduling packet transmissions where the packets are of variable size and a discrete action represents transmitting a single bit, then Corollary 1 ensures that a sequence of bits can be transmitted until a whole packet has been sent. The condition that $\bar{\tau}$ must be finite corresponds in this case to requiring that packets have finite length.

In the case of myopic updates it is difficult to give an algorithm without specifying a problem; nevertheless, we can establish the conditions that a generic algorithm should check when selecting a sequence of actions. As shown in the previous section, for finite $|Y|$ it is possible to construct a sequence of actions that breaks free from the past for a subsequence that is sufficiently large. The same concept can be applied to the myopic case, but now we must ensure that $\|s_{k}\|_{\infty}$ is bounded for all $k$. This motivates the following theorem.

Theorem 5**.**

Let $\{u_{k}\}$ be a sequence of points from $U$. For any sequence $\{e_{k}\}$ of points from $E$ we have that

[TABLE]

where $\gamma_{k}=-\min_{j\in\{1,\dots,|Y|\}}s_{k}(j)$, $s_{k}=\sum_{i=1}^{k}u_{i}-e_{i}$ and $C=\sqrt{|Y|}(|Y|-1)$.

Proof:

See Appendix C. ∎

Theorem 5 says that when we can construct a sequence of actions $\{e_{k}\}$, such that $\gamma_{k}$ is bounded then the difference $\|\sum_{i=1}^{k}u_{i}-e_{i}\|_{2}$ will be bounded. However, now $e_{k}$ does not need to be obtained as in (7) as long as it is selected with some “care”. Namely, by not selecting actions that decrease lower bound $\gamma_{k}$ “excessively”. For example, choosing a vector $e_{k}$ that decreases a positive component of vector $s_{k}$ will be enough. In addition to providing flexibility to respect constraints in the admissible actions, the implications of this are also important for scalability, namely when action set is large we do not need to do an exhaustive search over all the elements to select a vector from $E$.

V Remarks

V-A Discrete Queues

Perturbations on the Lagrange multipliers can be used to model the dynamics of queues that contain discrete valued quantities such as packets, vehicles or people. Suppose that the vector of approximate Lagrange multipliers $\mu_{k}$ takes a queue-like form, i.e.,

[TABLE]

where $\rho_{k}\in Y\subset\mathbf{Z}^{m}$ and represents the number of discrete elements that enter and leave the queues. Since $[\cdot]^{+}$ is a homogeneous function we can define $Q_{k}:=\mu_{k}/\alpha$ and obtain iterate

[TABLE]

That is, since $\rho_{k}$ takes values from a subset of integers we have that $Q_{k}$ is also integer valued, and therefore update (8) models the dynamics of a queue with discrete quantities. Provided $\epsilon_{k}=\lambda_{k}-\alpha Q_{k}$ satisfies the conditions of Theorem 1 then we can use $\alpha Q_{k}$ (which is equal to $\mu_{k}$) in update (\ref{eq:th_l_udpate}).

V-B Queue Stability

A central point in max-weight approaches is to show the stability of the system, which is usually established by the use of Foster-Lyapunov arguments. In our approach it is sufficient to use the boundedness of the Lagrange multipliers (i.e., claim (iv) in Theorem 1) and the fact that the difference $\|\lambda_{k}-\mu_{k}\|_{2}=\|\lambda_{k}-\alpha Q_{k}\|_{2}$ remains uniformly bounded. Observe that the uniform boundedness of $\|\lambda_{k}-\alpha Q_{k}\|_{2}$ implies that $\mathbf{E}(k^{-1}\sum_{i=1}^{k}\alpha Q_{i})=\alpha\mathbf{E}(k^{-1}\sum_{i=1}^{k}Q_{i})\prec\infty$ for all $k$ as well. Also, by the linearity of the expectation we can take the expectation inside the summation, and by dividing by $\alpha$ and taking $k\to\infty$ we can write

[TABLE]

which is the definition of strong stability given in the literature of max-weight—see, for example, [16, Definition 4].

V-C Primal-Dual Updates

The focus of the paper has been on solving the Lagrange dual problem directly, but the analysis encompasses primal-dual approaches as a special case. For instance, instead of obtaining an $x_{k}\in\arg\min_{x\in X}L(x,\lambda_{k},\delta)$ in each iteration we could obtain an $x_{k}$ that provides “sufficient descent” i.e., an update that makes the difference $L(x_{k},\lambda_{k},\delta)-h(\lambda_{k})$ decrease monotonically. This strategy is in spirit very close to the dynamical system approaches presented in [17, 18, 19], which usually require the objective function and constraints to have bounded curvature. In our case, having that the difference $L(x_{k},\lambda_{k},\delta)-h(\lambda_{k})$ decreases monotonically translates into having a sequence $\{\epsilon_{k}\}$ that converges to zero and so the perturbation vanishes.101010See Section II-C2 for a more detailed explanation of how the difference $L(x_{k},\lambda_{k},\delta)-h(\lambda_{k})$ relates to the $\epsilon_{k}$ perturbations.

V-D Unconstrained Optimization

The main motivation for using Lagrange relaxation is to tackle resource allocation problems of the sort tackled by max-weight approaches in the literature. However, our results (Lemma 1) naturally extend to unconstrained optimization problems. In this case $h$ becomes the unconstrained objective function that takes values from $\mathbf{R}^{m}\to\mathbf{R}$, the Lagrange multiplier $\lambda$ is the “primal” variable from a convex set $D$, and $\Pi_{D}$ (instead of $[\cdot]^{+}$) the Euclidean projection of a vector $\lambda\in\mathbf{R}^{m}$ onto $D$. The proof of Lemma 1 remains unchanged, and it is sufficient to note that the Euclidean projection onto a convex set is nonexpansive. That is, $\|\Pi_{\mathcal{D}}(\lambda_{1})-\Pi_{D}(\lambda_{2})\|_{2}\leq\|\lambda_{1}-\lambda_{2}\|_{2}$ for all $\lambda_{1},\lambda_{2}\in\mathbf{R}^{m}$.

VI Numerical Example

VI-A Problem Setup

Consider the network shown in Figure 2 where an Access Point (AP), labelled as node 1, transmits to two wireless stations, labelled as nodes 2 and 3. Time is slotted and in each slot packets arrive at the queues $Q(1)$ and $Q(2)$ of node 1, which are later transmitted to nodes 2 and 3. In particular, node 1 transmits packets from $Q(1)$ to $Q(3)$ (node 2) using link $l(1)$, and packets from $Q(2)$ to $Q(4)$ (node 3) using link $l(2)$ (see Figure 2).

We represent the connection between queues using the incidence matrix $A\in\{-1,0,1\}^{n\times l}$ where $-1$ indicates that a link is leaving a queue; $1$ that a link is entering a queue; and [math] that a queue and a link are not connected. For example, a $-1$ in the $j$’th element of the $i$’th row of matrix $A$ indicates that link $j$ is leaving queue $i$. The incidence matrix of the network illustrated in Figure 2 is given by

[TABLE]

In each time slot, the AP (node 1) takes an action from action set $Y:=\{y{(0)},y{(1)},y{(2)}\}=\{[0,0]^{T},[1,0]^{T},[0,1]^{T}\}$, where each action indicates which link to activate. For simplicity of exposition we will assume that activating a link corresponds to transmitting a packet, and therefore selecting action $y(1)$ corresponds to transmitting a packet from $Q(1)$ to node $Q(3)$; action $y(2)$ to transmitting a packet from $Q(2)$ to $Q(4)$; and action $y(0)$ to doing nothing, i.e., not activating any of the links.

The goal is to design a scheduling policy for the AP (select actions from set $Y$) in order to minimize a convex cost function of the average throughput (e.g., this might represent the energy usage), and ensure that the system is stable, i.e., the queues do not overflow and so all traffic is served.

The convex or fluid-like formulation of the problem is

[TABLE]

where $f:\mathbf{R}^{2}\to\mathbf{R}$ is a convex utility function, $X\subseteq\mathrm{conv}{(Y)}$, $A$ the incidence matrix given in (9), and $b$ a vector that captures the mean exogenous packet arrivals/departures in the system.111111More precisely, $b(1)$ and $b(2)$ capture, respectively, the mean arrival into $Q(1)$ and $Q(2)$; and $b(3)$ and $b(4)$, respectively, the mean departure rate from $Q(3)$ and $Q(4)$.

VI-B Unconstrained Control Actions

Problem (12) can be solved with the optimization framework presented in Section III. That is, with update

[TABLE]

where $x_{k}\in\arg\min_{x\in X}\ \{f(x)+\lambda_{k}^{T}Ax\}$, and note we have replaced $b$ with random variable $B_{k}$ in order to capture the fact that packet arrivals at node 1 might be a realization of a stochastic process.

Observe from update (13) that by selecting an $x_{k}\in X:=\mathrm{conv}(Y)$ we obtain the fraction of time each of the links should be used in each iteration, but not which packet to transmit from each of the queues in each time slot. Nonetheless, we can easily incorporate (discrete) control actions $y_{k}\in Y$ by using, for example, Theorem 4. Also, note that if we define approximate Lagrange multiplier $\mu_{k+1}=[\mu_{k}+\alpha(Ay_{k}+B_{k})]^{+}$ and let $Q_{k}:=\mu_{k}/\alpha$ we obtain queue dynamics

[TABLE]

which are identical to those of the real queues in the system. By Theorem 1 we can use update $x_{k}\in\arg\min_{x\in X}\ \{f(x)+\mu_{k}^{T}Ax\}=\arg\min_{x\in X}\ \{f(x)+\alpha Q_{k}^{T}Ax\}$, and with this change we now do not need to compute the Lagrange multiplier $\lambda_{k}$. Looking at the current queue occupancies $Q_{k}$ in the system is enough for selecting control actions.

VI-C Constrained Control Actions

We now extend the example to consider the case where the admissible sequence of control actions is constrained. Specifically, suppose it is not possible to select action $y{(1)}$ after $y{(2)}$ without first selecting $y{(0)}$; and in the same way, it is not possible to select $y{(2)}$ after $y{(1)}$ without first selecting $y{(0)}$. However, $y{(1)}$ or $y{(2)}$ can be selected consecutively. An example of an admissible sequence would be

[TABLE]

This type of constraint appears in a variety of applications —they are usually known in the literature as reconfiguration or switchover delays [20]— and capture the cost of selecting a “new” control action. In this wireless communication example, the requirement of selecting action $y(0)$ every time the AP wants to change from action $y(1)$ to $y(2)$ and from $y(2)$ to $y(1)$ might be regarded as the time required for measuring Channel State Information (CSI) in order to adjust the transmission parameters.121212The CSI in wireless communications is in practice measured periodically, and not just at the beginning of a transmission, but we assume this for simplicity. The extension is nevertheless straightforward.

In this case, the constraints on the selection of control actions will affect the definition of set $X$ in the problem. To see this, observe that if we select a sequence of actions in blocks of length $T|Y|$ with $T\in\mathbf{N}$, and $y{(1)}$ and $y{(2)}$ appear (each) consecutively, then $y{(0)}$ should appear at least twice in order for the subsequence to be compliant with the transmission protocol. Conversely, any subsequence of length $T|Y|$ in which $y{(0)}$ appears at least twice can be reordered to obtain a subsequence that is compliant with the transmission protocol. For example, if $T=3$ and we have subsequence

[TABLE]

we can reorder it to obtain

[TABLE]

which is a subsequence compliant with the transmission protocol. Since from Section IV-C, we can always choose a subsequence of discrete actions and then reorder its elements, we just need to select a set $X$ such that $y{(0)}$ can be selected twice in a subsequence of $T|Y|$ elements. This will be the case when every point $x\in X$ can be written as a convex combination of points from $Y$ that uses action $y({0})$ at least fraction $2/(T|Y|)$. That is, when

[TABLE]

Observe from (14) that as $T$ increases we have that $X\to\mathrm{conv}(Y)$, which can be regarded as increasing the capacity of the network since it will be possible to use links 1 and 2 a larger fraction of time. Figure 3 illustrates the capacity of the network (set $X$) changes depending on parameter $T$.

VI-C1 Simulation

We run a simulation using $f(x)=\|Sx\|_{2}^{2}$, $S=\mathrm{diag}([1,3])$, $b=[0.25,0.5,-1,-1]^{T}$, $\lambda_{1}=\alpha Q_{1}=0$ and $\alpha=0.01$. At each iteration we perform update (13) where $x_{k}\in\arg\min_{x\in X}\{f(x)+\alpha Q_{k}^{T}Ax\}$; and $B_{k}(1)$ and $B_{k}(2)$ are Bernoulli random variables with mean $b(1)$ and $b(2)$ respectively; $B_{k}(3)$ and $B_{k}(4)$ are equal to $-1$ for all $k$ and so the service of nodes 2 and 3 is deterministic. Discrete control actions are selected with update (6) with $T=3$, and so we have that the number of elements in a block or subsequence is 9. The elements in a block are reordered in order to match the protocol constraints of the AP.

Figure 4 plots the occupancy of the queues in the system. Observe that $\alpha Q_{k}$ converges to a ball around $\lambda^{\star}=[2,1]^{T}$, the optimal dual variable associated to the fluid/convex problem (12). Figure 4 (b) is the detail of (a) for an interval of 200 iterations, and shows that queues are indeed integer valued.

Figure 5 plots the convergence of the objective function for step sizes $\alpha\in\{10^{-2},10^{-3}\}$. Observe that using a smaller step size heavily affects the convergence time, which is in line with the accuracy vs. convergence time tradeoff in subgradient methods with constant step size.

VII Related Work

In this section, we explain the differences of our contributions with previous work.

VII-A Contribution (i) — Unifying Framework

VII-A1 Lagrange Multipliers and Queues

Concerning the existence of a connection between the discrete-valued queue occupancy in a queueing network and continuous-valued Lagrange multipliers, [17] shows that asymptotically, as the design parameters $\beta\rightarrow 0$ and $t\rightarrow\infty$, the scaled queue occupancy converges to the set of dual optima. Also, in [21] it is established that a discrete queue update is exponentially attracted to a “static” vector of Lagrange multipliers. Regarding queues with continuous-valued occupancy (so “fluid” type queues), it is shown in [5] that the scaled queue occupancy is equal to the Lagrange multipliers generated by an associated dual subgradient update. However, this approach does not encompass common situations such as discrete packet arrivals. In [6], the authors extend this observation to consider queues that contain errors due to asynchronism in the network but do not present any analytic bounds.

Our work extends this by identifying queues with $\alpha$-scaled approximations of the Lagrange multipliers, providing both a non-asymptotic bound on the approximation and establishing how this affects convergence. In particular, we show how the convergence of the dual subgradient method is affected by $\epsilon_{k}$-subgradients in the form of approximate Lagrange multipliers.

VII-A2 Generality

Our framework is an extension of Nedić’s and Ozdaglar’s work in [11] to consider stochastic and $\epsilon_{k}$-subgradients, which we later identify, respectively, with myopic and discrete actions. Unlike previous works, we have abstracted these two features and made them accessible from within a clean mathematical framework that does not rely on a specific application. Hence, our contribution is in spirit very different to the works in [5, 6, 7, 8, 9] which focus on specific congestion control and scheduling applications. For example, [7] addresses the problem of designing a congestion controller that is more gradual than previous dual controllers so that it “mimics” TCP’s behavior, and [9] deals with the joint congestion control, scheduling, and routing in networks with time-varying channel conditions.

VII-A3 Convergence

The max-weight analysis in the literature [5, 6, 7, 8, 9] provides asymptotic convergence and stability results. This is in marked contrast to our framework, in which we can make quantitative statements of the system state in finite time. In particular, we provide upper and lower bounds on the objective function, bound the constraints violation, and show that the expected value of the Lagrange multipliers is bounded. Compare, for example, [9, Theorem 1] with our Theorem 1. The work that is perhaps closest to our work in terms of its convergence analysis is Neely’s “drift-plus-penalty” algorithm, which, as shown by Huang in [21], can be regarded as a randomized version of the dual subgradient method [22]. Nonetheless, our analysis is significantly more general since it separates the choice of discrete actions from a specific choice of subgradient update. The latter allows us to construct discrete control policies in a much more flexible manner than previously, which is the second contribution of the paper.

VII-B Contribution (ii) — General Control Policies

Previous work in [5, 6, 7, 8, 9] shows that discrete actions can be obtained as a result of minimizing a linear program. This is very different to our work, where we can have discrete actions in any variable in the optimization and this is not restricted to linear programs. In our previous work [19], we revisited max-weight policies and provided two different classes of discrete-like updates (Greedy and Frank-Wolfe; see Theorems 1 and 2 in [19]). However, those two classes of updates do not provide enough flexibility to design control policies that capture the constraints on the choice of action in many applications; for example, when changes in the action are costly or cannot be made instantaneously.

In the present work, we show that discrete actions can be selected as any sequence $\{y_{k}\}$ that stays close, in the appropriate sense, to a sequence $\{x_{k}\}$ obtained with the subgradient method. One of the key consequences is that the selection of a discrete action in a time slot can then depend on the action selected in the previous time slot, and so the elements in the sequence of actions can be strongly correlated. Another key consequence is that this enables the construction of discrete control policies by only checking that $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}$ is uniformly bounded. That is, without requiring to re-prove the stability/optimality of the system for every new policy. The question of how to select sequence $\{y_{k}\}$ so that $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}$ remains uniformly bounded is answered in our third contribution.

VII-C Contribution (iii) — Discrete Control Policies

In the paper, we propose two new types of discrete control policies: an online policy and another that works with blocks. The policy that works with blocks is more complex but allows us to reshuffle discrete actions, and so we have much more flexibility as to how to select discrete control actions.

Constraints on the sequence of actions have received relatively little attention in the max-weight literature. In [23], Çelik et al. show that the original max-weight (myopic) policy fails to stabilize a system when there are reconfiguration delays associated with selecting new actions/configurations, and propose a variable frame size max-weight policy where actions are allocated in frames (i.e., blocks) to minimize the penalties associated with configuration changes. This algorithm is similar to the block-algorithm used in the numerical example in Section VI, but a notable difference is that in our case the design of the scheduling policy is done simply by ensuring that the $\alpha$-scaled queues stay close to the Lagrange multipliers which yields an approach of much greater generality. In [3], Maguluri et al. consider also scheduling in a frame based fashion. However, in their case suboptimal schedules are selected within a frame so that a max-weight schedule can always be selected at the beginning of a frame. Our work differs from [3] fundamentally because we never require to select a max-weight schedule to guarantee optimality. That is, keeping difference $\|\sum_{i=1}^{k}x_{i}-y_{i}\|_{2}$ uniformly bounded is enough. We also note Lin and Shroff’s work in [6], where it is observed that the capacity region of the system is reduced as a result of selecting imperfect schedules.131313Imperfect schedules are the result of not being able to select a maximum weight matchings in every time slot. This is different from our work because we can amortize the complexity of selecting a new schedule/action by selecting the action used in the previous iteration (see Corollary 1). Hence, one does not need to sacrifice resources due to the complexity of selecting a new schedule.

VIII Conclusions

We have presented a framework that brings the celebrated max-weight features (discrete control actions and myopic scheduling decisions) to the field of convex optimization. In particular, we have shown that these two features can be encompassed in the subgradient method for the Lagrange dual problem by the use of $\delta_{k}$ and $\epsilon_{k}$ perturbations, and have established different types of convergence depending on their statistical properties. One of the appealing features of our approach is that it decouples the selection of a control action from a particular choice of subgradient, and so allows us to design a range of control policies without changing the underlying convex updates. We have proposed two classes of discrete control policies: one that can make discrete actions by looking only at the system’s current state, and another which selects actions using blocks. The latter class is useful for handling systems that have constraints on how actions can be selected. We have illustrated this with an example where there are constraints on how packets can be transmitted over a wireless network.

Appendix A Preliminaries

A-A Subgradient Convergence

Let $x_{k}\in\arg\min_{x\in X}L(x,\lambda_{k})$ and observe we can write upper bound

[TABLE]

and lower bound

[TABLE]

If we use the fact that $\|g(x)\|_{2}\leq\sigma_{g}$ for all $x\in X$ by Assumption 2, we have

[TABLE]

From (15) we can see that if difference $\|\lambda_{k}-\lambda^{\star}\|_{2}$ decreases then the difference $|h(\lambda_{k})-h(\lambda^{\star})|$ must eventually also decrease.

A-B $\epsilon_{k}$-subgradients

In this section, we show that the use of $\epsilon_{k}$-subgradients is equivalent to approximately minimizing the Lagrangian. Similar results can be found in Bertsekas [24, pp. 625], but we include them here for completeness.

Let $x_{k}\in\arg\min_{x\in X}L(x,\mu_{k})$ with $\mu_{k}=\lambda_{k}+\epsilon$ for some $\epsilon\in\mathbf{R}^{m}$, and observe that $h(\mu_{k})=L(x_{k},\mu_{k})=L(x_{k},\lambda_{k}+\epsilon)=f(x_{k})+(\lambda_{k}+\epsilon)^{T}g(x_{k})\leq f(x_{k})+\lambda_{k}^{T}g(x_{k})+\|\epsilon\|_{2}\|g(x_{k})\|_{2}\leq f(x_{k})+\lambda_{k}^{T}g(x_{k})+\|\epsilon\|_{2}\sigma_{g}$ where the last inequality follows since $\sigma_{g}:=\max_{x\in X}\|g(x)\|_{2}$. Hence,

[TABLE]

We now proceed to show that $|h(\mu_{k})-h(\lambda_{k})|$ is bounded. Consider two cases. Case (i) $h(\mu_{k})<h(\lambda_{k})$. From the concavity of $h$ we have that $h(\lambda_{k})\leq h(\mu_{k})+\partial h(\mu_{k})^{T}(\lambda_{k}-\mu_{k})=h(\mu_{k})+\partial h(\lambda_{k}+\epsilon)^{T}\epsilon=h(\mu_{k})+g(x_{k})^{T}\epsilon,$ and therefore

[TABLE]

Case (ii) $h(\lambda_{k})>h(\mu_{k}$). Following the same steps than in the first case we obtain $h(\mu_{k})\leq h(\lambda_{k},\delta)-g(x_{k})^{T}\epsilon$, and therefore $0\leq h(\mu_{k})-h(\lambda_{k})\leq\|g(x_{k})\|_{2}\|\epsilon\|_{2}\leq\|\epsilon\|_{2}\sigma_{g}.$ Combining both cases yields $|h(\lambda_{k})-h(\mu_{k})|\leq\|\epsilon\|_{2}\sigma_{g},$ and using (16) we have

[TABLE]

where the lower bound follows immediately since $h(\lambda_{k})\leq L(x,\lambda_{k})$ for all $x\in X$. Hence, the error obtained as a result of selecting an $x_{k}$ that minimizes $L(x,\mu_{k})$ (instead of $L(x,\mu_{k})$) is proportional to the difference between $\lambda_{k}$ and $\mu_{k}$.

Appendix B Proofs of Section III

B-A Proof of Lemma 1

For any vector $\theta\in\mathbf{R}^{m}$ we have

[TABLE]

where in the last equation we have used the fact that $\alpha^{2}\|g(x_{k})+\delta_{k}\|_{2}^{2}=\alpha^{2}\|g(x_{k})+\delta_{k}-\delta+\delta\|_{2}^{2}=\alpha^{2}\|g(x_{k})+\delta\|_{2}^{2}+\alpha^{2}\|\delta_{k}-\delta\|_{2}^{2}+2\alpha^{2}(g(x_{k})+\delta)^{T}(\delta_{k}-\delta).$ Similarly, observe that $(\lambda_{k}-\theta)^{T}(g(x_{k})+\delta_{k})=(\lambda_{k}-\theta)^{T}(g(x_{k})+\delta+\delta_{k}-\delta)=(\lambda_{k}-\theta)^{T}(g(x_{k})+\delta)+(\lambda_{k}-\theta)^{T}(\delta_{k}-\delta)$ and since

[TABLE]

we have

[TABLE]

where (18) follows from the fact that $h(\theta)=\min_{x\in X}L(x,\theta)\leq L(x_{k},\theta)$. Applying the expansion recursively for $i=1,\dots,k$ we have

[TABLE]

Next, observe that since

[TABLE]

we have that

[TABLE]

and therefore

[TABLE]

Rearranging terms and dividing by $2\alpha k$ yields the stated result.

B-B Proof of Theorem 1

Let $\theta=\lambda^{\star}(\delta)$ in Lemma 1. From (2) and (21) we can write

[TABLE]

where the last equation follows from the convexity of $f$. Rearranging terms

[TABLE]

Now, let $\theta=0$ in (17) to obtain

[TABLE]

Using the fact that $\|g(x_{k})+\delta\|_{2}^{2}\leq\sigma_{g}^{2}$ for all $k$ and applying the latter expansion recursively

[TABLE]

Rearranging terms, dropping $\|\lambda_{k+1}\|_{2}$ since it is nonnegative, and dividing by $2\alpha k$ yields

[TABLE]

Combining the last bound with (23), and using the fact that $h(\lambda^{\star}(\delta),\delta)=f^{\star}(\delta)$ (by strong duality, c.f., Assumption 1) yields

[TABLE]

Taking expectations with respect to $\delta_{i}$, $i=1,2,\dots,k$ we have $\mathbf{E}(\|\delta_{i}-\delta\|_{2}^{2})=\sigma_{\delta}^{2}$, and $\mathbf{E}((\delta_{i}-\delta)^{T}(g(x_{i})+\delta))=0$ since $x_{i}$ and $\delta_{i}$ are independent. Therefore,

[TABLE]

and we have obtained claim (i).

We now proceed to lower bound (25). Taking expectations with respect to $\delta_{i}$, $i=1,2,\dots,k$ in Lemma 1, and using the fact that $\lambda_{i}$ and $\delta_{i}$ are independent, so $\mathbf{E}((\lambda_{i}-\theta)^{T}(\delta_{i}-\delta))=0$ and $\mathbf{E}((\delta_{i}-\delta)^{T}(g(x_{i})+\delta))=0$, we have

[TABLE]

Next, by the convexity of $-h(\cdot,\delta)$ we can write

[TABLE]

and letting $\theta=\lambda^{\star}(\delta)$

[TABLE]

where the upper bound follows from the fact that $h(\lambda^{\star}(\delta),\delta)=\sup_{\lambda\succeq 0}h(\lambda,\delta)$. Next, from the saddle point property of the Lagrangian

[TABLE]

where the expectation on $\bar{x}_{k}$ in the RHS of (a) is taken with respect to $\delta_{i}$, $i=1,\dots,k$; and (b) follows from the convexity of $f$ and $g$. Therefore,

[TABLE]

We need to show that $\mathbf{E}(\bar{\lambda}_{k})^{T}\mathbf{E}(g(\bar{x}_{k})+\delta)$ is upper bounded. Observe first that for any sequence $\{x_{k}\}$ from $X$ we can write

[TABLE]

and applying the latter recursively we have that

[TABLE]

Dropping $\lambda_{1}$ since it is nonnegative, dividing by $\alpha k$, and using the convexity of $g$ follows that

[TABLE]

and taking expectations with respect to $\delta_{i}$, $i=1,\dots,k$

[TABLE]

Multiplying both sides by $\mathbf{E}(\bar{\lambda}_{k})$ (where the expectation is with respect to $\delta_{i}$, $i=1,\dots,k$) and using Cauchy-Schwarz

[TABLE]

We proceed to show that $\|\mathbf{E}(\bar{\lambda}_{k})\|_{2}$ is bounded using Lemma 6 in [19]. This lemma says that for any $\chi\geq 0$ then $\mathcal{Q}_{\chi}:=\{\lambda\succeq 0\mid h(\lambda)\geq h(\lambda^{\star}(\delta))-\chi\}$ is a bounded set. Further, for any $\lambda\in\mathcal{Q}_{\chi}$ we have that $\|\lambda\|_{2}\leq\frac{1}{\upsilon}(f(\hat{x})-h(\lambda^{\star}(\delta))+\chi)$ where $\hat{x}$ is a Slater point, and $\upsilon:=\min_{j\in\{1,\dots,m\}}-g_{j}(\hat{x})$ a constant that does not depend on $\chi$. Note that $\upsilon>0$. Now, observe that since $\mathbf{E}(h(\bar{\lambda}_{k},\delta))\leq h(\mathbf{E}(\bar{\lambda}_{k}),\delta)$, from (27) we can write

[TABLE]

Hence, if we identify $-\chi$ with the LHS of (30) we obtain that $\|\mathbf{E}(\bar{\lambda}_{k})\|_{2}$ is bounded. That is,

[TABLE]

We continue by giving a bound on $\|\mathbf{E}(\lambda_{k+1})\|_{2}$. Taking expectations in (22) with respect to $\delta_{i},\ i=1,\dots,k$, letting $\theta=\lambda^{\star}(\delta)$, $\|g(x_{k})+\delta\|_{2}\leq\sigma_{g}$, and using the fact that $\lambda_{k}$ and $\delta_{k}$ are independent for all $k$, we have

[TABLE]

Next, observe that since $h(\lambda_{i},\delta)-h(\lambda^{\star}(\delta),\delta)\leq 0$ for all $i=1,\dots,k$ we can write $\mathbf{E}(\|\lambda_{k+1}-\lambda^{\star}(\delta)\|_{2}^{2})\leq\|\lambda_{1}-\lambda^{\star}(\delta)\|_{2}^{2}+\alpha^{2}(\sigma_{g}^{2}+\sigma_{\delta}^{2})k+2\alpha\sum_{i=1}^{k}\|\epsilon_{i}\|_{2}\sigma_{g}$ and by using the convexity of $\|\cdot\|_{2}^{2}$, $\|\mathbf{E}(\lambda_{k+1})-\lambda^{\star}(\delta)\|_{2}^{2}\leq\|\lambda_{1}-\lambda^{\star}(\delta)\|_{2}^{2}+\alpha^{2}(\sigma_{g}^{2}+\sigma_{\delta}^{2})k+2\alpha\sum_{i=1}^{k}\|\epsilon_{i}\|_{2}\sigma_{g}.$ That is, $\mathbf{E}(\lambda_{k+1})$ is within a ball around $\lambda^{\star}(\delta)$. Next, since $\|\lambda^{\star}(\delta)\|_{2}^{2}$ is bounded we can write $\|\mathbf{E}(\lambda_{k+1})\|_{2}^{2}\leq\|\lambda^{\star}(\delta)\|_{2}^{2}+\|\lambda_{1}-\lambda^{\star}(\delta)\|_{2}^{2}+\alpha^{2}(\sigma_{g}^{2}+\sigma_{\delta}^{2})k+2\alpha\sum_{i=1}^{k}\|\epsilon_{i}\|_{2}\sigma_{g}$ and by taking square roots in both sides and using the concavity of $x^{a}$ for $x\in\mathbf{R}_{++}$ with $0<a<1$ (see [10, pp. 71]) we have

[TABLE]

where the last inequality follows from the concavity of the square root and the fact that all the terms are nonnegative. Hence,

[TABLE]

and so we can use (28) to lower bound (25), and obtain the bound claimed in (ii).

Claim (iii) follows from (29) and (32). Claim (iv) follows from (31).

Appendix C Proof of Section IV

C-A Proof of Lemma 3

To start, let $V=|Y|$ and note we always have $\boldsymbol{1}^{T}(z+\delta)=V$ since $z$ is the sum of $V$ elements from $U$, $u^{T}\boldsymbol{1}=1$ and $\delta^{T}\boldsymbol{1}=0$ for all $u\in U$, $\delta\in D$. Further, $(z+\delta)\succeq-\boldsymbol{1}$ since $\delta\succeq-\boldsymbol{1}$ and $z\succeq 0$. Now, let $r:=(z+\delta)$ and define

[TABLE]

where the floor in $b$ is taken element-wise. That is, $a\in[-1,0]^{V}$, $b\in\{0,1,\dots,V\}^{V}$, $c\in[0,1)^{V}$. For example, if $r=[2.2,-0.2]^{T}$ then $a=[0,-0.2]^{T}$, $b=[2,0]^{T}$ and $c=[0.2,0]^{T}$. Observe,

[TABLE]

and since $b$ is integer valued $\boldsymbol{1}^{T}b\in\mathbf{Z}_{+}$, which implies $\boldsymbol{1}^{T}(a+c)\in\mathbf{Z}_{+}$. Next, let $\boldsymbol{1}^{T}b=V-\boldsymbol{1}^{T}(a+c):=V^{\prime}$, and observe $b$ can be written as the sum of $V^{\prime}$ elements from $E$, i.e.,

[TABLE]

Next, since $-\boldsymbol{1}\preceq a+c\prec\boldsymbol{1}$ and $\boldsymbol{1}^{T}(a+c)=V-V^{\prime}:=V^{\prime\prime}$, there must exist at least $V^{\prime\prime}$ elements in vector $(a+c)$ that are nonnegative. If we select $V^{\prime\prime}$ elements from $E$ that match the nonnegative components of vector $(a+c)$ we can construct a subsequence $\{e_{i}\}_{i=1}^{V^{\prime\prime}}$ such that

[TABLE]

Finally, letting $z^{\prime}=\sum_{i=1}^{V^{\prime}}e_{i}+\sum_{i=1}^{V^{\prime\prime}}e_{i}$ yields the result.

C-B Proof of Lemma 4

First of all, recall from the proof of Lemma 3 that $\boldsymbol{1}^{T}(\delta+z-z^{\prime})=0$, $\delta+z\succeq-\boldsymbol{1}$ and that $V:=\boldsymbol{1}^{T}(z+\delta)$ where $V:=|Y|$. Next, define $r_{i}:=\delta+z-\sum_{\kappa=1}^{i-1}e_{i}$, $i=1,2,\dots$ and note that update $e_{i}\in\arg\min_{e\in E}\|r_{i}-e\|_{\infty}$ decreases the largest component of vector $r_{i}$, i.e., in each iteration a component of vector $r_{i}$ decreases by $1$, and therefore $\boldsymbol{1}^{T}r_{i}=V-i+1$ with $i=1,\dots,|Y|+1$.

For the lower bound observe that if $r_{i+1}(j)<-1$ for some $j=1,\dots,|Y|$ we must have that $r_{i}\prec 0$ since the update $e_{i}\in\arg\min_{e\in E}\|r_{i}-e\|_{\infty}$ selects to decrease the largest component of vector $r_{i}$. However, since $\boldsymbol{1}^{T}r_{i}\geq 0$ for all $i=1,\dots,|Y|+1$ we have that vector $r_{|Y|}$ has at least one component that is nonnegative. Therefore, $r_{|Y|+1}\succeq-\boldsymbol{1}$ and $\delta+z-z^{\prime}\succeq-\boldsymbol{1}$. For the upper bound define $a_{i}=-[-r_{i}]^{+}$, $b_{i}=\lfloor r_{i}-a_{i}\rfloor$, $c_{i}=r_{i}-a_{i}-b_{i}$, $i=1,\dots,|Y|+1$ and note that $-\boldsymbol{1}\preceq a_{i}\preceq 0$ and $0\preceq c_{i}\prec\boldsymbol{1}$ for all $i=1,\dots,|Y|+1$, and that $\boldsymbol{1}^{T}b_{i}$ decreases by $1$ in each iteration if $\boldsymbol{1}^{T}b_{i}\geq 1$. Hence, $b_{|Y|+1}=0$ and therefore $-\boldsymbol{1}\preceq r_{|Y|+1}=a_{|Y|+1}+c_{|Y|+1}=\delta+z-z^{\prime}\preceq\boldsymbol{1}$ and we are done.

C-C Proof of Theorem 4

We begin by noting that since $\boldsymbol{1}^{T}u_{k}=1=\boldsymbol{1}^{T}e_{k}$ then $\boldsymbol{1}^{T}s_{k}=0$ for all $k=1,2,\dots$. Also note that since $u_{k}\in U$ all elements of $u_{k}$ are nonnegative and at least one element must be non-zero since $\boldsymbol{1}^{T}u_{k}=1$.

We now proceed by induction to show that there always exists a choice of $e_{k+1}$ such that $s_{k}\succeq-\boldsymbol{1}$, $k=1,2,\dots$. When $k=1$ let element $u_{1}(j)$ be positive (as already noted, at least one such element exists). Selecting $e_{1}=v_{j}$ then it follows that $-1<u_{1}(j)-e_{1}(j)\leq 0$ and so $-1\prec u_{1}-e_{1}\prec 1$. That is, ${s}_{1}\succeq-\boldsymbol{1}$. Suppose now that ${s}_{k}\succ-\boldsymbol{1}$. We need to show that ${s}_{k+1}\succeq-\boldsymbol{1}$. Now ${s}_{k+1}={s}_{k}+{u}_{k+1}-{e}_{k+1}$. Since ${s}_{k}\succeq-\boldsymbol{1}$, ${s}_{k}(j)\geq-1$ $\forall j=1,\dots,|Y|$. Also, $\boldsymbol{1}^{T}s_{k}=0$, so either all elements are 0 or at least one element is positive. If they are all zero then we are done (we are back to the $k=1$ case). Otherwise, since all elements of ${u}_{k+1}$ are nonnegative then at least one element of ${s}_{k}+{u}_{k+1}$ is positive. Let element $s_{k}(j)+u_{k+1}(j)$ be the largest positive element of $s_{k}+u_{k+1}$. Selecting $e_{k+1}={v}_{j}$ then it follows that $s_{k}(j)+u_{k+1}(j)-e_{k+1}(j)\geq-1$. That is, $s_{k+1}\succeq-\boldsymbol{1}$.

We now show that $s_{k}$ is upper bounded. Recall ${e}_{k+1}$ can always be selected such that ${s}_{k}\succeq-\boldsymbol{1}$, and also $\boldsymbol{1}^{T}{s}_{k}=0$. Since $\boldsymbol{1}^{T}{s}_{k}=0$ either $s_{k}$ is zero or at least one element is positive. Since $\boldsymbol{s}_{k}\succeq-\boldsymbol{1}$ and at most $|Y|-1$ elements are negative, then the sum over the negative elements is lower bounded by $-(|Y|-1)$. Since $\boldsymbol{1}^{T}{s}_{k}=0$ it follows that the sum over the positive elements must be upper bounded by $|Y|-1$. Hence, $\|{s}_{k}\|_{\infty}\leq(|Y|-1)$.

C-D Proof of Corollary 1

Since $s_{k}$ has at least one component that is nonnegative, and update (7) selects the largest component of vector $s_{k}$ when $k\in\mathcal{T}$, we have that a component of vector $s_{k}$ can decrease at most by $\bar{\tau}$ in an interval $\{\tau_{j}-\tau_{j+1}\}$ for all $j=1,2,\dots$. Hence, $s_{k}\succeq-\bar{\tau}\boldsymbol{1}$ for all $k$. Next, since $s_{k}^{T}\boldsymbol{1}=0$ for all $k$ and the sum over the negative components is at most $-\bar{\tau}(|Y|-1)$, we have that $s_{k}\preceq\bar{\tau}(|Y|-1)\boldsymbol{1}$. The rest of the proof follows as in Theorem 4.

C-E Proof of Theorem 5

Recall that since $\boldsymbol{1}^{T}u_{k}=1=\boldsymbol{1}^{T}e_{k}$ then $\boldsymbol{1}^{T}s_{k}=0$ for all $k=1,2,\dots$, and therefore $s_{k}$ is either [math] or at least one of its components is strictly positive. Next, observe that since $\gamma_{k}=-\min_{j\in\{1,\dots,|Y|\}}s_{k}(j)$ we have that $\max_{j\in\{1,\dots,|Y|\}}s_{k}(j)\leq\gamma_{k}(|Y|-1)$, which corresponds to the case where $|Y|-1$ components of vector $s_{k}$ are equal to $\gamma_{k}$. The rest of the proof continues as in the proof of Theorem 4.