Distributed Constrained Online Learning

TL;DR

None

Contribution

None

Abstract

In this paper, we consider groups of agents in a network that select actions in order to satisfy a set of constraints that vary arbitrarily over time and minimize a time-varying function of which they have only local observations. The selection of actions, also called a strategy, is causal and decentralized, i.e., the dynamical system that determines the actions of a given agent depends only on the constraints at the current time and on its own actions and those of its neighbors. To determine such a strategy, we propose a decentralized saddle point algorithm and show that the corresponding global fit and regret are bounded by functions of the order of $\sqrt{T}$. Specifically, we define the global fit of a strategy as a vector that integrates over time the global constraint violations as seen by a given node. The fit is a performance loss associated with online operation as opposed to offline clairvoyant operation which can always select an action if one exists, that satisfies the constraints at all times. If this fit grows sublinearly with the time horizon it suggests that the strategy approaches the feasible set of actions. Likewise, we define the regret of a strategy as the difference between its accumulated cost and that of the best fixed action that one could select knowing beforehand the time evolution of the objective function. Numerical examples support the theoretical conclusions.

Figures

Equations

$${\mathbf{x}}_i^{\ell }:=$$

$$f_i(t,{\mathbf{x}}_i)⪯0,\forall t\in [0,T].$$

$$\displaystyle{\mathbf{x}}_{i}^{*}\ :=\$$

$$f_j(t,{\mathbf{x}}_i)⪯0,\forall j\text{ and }t\in [0,T].$$

$$\displaystyle{\mathbf{x}}_{i}^{*}\ =\$$

$$f(t,{\mathbf{x}}_i)⪯0,\forall t\in [0,T].$$

$$\big{|}f_{0i}(t,{\mathbf{x}})-f_{0i}(t,{\mathbf{y}})\big{|}\leq L_{0}\big{\|}{\mathbf{x}}-{\mathbf{y}}\big{\|},$$

$$\big{|}f_{kj}(t,{\mathbf{x}})-f_{kj}(t,{\mathbf{y}})\big{|}\leq L_{f}\big{\|}{\mathbf{x}}-{\mathbf{y}}\big{\|},$$

$$f_i(t,{\mathbf{x}}^{†})\prec 0,\forall i\text{ and }t\in [0,T].$$

$$f_0(t,{\mathbf{x}}^{\ast })-\underset{x\in {\mathcal{X}}^N}{\min }{f}_0(t,\mathbf{x})\le K,$$

$${\mathcal{R}}_{Tj}^i:={\int }_0^T{f}_{0j}(t,{\mathbf{x}}_i(t))dt-{\int }_0^T{f}_{0j}(t,{\mathbf{x}}_i^{\ast })dt.$$

$${\mathcal{F}}_{Tj}^i:={\int }_0^T{f}_j(t,{\mathbf{x}}_i(t))dt-{\int }_0^T{f}_j(t,{\mathbf{x}}_i^{\ast })dt.$$

$${\mathcal{F}}_T^i:={\int }_0^{T}f(t,{\mathbf{x}}_i(t))dt\le C_{f}h(T).$$

$${\mathcal{R}}_T^i:={\int }_0^T{f}_0(t,{\mathbf{x}}_i(t))dt-{\int }_0^T{f}_0(t,{\mathbf{x}}^{\ast })dt\le Ch(T).$$

$${\mathbf{x}}^{\ast }:=$$

$$f_i(t,{\mathbf{x}}_i)⪯0,\forall t\in [0,T],\forall i\in \mathcal{V},$$

$${\mathbf{x}}_i={\mathbf{x}}_j,\forall i\in \mathcal{V},j\in {\mathcal{N}}_i.$$

$$g_{ij}({\mathbf{x}}_i,{\mathbf{x}}_j)={\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }^2-\gamma \le 0,$$

$${\tilde{\mathbf{x}}}^{\ast }:=$$

$$f_i(t,{\mathbf{x}}_i)⪯0,\forall t\in [0,T],\forall i\in \mathcal{V},$$

$$g_{ij}({\mathbf{x}}_i,{\mathbf{x}}_j)={\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }^2-\gamma \le 0,\forall i,j\in \mathcal{V}.$$

$$\mathcal{L}(t,\mathbf{x},\mathbf{\lambda },\mathbf{\mu })=f_0(t,\mathbf{x})+\sum _{i=1}^N\left({\mathbf{\lambda }}_i^{\top }{f}_i(t,{\mathbf{x}}_i)+{\mathbf{\mu }}_i^{\top }{g}_i(\mathbf{x})\right),$$

$$\begin{array}{rl}\dot{\mathbf{x}}& =-{\mathcal{L}}_x(t,\mathbf{x},\mathbf{\lambda },\mathbf{\mu })\\ & =-f_{0,x}(t,\mathbf{x})-\sum _{i=1}^N{f}_{i,x}(t,{\mathbf{x}}_i{)}^{\top }{\mathbf{\lambda }}_i-\sum _{i=1}^N\sum _{j\in {\mathcal{N}}_i}{\mu }_{ij}{g}_{ij,x}(\mathbf{x}),\end{array}$$

$$\dot{\mathbf{\lambda }}={\mathcal{L}}_{\lambda }(t,\mathbf{x},\mathbf{\lambda },\mathbf{\mu })=f(\mathbf{x}),$$

$$\dot{\mathbf{\mu }}={\mathcal{L}}_{\mu }(t,\mathbf{x},\mathbf{\lambda },\mathbf{\mu })=g(\mathbf{x}).$$

$$\begin{array}{rl}{\mathcal{L}}^i(t,\mathbf{x},\mathbf{\lambda },\mathbf{\mu })& =f_0^i(t,{\mathbf{x}}_i)+{\mathbf{\lambda }}_i^{\top }{f}_i(t,{\mathbf{x}}_i)\\ & +\sum _{j\in {\mathcal{N}}_i}{\mu }_{ij}\left({\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }^2-\gamma \right),\end{array}$$

$${\Pi }_K\left[\mathbf{y},\mathbf{v}\right]=\underset{\xi \to 0^{+}}{\lim }\frac{P_K(\mathbf{y}+\xi \mathbf{v})-\mathbf{y}}{\xi },$$

$${\dot{\mathbf{x}}}_i={\Pi }_{\mathcal{X}}\left[{\mathbf{x}}_i,-\epsilon {\mathcal{L}}_{x_i}(t,\mathbf{x},\mathbf{\lambda },\mathbf{\mu })\right],$$

$${\dot{\mathbf{\lambda }}}_i={\Pi }_{+}\left[{\mathbf{\lambda }}_i,\epsilon {\mathcal{L}}_{{\lambda }_i}(t,\mathbf{x},\mathbf{\lambda },\mathbf{\mu })\right],$$

$${\dot{\mu }}_{ij}={\Pi }_{+}\left[{\mu }_{ij},\epsilon {\mathcal{L}}_{{\mu }_{ij}}\left(t,\mathbf{x},\mathbf{\lambda },\mathbf{\mu }\right)\right].$$

$$V_{\tilde{\mathbf{x}},\tilde{\mathbf{\lambda }},\tilde{\mathbf{\mu }}}(\mathbf{x},\mathbf{\lambda },\mathbf{\mu })=\frac{1}{2}\left(\Vert \mathbf{x}-\tilde{\mathbf{x}}{\Vert }^2+\Vert \mathbf{\lambda }-\tilde{\mathbf{\lambda }}{\Vert }^2+\Vert \mathbf{\mu }-\tilde{\mathbf{\mu }}{\Vert }^2\right).$$

$$\begin{array}{rl}{\int }_0^T\mathcal{L}(t,\mathbf{x}(t),\tilde{\mathbf{\lambda }},\tilde{\mathbf{\mu }})& -\mathcal{L}(t,\tilde{\mathbf{x}},\mathbf{\lambda }(t),\mathbf{\mu }(t))dt\\ & \le \frac{V_{\tilde{\mathbf{x}},\tilde{\mathbf{\lambda }},\tilde{\mathbf{\mu }}}(\mathbf{x}(0),\mathbf{\lambda }(0),\mathbf{\mu }(0))}{\epsilon },\end{array}$$

$$\begin{array}{rl}& {\int }_0^T\Vert {\mathbf{x}}_i(t)-{\mathbf{x}}_j(t)\Vert dt\\ & \le D\sqrt{(K+\gamma )T+\frac{1}{2\epsilon }\left(1+{\Vert {\mathbf{x}}^{\ast }-\mathbf{x}(0)\Vert }^2\right)},\end{array}$$

$$\mathcal{L}(t,\mathbf{x}(t),\tilde{\mathbf{\lambda }},\tilde{\mathbf{\mu }})=f_0(\mathbf{x}(t))+\left({\Vert {\mathbf{x}}_i(t)-{\mathbf{x}}_j(t)\Vert }^2-\gamma \right).$$

$$\begin{array}{r}{\int }_0^T{f}_0(\mathbf{x}(t))+\left({\Vert {\mathbf{x}}_i(t)-{\mathbf{x}}_j(t)\Vert }^2-\gamma \right)dt\\ -{\int }_0^T\left(f_0({\mathbf{x}}^{\ast })+\mathbf{\lambda }(t{)}^{\top }f({\mathbf{x}}^{\ast })\right)dt\\ -{\int }_0^T\sum _{i=1}^N\sum _{j\in {\mathcal{N}}_i}{\mu }_{ij}(t)\left({\Vert {\mathbf{x}}_i^{\ast }-{\mathbf{x}}_j^{\ast }\Vert }^2-\gamma \right)dt\\ \le \frac{V_{\tilde{\mathbf{x}},\tilde{\mathbf{\lambda }},\tilde{\mathbf{\mu }}}(\mathbf{x}(0),\mathbf{\lambda }(0),\mathbf{\mu }(0))}{\epsilon }.\end{array}$$

Full text

I Introduction

Distributed optimization has applications in several engineering problems such as source localization [1], resource allocation problems in multi cellular communication networks [2], machine learning [3, 4], multi-robot teams [5, 6, 7] and the internet of things [8, 9]. In these problems agents try to optimize collectively a common objective function that is separable in local objectives. In cases where a centralized solution of such problems is acceptable and the objectives and constraints are convex, the problem reduces to solve a classic convex optimization problem. Several methods to do so exist, notably the saddle point algorithm by Arrow and Hurwicz [10], which has the advantage of admitting a distributed implementation [11, 12, 13, 14]. In this framework the most studied classes of problems are when the constraints and the objective function are constant with respect of the time and when their variation is according to a stationary probability distribution. Solutions to the former problem have been established [11, 12, 13, 14]. In the latter, the problem that is studied is that of selecting actions that minimize the expectation of the objective while satisfying the constraints in average. When the problem is unconstrained, centralized and decentralized implementations of stochastic gradient descent converge to such solutions [15, 16].

In this paper we consider online formulations in which the cost and constraints can vary arbitrarily over time, even strategically. In this case, cost minimization can be formulated in the language of regret [17, 18, 19] whereby agents select online actions that result in a cost chosen by nature. The cost functions are revealed to the agents after the action are selected and these values are used to adapt the future strategy. Regret is defined as the difference between the total cost incurred by each agent and the cost of the optimal fixed solution that a clairvoyant agent could select. In that sense, it measures the effect of not knowing the temporal evolution of the cost function and, therefore, it can be used as a performance measure for online operation. Likewise, the fit of a strategy [20, 21] is a vector that integrates over time the constraint violation incurred by each agent. In that sense the fit is a performance loss associated with online operation as opposed to offline clairvoyant operation which can always select an action that satisfies the constraints at all times, if such action exists. It is a remarkable fact that online versions of the Arrow-Hurwicz algorithm for centralized problems achieve regret bounded by a constant and fit bounded by a function that grows at a sublinear rate [20, 21, 22]; this suggests vanishing per play penalties and constraint violation of online plays with respect to clairvoyant agent. Since the fit compares the accumulation of the constraints it is possible to achieve small fit by alternating periods in which the constraints are satisfied with slack and periods in which the constraints are violated. This notion is appropriated for constraints that have a cumulative nature. However, when this is not the case it is possible to work with the notion of saturated fit, where only violations of the constraints are accumulated. A trajectory with small saturated fit is one in which the constraints are violated by a significant amount only for a short period of time.

The problem of distributed online constrained convex optimization has been studied in [23, 24, 25]. Continuous time approaches have advantages in the context of distributed control systems whenever signals are inherently continuous. Moreover, discrete-time approaches can be characterized as the discretization of continuous-time dynamics. The work in [23, 24] considers unconstrained settings where agents minimize a time-varying convex loss, whereas here we consider constrained problems. These have been considered in [25] using a Saddle Point algorithm establishing sublinear bounds on the network disagreement and on the regret achieved by the strategy. The main difference with [25] is that instead of imposing exact consensus among agents, here we allow for small disagreement. This idea has been used for unconstrained problems in[26]. With this modification we are able to establish that the trajectories that arise from the distributed online saddle point dynamics are such that the disagreement of agents, the regret and the fit are bounded by sublinear functions of the time horizon (Section IV). This suggests that our proposed algorithm achieves consensus, feasibility and optimality as the time goes to infinity. Central to the development of this result are the definitions of global fit and regret (Section II). The former is defined as a vector that contains the time integrals of the constraints of all the agents evaluated across the trajectory of a specific agent. Having small global fit means that the agent is able to satisfy the constraints of every other agent, and thus if it is placed in a different position in the network its performance is maintained. Likewise, the global regret is the evaluation of the accumulated global cost of a specific agent’s trajectory with respected to the optimal centralized clairvoyant solution.

To solve the constrained online optimization problem under consideration, we propose an online distributed saddle point algorithm to control the growth of the global fit and regret. Saddle point algorithms update the primal variables – the actions – along the negative of a weighted linear combination of the gradients of each constraints. Since the feasible set of actions is typically represented by the intersection of the sub level sets of the constraints functions, this linear combination pushes the actions towards feasibility. The weights of this linear combination are updated according to the current violation of the constraints. If an action violates a constraint by much, its corresponding weight is increased faster, whereas if a constraint is satisfied its corresponding weight is reduced. We start by showing that if an action that is feasible for all agents and for all times exists, then the network disagreement is bounded by a function that is sublinear with respect to the time horizon. Based on this result we establish sublinear global fit and global regret. We also illustrate our algorithm on a problem involving a team of robots driving through an urban environment to perform real-time texture classification for the purpose of mapping and object recognition. We show that the team of robots succeds in training a common classifier that allows them to distinguish between grass and pavement images even when some of the agents have only observed one of the classes. The rest of this paper is organized as follows. In Section II we formalize the online distributed optimization problem. Later we present the Distributed Online Saddle Point Algorithm (Section III) and we establish that it achieves consensus, feasibility and optimality in Section IV. Other than concluding remarks the paper finishes with numerical examples in section V.

II Constrained Online Learning in Networks

We consider a group of $N$ agents linked by an undirected connected graph ${\mathcal{G}}=\left\{{\mathcal{V}},{\mathcal{E}}\right\}$ where ${\mathcal{V}}=\{1,\ldots,N\}$ is a set of nodes and ${\mathcal{E}}$ is a set of edges so that $(i,j)\in{\mathcal{E}}$ means that $i$ and $j$ are connected to each other. The set ${\mathcal{N}}_{i}:=\{j:(i,j)\in{\mathcal{E}}\}$ contains all nodes that are connected to $i$ and is called the neighborhood of $i$. Note that since the graph is undirected, node $j$ is in the neighborhood of $i$ if and only if node $i$ is in the neighborhood of $j$.

We are interested in situations where the agents in ${\mathcal{G}}$ have access to arbitrarily time varying local constraints and local objective functions and continuously select actions that are good not only for their local constraints and costs but for the local constraints and costs of other agents. To explain this formally, let $t\in{\mathbb{R}}^{+}$ be a continuous time index, $f_{i}(t,\cdot):{\mathbb{R}}^{n}\to{\mathbb{R}}^{m_{i}}$ be a set of $m_{i}$ convex constraints at agent $i$ and $f_{0i}(t,\cdot):{\mathbb{R}}^{n}\to{\mathbb{R}}$ be a local convex cost incurred at node $i$. A local goal of node $i$ is to select an action ${\mathbf{x}}_{i}\in{\mathcal{X}}\subseteq{\mathbb{R}}^{n}$ that satisfies local constraints $f_{i}(t,{\mathbf{x}}_{i})\preceq 0$ across a time interval $[0,T]$ while minimizing the cost $f_{0i}(t,{\mathbf{x}}_{i})$ integrated over the same interval. This is tantamount to defining the locally optimal solution ${\mathbf{x}}_{i}^{\ell}$ over the interval $[0,T]$ as

[TABLE]

Problem (II) models situations in which each of the agents is acting independently since the optimal action of $i$ depends on its local cost and constraints, and not on those of other agents. Instead, here we are interested in situations where the actions of agents are coordinated, so that an action ${\mathbf{x}}_{i}$ of agent $i$ can affect the costs and constraints of other agents. This results in a global formulation in which the optimal action of each agent ${\mathbf{x}}_{i}^{*}$ is defined as the one that satisfies the constraints of all agents and minimizes the integral of the sum cost,

[TABLE]

We say that the problem in (II) is global because the action ${\mathbf{x}}_{i}$ is evaluated at the constraint and costs of all nodes. This readily implies that ${\mathbf{x}}_{i}^{*}={\mathbf{x}}_{j}^{*}$ and that there is a single global action that is optimal for all nodes. For future reference we define the sum cost function $f_{0}(t,{\mathbf{x}}_{i}):=\sum_{j=1}^{N}f_{0j}(t,{\mathbf{x}}_{i})$ and the aggregate constraint $f(t,{\mathbf{x}}_{i}):=[f_{1}(t,{\mathbf{x}}_{i});\ldots;f_{N}(t,{\mathbf{x}}_{i})]$. With this notation, the problem in (II) can be rewritten as

[TABLE]

While (II) models agents acting in isolation, (II) and (II) model agents acting in concert. The latter situation arises when local functions are related to a common variable. E.g., the costs and constraints can represent local observations of a parameter to be estimated [27] or local observations and costs of a plant to be controlled [28]. Problems having the form of (II) also arise in large scale optimization where costs and constraints are not acquired locally but are distributed over several servers to reduce computation and storage [29].

If the functions $f_{0i}(t,\cdot)$ and $f_{i}(t,\cdot)$ are available for all times $t\in[0,T]$, solving (II) reduces to solving a distributed convex optimization problem for which a number of standard algorithms are applicable; see e.g., [30, 31]. In this paper we consider problems in which the constraints $f_{i}(t,\cdot)$ and costs $f_{0}(t,\cdot)$ are arbitrary and observed causally and locally by node $i$. In this setting it makes sense to consider time varying strategies ${\mathbf{x}}_{i}(t)$ that adapt the action of agent $i$ to the information that is revealed at time $t$. In this context the optimal argument in (II) is a clairvoyant action that would be chosen when agents have knowledge of the future evolution of the system at time $t=0$. The appropriate figures of merit in this case are the notions of regret [18, 32, 33] and fit [20] that we generalize to network settings in the following section. These quantities compare the performance of the online distributed operation with the offline centralized solution of (II).

Before proceeding with definitions of network regret and fit, notice that for the definition in (II) to be valid the function $f_{0}(t,{\mathbf{x}})$ has to be integrable with respect to the time variable $t$. In subsequent definitions and analyses we further require the network to be connected and the constraints $f_{i}$ for all $i\in{\mathcal{V}}$, to be integrable, convex and Lipschitz continuous with respect to ${\mathbf{x}}$ for all times $t$. We formally state these assumptions next.

Assumption 1**.**

The network is connected with diameter $D$, i.e., the shortest distance between the two most distant nodes in the network is $D$.

Assumption 2**.**

Let ${\mathcal{X}}$ be a compact convex set and the functions $f_{0i}(t,{\mathbf{x}})$ and $f_{i}(t,{\mathbf{x}})$ be integrable with respect to $t$ and convex with respect to ${\mathbf{x}}\in{\mathcal{X}}$ for all $t\in[0,T]$. We further assume that cost and constraints are Lipschitz continuous over ${\mathcal{X}}$ with respective constants $L_{0}>0$ and $L_{f}>0$. I.e., for any ${\mathbf{x}},{\mathbf{y}}\in{\mathcal{X}}$ and all $t\in[0,T]$ the cost functions satisfy

[TABLE]

and the constraint functions satisfy

[TABLE]

where $f_{kj}(t,\cdot)$ denotes the $j$th component of the vector valued constraint function $f_{k}(t,\cdot)$.

We remark that integrability with respect to $t$ is a weak condition. We do not require differentiability, not even continuity. This entails a fundamental difference with time varying optimization problems that strive to track a time varying optimal argument under the assumption of smooth time varying costs and constraints [27, 34, 35]. The goal here is to design an algorithm that can adapt to unexpected changes in the system, including, indeed, most importantly, to those that arise because of discontinuities in the cost and constraint functions. Another requirement for ${\mathbf{x}}_{i}^{*}$ to be well defined is existence of an action ${\mathbf{x}}^{\dagger}\in{\mathcal{X}}$ that satisfies the constraints at all times and all nodes as we formally state next.

Assumption 3**.**

There exists an action ${\mathbf{x}}^{\dagger}\in{\mathcal{X}}$ that satisfies the constraints of all agents for all times $t\in[0,T]$,

[TABLE]

We say that ${\mathcal{X}}^{\dagger}:=\left\{{\mathbf{x}}^{\dagger}\in{\mathcal{X}}:f_{i}(t,{\mathbf{x}}^{\dagger})\preceq 0,\forall i\text{\ and\ }t\in[0,T].\right\}$ is the set of feasible actions.

We require as well that minimum of the objective function does not become progressively smaller with time so that a uniform bound $K$ holds for all times $t\in[0,T]$. The existence of the bound in (7) is a mild requierement. Since the function $f_{0}(t,{\mathbf{x}})$ is convex, for any time $t$ it is lower bounded for compact set of actions ${\mathcal{X}}$. The only restriction imposed is that $\min_{{\mathbf{x}}\in{\mathcal{X}}^{N}}f_{0}(t,{\mathbf{x}})$ does not become progressively smaller with time so that a uniform bound $K$ holds for all times $t\in[0,T]$.

Assumption 4**.**

There exists $K>0$ independent of the time horizon $T$ such that for all $t\in[0,T]$ it holds that

[TABLE]

where ${\mathbf{x}}^{*}$ is the solution to the problem (II).

II-A Network Regret and Network Fit

To evaluate the cost performance of such trajectories we define the notions of network regret and network fit. Begin then by considering a trajectory ${\mathbf{x}}_{i}(t)$ chosen by agent $i$ and the total accumulated cost $\int_{0}^{T}f_{0j}(t,{\mathbf{x}}_{i}(t))dt$ that this trajectory incurs for a possibly different agent $j$. We define the regret ${\mathcal{R}}_{Tj}^{i}$ as the difference of this accumulated cost relative to the corresponding cost that would be incurred by the optimal trajectory ${\mathbf{x}}_{i}^{*}$ of (II),

[TABLE]

Likewise, we consider the accumulation $\int_{0}^{T}f_{j}(t,{\mathbf{x}}_{i}(t))dt$ of constraint of agent $j$ incurred by the trajectory of agent $i$. The fit ${\mathcal{F}}_{Tj}^{i}$ is defined as the comparison of this constraint accumulation relative to the corresponding constraint accumulation of the optimal trajectory ${\mathbf{x}}_{i}^{*}$

[TABLE]

The action ${\mathbf{x}}_{i}^{*}$ can be considered as an offline reference that would be chosen by an entity that is clairvoyant, because it observes the future, and omniscient, because it observes the costs and constraints of all nodes. Our objective is to consider trajectories that are chosen online by agents that are causal, because they observe the past, and local, because they observe their local costs and exchange information with neighboring nodes only. In this context regret and fit can be interpreted as performance losses associated with online causal and local operation as opposed to offline clairvoyant and omniscient operation. If ${\mathcal{F}}_{Tj}^{i}$ is positive we are in a situation in which, had the constraints of all agents be known beforehand, we could have selected an action ${\mathbf{x}}^{\dagger}$ to satisfy all constraints. Due to its cumulative nature, it is possible to achieve small (local) fit by alternating between actions for which the constraints take positive and negative values. This is an appropriate model for quantities that can be stored – such as energy budgets enforced through average power constrains. However, in other settings this formulation can be a limitation. This drawback can be overcome by defining the saturated fit in which constraints slacks are saturated to a small negative constant. We discuss this in Section IV-A.

The fit measures how far the trajectory ${\mathbf{x}}(t)$ is from achieving that goal. Analogously, if the regret ${\mathcal{R}}_{Tj}^{i}$ is large we are in a situation in which prior knowledge of the objective functions and constraints would had resulted in the selection of an strategy ${\mathbf{x}}^{*}$ that achieves much better performance than the one achieved by ${\mathbf{x}}_{i}(t)$. In that sense ${\mathcal{R}}_{Tj}^{i}$ indicates how much we regret not having had that information a priori.

A good learning strategy is one that achieves small regret and fit as that would be an indication that the trajectory ${\mathbf{x}}(t)$ approaches ${\mathbf{x}}^{*}$. Notice however that since the objective function and the constraints are integrated over a time horizon $T$, it is natural to expect the cost and constraints to grow linearly with $T$. Thus, having regret and fit that grow at a sublinear rate is sufficient indication of a good learning strategy. This intuition motivates the following definitions of feasible and optimal trajectories.

Definition 1**.**

We define an environment as a set of constraints $f_{j}:{\mathbb{R}}\times{\mathbb{R}}^{n}\to{\mathbb{R}}^{m_{j}}$ and costs $f_{0j}:{\mathbb{R}}\times{\mathbb{R}}^{n}\to{\mathbb{R}}$ for all $j\in{\mathcal{V}}$. For a trajectory ${\mathbf{x}}_{i}(t)$ we consider the regret and fit definitions in (8) and (9) and further define the sum regret ${\mathcal{R}}_{T}^{i}:=\sum_{j\in{\mathcal{V}}}{\mathcal{R}}_{Tj}^{i}$ and the network wide fit ${\mathcal{F}}_{T}^{j}=[{\mathcal{F}}_{T1}^{j\top},\ldots,{\mathcal{F}}_{TN}^{j\top}]^{\top}$. We say that:

• Feasibility.

The trajectories are feasible in the environment if all the local fits ${\mathcal{F}}_{T}^{i}$ with $i\in{\mathcal{V}}$ grow sublinearly with $T$. I.e., there exist a function $h(T)$ with $\limsup_{T\to\infty}h(T)/T=0$ and a constant vector $C_{f}$ such that for all times $T$ it holds,

[TABLE]

• Optimality.

The trajectories are optimal in the environment if all regrets ${\mathcal{R}}_{T}^{i}$ grow sublinearlly for all $i\in{\mathcal{V}}$ and $T$. I.e. there exist a function $h(T)$ with $\limsup_{T\to\infty}h(T)/T=0$ and a constant $C$ such that for all times $T$ it holds,

[TABLE]

In the next section we develop the details of a distributed and online version of the Arrow-Hurwicz algorithm, such that its generated trajectories are feasible and optimal in the sense of Definition 1. The latter is formally stated and proved in Section IV along with an intermediate result that claims that the disagreement across agents is sublinear with respect to the time horizon, hence suggesting consensus. Before doing so, we make a pertinent remark highligthing the differences between the results in this work and those achieved for the centralized algorithm.

Remark 1**.**

The centralized version of the problem here considered (II), can be solved – when affordable – by an online saddle-point algorithm [20]. The trajectories that arise from such algorithm achieve (i) regret bounded by a constant independent of the time horizon $T$ for the unconstrained problem; (ii) fit bounded by a constant independent of the time horizon $T$ if the objective function is constant with respect to the action, i.e., feasibility problems and (iii) regret bounded by a constant independent of the time horizon $T$ and fit bounded by a sublinear function of $T$ otherwise. By considering the distributed version of the previous algorithm we cannot establish regret or fit bounded by constants independent of $T$ because of the disagreement across agents.

III Distributed Online Saddle point

The problem defined in (II) is a centralized optimization problem in which all agents should select the same action. Since each agent $i\in{\mathcal{V}}$ has access only to the local cost and constraints, a more natural representation of the problem (II) is one where each agent selects a local decision vector ${\mathbf{x}}_{i}\in\mathbb{R}^{n}$. Nodes then try to achieve the minimum of their local objective functions $f_{0i}(t,{\mathbf{x}}_{i})$ while satisfying the local constraints $f_{i}(t,{\mathbf{x}}_{i})\preceq 0$ and keeping their variables equal to the variables ${\mathbf{x}}_{j}$ of neighboring nodes $j\in{\mathcal{N}}_{i}$. By defining ${\mathbf{x}}=[{\mathbf{x}}_{1}^{\top},\ldots,{\mathbf{x}}_{N}^{\top}]^{\top}$, this alternative formulation can be written as

[TABLE]

Since the network is assumed to be connected (cf., Assumption 1), the constraints ${\mathbf{x}}_{i}={\mathbf{x}}_{j}$ for all $i$ and $j\in{\mathcal{N}}_{i}$ imply that (II) and (III) are equivalent. The previous problem can solved in a distributed manner with a variety of methods, one of which is he saddle point algorithm of Arrow and Hurwicz [10]. In this work we aim to extend the saddle point algorithm to control the growth of regret and fit. In doing so it is convenient to relax the consensus constraints ${\mathbf{x}}_{i}={\mathbf{x}}_{j}$ in (III) to allow for some controlled disagreement. We accomplish this by defining the set of constraints

[TABLE]

where $\gamma$ is a positive constant limiting how much constraint violation is allowed. Notice that the parameter $\gamma$ could be set to be arbitrarily small. The advantage of using a controlled disagreement is that it allows for agents to achieve a good global performance without damaging excessively the local performance, which in some applications might be important as well. By allowing larger values of $\gamma$, we allow more disagreement and therefore we prioritize the local performance, whereas by making $\gamma$ closer to zero the goal is set in the centralized performance. The same relaxation is considered in [26] in the case of unconstrained distributed optimization. The proximity constraints allows to write the problem of interest in the following form

[TABLE]

For the above online optimization problem we can construct the following time varying Lagrangian

[TABLE]

where $\bm{\lambda}_{i}\in\mathbb{R}_{+}^{m_{i}}$ for $i=1\ldots N$ and $\bm{\mu}_{i}\in\mathbb{R}_{+}^{|{\mathcal{N}}_{i}|}$ for $i=1\ldots N$ are the Lagrange multipliers and where $g_{i}({\mathbf{x}})\in\mathbb{R}^{|{\mathcal{N}}_{i}|}$ is the vector with components $g_{ij}({\mathbf{x}}_{i},{\mathbf{x}}_{j})$ for all $j\in{\mathcal{N}}_{i}$. Saddle point methods rely on the fact that for a constrained convex optimization problem, a pair is a primal-dual solution if and only if it is a saddle point of the Lagrangian associated with the problem, see e.g. [36]. This is the case in problem (III) since $f_{0}(\cdot,{\mathbf{x}})$, $f(\cdot,{\mathbf{x}})$ and $g(\cdot,{\mathbf{x}})$ are convex (c.f. Assumption 2). In addition, because $\bm{\lambda},\bm{\mu}\succeq 0$, the Lagrangian is convex with respect to ${\mathbf{x}}$ and therefore the subgradient with respect to ${\mathbf{x}}$ exists for all time $t\geq 0$, let us denote it by ${\mathcal{L}}_{x}(t,{\mathbf{x}},\bm{\lambda},\bm{\mu})$. The Lagrangian is linear with respect to $\bm{\lambda}$ and $\bm{\mu}$ and therefore its partial derivatives with respect to these variables exist. Let us denote them by ${\mathcal{L}}_{\lambda}(t,{\mathbf{x}},\bm{\lambda},\bm{\mu})$ and ${\mathcal{L}}_{\mu}(t,{\mathbf{x}},\bm{\lambda},\bm{\mu})$ respectively. The actions ${\mathbf{x}}$ are updated – as in the classic Arrow-Hurwicz algorithm – by following the negative subgradient of the Lagrangian with respect to ${\mathbf{x}}$

[TABLE]

where ${\mathcal{N}}_{i}$ is the set of neighbors of node $i$. The primal update interprets the constraints as a potentials with corresponding weights $\bm{\lambda}$ and $\bm{\mu}$ and descends along a linear combination of the gradients of said potentials. The multipliers are then updated by following the subgradient of the Lagrangian with respect to them

[TABLE]

The intuition behind the latter update is that if a constraint is violated, for instance $f_{1,1}({\mathbf{x}})>0$ the corresponding multiplier, $\bm{\lambda}_{1,1}$ will be increased, thus augmenting the relative weight of this potential in the linear combination in (16). Which in turn pushes the action towards satisfying said constraint. On the other hand, if the constraint is satisfied, the weight of that potential will be reduced, thus making the direction of the gradient of the function less important in the weighted linear combination. Observe that the multipliers need to remain positive at all time to ensure the convexity of the Lagrangian with respect to ${\mathbf{x}}$, yet if a multiplier takes the value zero and its corresponding constraint is satisfied, the previous update turns the multiplier negative. To avoid this issue, we will require a projection over the positive orthant. We formalize this idea next, after making the observation that the update (16)–(17) is indeed distributed. To see this, write the Lagrangian as a sum of the following local Lagrangians

[TABLE]

where to compute each local Lagrangian, agent $i$ needs only information regarding its variables and those of its neighbors. Then, each agent can compute locally the gradient of the Lagrangian with respect to its local variable ${\mathbf{x}}_{i}$ and perform the update described in (17)–(16), with the caveat that to ensure that the multipliers are always positive we need to consider a projected dynamical system. We formalize this idea next.

Definition 2** **(Projection of a vector at a point).

Let $K\subset\mathbb{R}^{n}$ be a compact convex set. Then, for any ${\mathbf{y}}\in K$ and ${\mathbf{v}}\in\mathbb{R}^{n}$, we defined the projection of ${\mathbf{v}}$ over the set $K$ at the point ${\mathbf{y}}$ as

[TABLE]

where the standard projection $P_{K}({\mathbf{z}})=\operatornamewithlimits{argmin}_{{\mathbf{y}}\in K}\|{\mathbf{y}}-{\mathbf{z}}\|^{2}$ is always well defined because $K$ is convex.

The intuition behind the projection is that, if the point ${\mathbf{y}}$ is in the interior of the set $K$ then the projection of the vector ${\mathbf{v}}$ is the vector itself. In cases where ${\mathbf{y}}$ is in the boundary of the set $K$, the projection of ${\mathbf{v}}$ is its component tangental to the boundary of $K$. With this definition at hand, and by defining the gain of the controller to be $\varepsilon>0$ we define the distributed online saddle point controller as follows. Each agent updates its action by following the negative subgradient of the Lagrangian with respect to its local copy of the action ${\mathbf{x}}_{i}$

[TABLE]

Likewise, the multipliers $\bm{\lambda}_{i}$ and $\bm{\mu}_{i}$ are updated by ascending along the direction of the gradient of the Lagrangian with respect to $\bm{\lambda}$ and $\bm{\mu}$ respectively, i.e.,

[TABLE]

The three gradients can be computed in a distributed fashion since they only depend on each agent’s own variables and those of their neighbors. In [20] it was shown that in the centralized case, a saddle point algorithm such as the one described by (20) and (21) achieves feasible and strongly optimal trajectories, i.e., fit bounded by a sublinear function of the time horizon and regret bounded by function that is constant with respect to the time horizon. In this work we show that the distributed version of said algorithm (c.f. (20) and (21)) achieves feasible and optimal trajectories in the sense of Definition 1. Moreover, the network disagreement is bounded by a function that is sublinear with respect to the time horizon. These results are the subject of the next section.

IV Feasible and Optimal trajectories

Let us consider an energy-like function which will be used in subsequent analysis. Let $\tilde{{\mathbf{x}}}\in{\mathcal{X}}^{N}$, $\tilde{\bm{\lambda}}\in\mathbb{R}_{+}^{\sum_{i}m_{i}}$, $\tilde{\bm{\mu}}\in\mathbb{R}_{+}^{\sum_{i}|{\mathcal{N}}_{i}|}$, where we denote by $|{\mathcal{N}}_{i}|$ the cardinality of the set of neighbors of node $i$, and define the function

[TABLE]

By considering the time derivative of the previous function along the dynamics (20)–(21) we establish that the integral of the difference of the Lagrangian evaluated at $({\mathbf{x}}(t),\tilde{\bm{\lambda}},\tilde{\bm{\mu}})$ and the Lagrangian evaluated at $(\tilde{{\mathbf{x}}},\bm{\lambda}(t),\bm{\mu}(t))$ is bounded by a constant independent of the time horizon $T$. The following lemma – key to establish that the saddle point dynamics yield feasible and optimal trajectories – formalizes this result.

Lemma 1**.**

Let Assumptions 1–3 hold. Then for any $T\geq 0$ the solutions of the dynamical system (20)–(21) satisfy

[TABLE]

where ${V}_{\tilde{{\mathbf{x}}},\tilde{\bm{\lambda}},\tilde{\bm{\mu}}}({\mathbf{x}}(0),\bm{\lambda}(0),\bm{\mu}(0))$ is the energy-function defined in (22) eavluated values for the actions and multipliers and time zero and $\tilde{{\mathbf{x}}},\tilde{\bm{\lambda}},\tilde{\bm{\mu}}$ are arbitrary.

Proof.

See Appendix A-A. ∎

By analyzing the expression (23) for different choices of $\tilde{{\mathbf{x}}},\tilde{\bm{\lambda}}$ and $\tilde{\bm{\mu}}$ it is possible to establish that the saddle point dynamics (20)–(21) yields sublinear network disagreement for all $T>0$. We formalize this result in Proposition 1.

Proposition 1** **(Sublinear Network Disagreement).

Let Assumptions 1–4 hold. Then for any $T\geq 0$ the solutions of the dynamical system (20)–(21) are such that the network disagreement is sublinear with respect to $T$. In particular for $\bm{\lambda}(0)=0$ and $\bm{\mu}(0)=0$, for any $i,j\in{\mathcal{V}}$ we have that

[TABLE]

where $D$ is the network diameter defined in Assumption 1.

Proof.

Let us consider the expression (23) with $\tilde{{\mathbf{x}}}={\mathbf{x}}^{*}$, the solution of the problem (III), $\tilde{\bm{\lambda}}=\bm{0}$, and $\tilde{{\mu}}_{ij}=1$ for some $i\in{\mathcal{V}}$ and $j\in{\mathcal{N}}_{i}$ and $\tilde{\bm{\mu}}_{ik}={0}$ for all $k\neq j$ and $\tilde{\bm{\mu}_{l}}=\bm{0}$ for all $l\neq i$. For this selection of variables the Lagrangian evaluated at $(t,{\mathbf{x}}(t),\tilde{\bm{\lambda}},\tilde{\bm{\mu}})$ yields

[TABLE]

Applying Lemma 1 for this particular choice of $\tilde{{\mathbf{x}}},\tilde{\bm{\lambda}}$ and $\tilde{\bm{\mu}}$, (23) reduces to

[TABLE]

Since ${\mathbf{x}}^{*}$ is the solution to (III) it holds that ${\mathbf{x}}^{*}_{i}={\mathbf{x}}_{j}^{*}$ and that $f({\mathbf{x}}^{*})\preceq\bm{0}$. Since $\bm{\lambda}(t)$ and $\bm{\mu}(t)$ are always in the positive orthant, due to the projection in their update (c.f. (21a) and (21b)), we have that $\bm{\lambda}(t)^{\top}f({\mathbf{x}}^{*})\leq 0$ and that ${\mu}_{ij}(t)\left(\left\|{\mathbf{x}}_{i}^{*}-{\mathbf{x}}_{j}^{*}\right\|^{2}-\gamma\right)=-\mu_{ij}(t)\gamma\leq 0$ for all $t\in[0,T]$. These observations imply that

[TABLE]

From the definition of a minimum and Assumption 4 it follows that

[TABLE]

Substituting (28) into (27) yields

[TABLE]

Rearranging the terms in the previous expression it holds that

[TABLE]

Because the square function is convex, by virtue of Jensen’s inequality we have that

[TABLE]

The previous inequality provides a lower bound for (30), hence we have that

[TABLE]

By taking the square root of the previous inequality we observe that the disagreement among neighbors is sublinear. In particular, by evaluating ${V}_{\tilde{{\mathbf{x}}},\tilde{\bm{\lambda}},\tilde{\bm{\mu}}}({\mathbf{x}}(0),\bm{\lambda}(0),\bm{\mu}(0))$ for the selection of $\tilde{{\mathbf{x}}}$, $\tilde{\bm{\lambda}}$ and $\tilde{\bm{\mu}}$ done at the begining of the proof and for the initial conditions $\bm{\lambda}(0)=\bm{0}$, $\bm{\mu}(0)=\bm{0}$ yields

[TABLE]

The latter establishes a sublinear disagreement among one hop neighbors on the network. To show that the result holds for every pair of nodes use the triangular inequality and the fact that the diameter of the network is $D$ (Assumption 1). ∎

The sublinear network disagreement that the previous proposition establishes, along with the result of Lemma 1, allows us to prove that the local Fit and Regret are bounded by sublinear functions with respect to the time horizon $T$. The latter means that the trajectories that arise from the Distributed Online Saddle Point Dynamics (20)–(21) are feasible and optimal in the sense of definition 1. We formalize these results in theorems 1 and 2 respectively.

Theorem 1** **(Feasibility).

Let Assumptions 1–4 hold. Then for any $T\geq 0$ the solutions of the dynamical system (20)–(21), with $\varepsilon>1/2$, are such that the $k$-th component of the local fit ${\mathcal{F}}_{Tj}^{i}$ for any $i,j\in{\mathcal{V}}$ is bounded by $\left({\mathcal{F}}_{Tj}^{i}\right)_{k}\leq O(\sqrt{T})$.

Proof.

We evaluate the expression (23) for the particular choice of $\tilde{{\mathbf{x}}}={\mathbf{x}}^{*}$ and $\tilde{\bm{\mu}}=0$

[TABLE]

By virtue of Assumption 4 and the bound derived in (28), we can lower bound the difference of the integrals of the objective functions by

[TABLE]

Substituting the previous bound in (34) yields

[TABLE]

Observe that, since ${\mathbf{x}}^{*}$ is the solution of the decentralized problem (III), we have that $f_{i}(t,{{\mathbf{x}}}^{*}_{i})\preceq\bm{0}$. Moreover, because the Lagrange Multipliers $\bm{\lambda}_{i}(t)$ are in the positive orthant (cf., (21a)) the product $-\bm{\lambda}_{i}^{\top}(t)f_{i}(t,{{\mathbf{x}}}_{i}^{*})$ is always positive. Likewise, the product $\bm{\mu}_{ij}(t)\left(\gamma-\left\|{\mathbf{x}}_{i}^{*}-{\mathbf{x}}_{j}^{*}\right\|^{2}\right)=\bm{\mu}_{ij}(t)\gamma\geq 0$, for all $t\geq 0$. With these considerations the following bound holds

[TABLE]

Denote by $\left[\cdot\right]^{+}$ the projection over the positive orthant. Then, for a particular $i\in{\mathcal{V}}$ choose $\tilde{\bm{\lambda}}_{i}=\left[{\mathcal{F}}_{Ti}^{i}\right]^{+}$ and $\tilde{\bm{\lambda}}_{j}=\bm{0}$ for all $j\in{\mathcal{V}}$ such that $j\neq i$. Then, (37) reduces to

[TABLE]

Without loss of generality assume $\bm{\lambda}(0)=0$ and $\bm{\mu}(0)=0$. Then, the right hand side of the previous expression reduces to

[TABLE]

and (38) can be written as

[TABLE]

Then, for any $\varepsilon>1/2$, the previous expression yields

[TABLE]

Taking the square root of both sides of the previous inequality shows that the norm of the projection of the fit is bounded by a sublinear function. In particular we have that each component $k=1\ldots m_{i}$ of ${\mathcal{F}}_{Ti}^{i}$ is also upper bounded by the a sublinear function that grows as $O(\sqrt{T})$. We are left to show that for any $j\neq i$ we have fit that is bounded by a function of the order of $\sqrt{T}$. The latter is a consequence of the Liptchitz continuity and the sublinear network disagreement as we show next. Add and subtract $f_{i,k}(t,{\mathbf{x}}_{i}(t))$ to the definition of the local fit to write

[TABLE]

The first term on the right hand side of the previous expression is, by definition, $\left({\mathcal{F}}_{Ti}^{i}\right)_{k}$. This term is bounded by a function of the order as $\sqrt{T}$ as previously shown. On the other hand, the second integral can be bounded using the Liptchitz continuity of $f_{i,k}(t,{\mathbf{x}})$ (c.f. Assumption 2) by

[TABLE]

Then, the result of Proposition 1 completes the proof. ∎

The previous theorem establishes that the local fit achieved by a system that follows saddle point dynamics (20)–(21) is bounded by a function whose rate of growth is sublinear, thus suggesting vanishing penalties. However, as discussed previously this can be achieved by solutions that oscillate, i.e., trajectories that violate the constraints at some times and that satisfy them with slack at other periods. Since this is not desirable in some applications, we overcome this limitation by showing that the saturated global fit has the same property. We address this in Section IV-A. The fact that the fit grows sublinearly is equivalent to achieving trajectories that are feasible in the sense of Definition 1. In the next theorem we establish that the local regrets are also bounded and thus, that the saddle point dynamics give origin to trajectories that are also optimal.

Theorem 2** **(Optimality).

Let Assumptions 1–4 hold. Then for any $T\geq 0$ the solutions of the dynamical system (20),(21a) and (21b) are such that the local regret ${\mathcal{R}}_{T}^{i}$ for any $i\in{\mathcal{V}}$ is bounded by a function of $O(\sqrt{T})$. In particular, for $\bm{\lambda}(0)=0$ and $\bm{\mu}(0)=0$ we have that

[TABLE]

Proof.

Let us consider the local regret of agent $j$, with $j\in{\mathcal{V}}$, defined in (11)

[TABLE]

Add and subtract $\sum_{i=1,i\neq j}^{N}f_{0}^{i}(t,{\mathbf{x}}_{i}(t))$ to the previous equation and rewrite the previous expression as

[TABLE]

From Lemma 1 and by choosing $\tilde{{\mathbf{x}}}={\mathbf{x}}^{*},\tilde{\bm{\lambda}}=0$, $\tilde{\bm{\mu}}=0$ it follows that

[TABLE]

As it was previously argued, because ${\mathbf{x}}^{*}$ is the solution of (III) it follows that $f_{i}({\mathbf{x}}^{*})\preceq 0$ and that ${\mathbf{x}}_{i}^{*}={\mathbf{x}}_{j}^{*}$. Combining these observations with the fact that the multipliers always lie in the positive orthants due to the projections introduced in their updates (21), we have that $-\bm{\lambda}_{i}(t)^{\top}f_{i}(t,{\mathbf{x}}_{i}^{*})\geq 0$ and $\bm{\mu}_{ij}(t)\left(\left\|{\mathbf{x}}_{i}^{*}-{\mathbf{x}}_{j}^{*}\right\|^{2}-\gamma\right)\geq 0$. Thus it follows that

[TABLE]

Notice that the difference in the left hand side of the previous expression is equal to the first term in (46). Thus, the local regret in (46) can be upper bounded by

[TABLE]

To bound the second term in the previous expression, use the Lipschitz continuity of the objective function (c.f. Assumption 2)

[TABLE]

The latter is bounded by a function of the order of $\sqrt{T}$ as a consequence of the sublinear network disagreement established in Proposition 1. Since $V_{{\mathbf{x}}^{*},\bm{0},\bm{0}}\left({\mathbf{x}}(0),\bm{\lambda}(0),\bm{\mu}(0)\right)/\varepsilon$ is a constant, it holds that (49) is upper bounded by a function of the order of $\sqrt{T}$. To complete the proof of the bound in (44) it suffices to replace in the previous expression the network disagreement by the result of Proposition 1. ∎

We have established that agents operating distributedly achieve sublinear network disagreement, fit and regret. In the next section we discuss the case where the constraint is lower bounded and therefore it cannot be satisfied with slack. This prevents the fit to be bounded due to trajectories that alternate between feasibility and large periods of infeasibility.

IV-A Saturated Fit

We start by definig a saturated function to prevent the constraint to take negative values smaller than a given threshold. Formally, let $\delta>0$ and define the function $\tilde{f}_{\delta}(t,{\mathbf{x}})=\max\left\{f(t,{\mathbf{x}}),-\delta\right\}$. Then we can define the notion of saturated local fit as

[TABLE]

By taking small values of $\delta$ we can arbitrarily reduce the negative portion of the fit. Ideally one would like to set $\delta=0$.We next establish that the sublinear bound for the fit established in Theorem 1 holds as well for the saturated fit.

Corollary 1**.**

Let Assumptions 1–4 hold. Then for any $T\geq 0$ the solutions of the dynamical system (20)–(21), with $\varepsilon>1/2$, are such that the $k$-th component of the saturated fit $\tilde{{\mathcal{F}}}_{\delta,Tj}^{i}$ for any $i,j\in{\mathcal{V}}$ is bounded by $\left(\tilde{{\mathcal{F}}}_{\delta,Tj}^{i}\right)_{k}\leq O(\sqrt{T})$.

Proof.

Recall that $\tilde{f}_{\delta,i}(t,{\mathbf{x}})=\max\left\{f_{i}(t,{\mathbf{x}}),-\delta\right\}$. Because $\tilde{f}_{\delta,i}(t,{\mathbf{x}})$ is the point-wise maximum of two convex functions in ${\mathbf{x}}$ it is also convex. Thus, it satisfies the hypotheses of Theorem 1 and it follows that the local saturated fit is also bounded by a function of $O(\sqrt{T})$. ∎

In the next section we present numerical examples that support the theoretical conclusions.Before doing so, we discuss the bound on the network disagreement established in Proposition 1.

Remark 2**.**

In Proposition 1 we showed that the network disagreement depends on $T$ as $O(\sqrt{T})$. However, the bound achieved is a direct consequence of the choice of the relaxation of the consensus constraint. This choice being arbitrary, it is possible to chose different relaxations to obtain different bounds on the network disagreement. If one desires to bound this quantity by a sublinear function $h(T)$, it suffices to impose the constraint $g(\left\|{\mathbf{x}}_{i}-{\mathbf{x}}_{j}\right\|)-\gamma<0$ for any $g(y)=h^{-1}(y)$ as long as $g(y)$ is a convex function. The latter holds because the main component of the proof of the proposition is Jensen’s inequality.

V Numerical Examples

In this section we consider a team of $N$ robots tasked with classifying in real time and in a distributed manner the different objects and terrains that compose the environment in which they are deployed. This problem, has been studied in [5], although the method presented here is different. Each robot has only access to information about the environment based on the path it has traversed and the images gathered. Therefore, its local information may not be enough to achieve the task of classification since the information gathered may omit regions of the feature space that are crucial. See for instance Figure 1 where we depict random trajectories of twenty agents driving around an intersection. When the agent is on the pavement i.e., the absolute value of its horizontal or vertical coordinate is less than five, then it observes pavement images. On the other hand, outside this region it observes grass. As it can be observed in that figure only some of the agents visit both regions and the interest is that the whole team can learn a common classifier. The advantage of learning such classifier is that a robot can identify if it is on grass even if it has not seen grass in the training process. In particular we consider a problem in which each robot receives features ${\mathbf{z}}_{i}(t)\in\mathbb{R}^{n}$ from the scene and corresponding labels $y_{i}(t)\in\left\{-1,1\right\}$ depending on whether the terrain is grass or pavement. The details of the feature extraction from image data is provided in Section V-A. The common objective of the agents can be formulated as training a common linear classifier ${\mathbf{x}}\in\mathbb{R}^{n}$ that minimizes a loss function. The loss function is such that its value is small when the classification is accurate and it takes large values in the opposite case. In particular for this problem we consider logistic regression

[TABLE]

The classifier is designed so that the prediction is defined by the sign of the inner product between the classifier ${\mathbf{x}}$ and the feature vector ${\mathbf{z}}_{i}(t)$ observed by robot $i$ at time $t$. This is, the predicted label is given by $\hat{y}_{i}(t)=\text{sign}({\mathbf{x}}^{\top}{\mathbf{z}}_{i}(t))$. Notice that if the prediction is correct, both $\hat{y}_{i}(t)$ and $y_{i}(t)$ have the same sign and thus, the exponential in (52) takes a small value. Which in turn results in $f_{i}(t,{\mathbf{x}})$ being small. On the other hand, if the classification is incorrect, the sign of the exponential is positive, which results in a large value of $f_{i}(t,{\mathbf{x}})$. Hence, the expression in (52) is a surrogate of the error function since it results in small values when the prediction is correct and on large values on the other hand. Notice that agents need to exchange their current actions ${\mathbf{x}}(t)$ with their neighbors to solve the minimization of (52) using the algorithm defined by (20) and (21). Since the dimensionality of the actions is as large as the feature vector we want to find a sparse classifier in order to reduce the communication cost. A way of doing so is to include a $\ell_{1}$ norm regularization in the cost. This regularizer is known to promote sparsity. Let, $\alpha>0$ and define the following local cost

[TABLE]

The previous objective introduces a tradeoff between classification performance and sparsity. Instead, one can define a desired tolerance for classification error – by imposing that $f_{i}(t,{\mathbf{x}})$ is smaller than a given tolerance $\delta>0$ for all $i=1\ldots N$– and by minimizing the objective $\|{\mathbf{x}}\|_{1}$, so to get the sparsest of the solutions. With this idea we define the following centralized problem

[TABLE]

To solve this problem in a distributed manner, we define – as done in Section II – local copies of the classifier ${\mathbf{x}}_{i}$ for each agent. The decentralized version of the previous problem then yields

[TABLE]

We evaluate the performance of the saddle-point algorithm (20)–(21) by solving the problem (55) applied to the team of robots navigating around the intersection depicted in Figure 1. The positions of the $N$ agents is initialized by drawing it from a uniform distribution on the square $[-L,L]^{2}$ and their paths are random walks updated every $T_{s}$ seconds, where each step is drawn from a two-dimensional Gaussian variable, with zero mean and covariance matrix $\text{diag}(\sigma_{w},\sigma_{w})$. Ever $T_{s}$ seconds each agent has observed $I$ images in the IRA111Integrated Research Assessment for the U.S. Army s Robotics Collaborative Technology Alliance database [5] of either grass or pavement. Do notice that even though the algorithm proposed is derived in continuous time, for this application we propose to work with a discrete time system. In Section V-B we present the results achieved by the saddle-point algorithm in the previously described problem. Before doing so, we describe in the next section the feature extraction from the images.

V-A Data from image database

The feature extraction is done as in [5], a procedure inspired in the two-dimensional texton [37]. We describe it next for completeness. The texture features ${\mathbf{z}}_{i}(t)$ are generated as the sum of a sparse dictionary representation of subpatches of size 24-by-24. This is, each robot classifies images patches of size 24-by-24 by first extracting the nine non-overlapping 8-by-8 sub-patches within it. Each sub-patch is then vectorized, the sample mean subtracted off and divided by its norm. Such that the resulting sub-patch $j$ observed by agent $i$, yields a zero-mean vector ${\mathbf{z}}_{i}^{j}$ with norm one. The $9$ vectors resulting from each sub-patch are stacked as columns in a matrix ${\mathbf{Z}}_{i}=[{\mathbf{z}}_{i}^{1};\ldots;{\mathbf{z}}_{i}^{9}]$. On the other hand, the agents have a dictionary of textures that has been trained offline following [5]. An example of this dictionary can be observed in Figure 2. The dictionary can be represented by a matrix ${\mathbf{D}}\in{\mathcal{M}}^{n\times 64}$, where $n$ is the number of features that one wants to extract. The feature used for classification by agent $i$ is the aggregate sparse coding ${\mathbf{z}}_{i}(t)$, defined as ${\mathbf{z}}_{i}(t)=\sum_{j=1}^{9}\bm{{\mathbf{z}}}^{*}({\mathbf{D}};{\mathbf{z}}_{i}^{j}(t))$, where ${\mathbf{z}}^{*}({\mathbf{D}};{\mathbf{z}}_{i}^{j}(t))$ is the solution to the following optimization problem.

[TABLE]

where $\zeta>0$ the coefficient of the regularization.

V-B Results

In this section we present the bahavior of the Online Distributed Online Algorithm (20)–(21) for a team of robots that drive in the intersection as the one depicted in Figure 1. For this particular example we consider $N=20$ agents, $L=15$, $\sigma_{w}$ and $T_{s}=1$ . The parameters of the feature extraction are set to $\zeta=0.125$, $n=128$. We chose $\delta=0.001$, $\gamma=10$ and the algorithm step size to be $\eta=0.02$, and we consider that each agent has acces to $24$ images per sampling period, in this case, $24$ images per second.

As it can be observed in Figure 3 the network disagreement converges to zero in approximately $6$ seconds, which implies consensus among the agents. This observation supports the theoretical result in Proposition 1. In Figure 4 we depict the network fit for one randomly selected node. As predicted by Theorem 1 the fit is sublinear. The effectiveness of the algorithm can be observed in the classification accuracy achieved by the agents in Figure 5. Notice that the classification error of all the agents is bellow $30\%$. It can be observed as well, that some agents classify with accuracy above $90\%$. The latter is the case for agents that are observing grass. There seems to be an intrinsic difficulty in classifying pavement in the current data set. To support this claim we compute the covariance matrix of $512$ features of images selected randomly. We then project the $192$-dimensional feature vector onto the first two principal components. This projection is depicted in Figure 6. As it can be observed the points corresponding to pavement cannot be separated from points corresponding to grass, yet there is a cluster of half of the grass points that is away from the points containing pavement. This suggests that it is indeed harder to classify the pavement images.

VI Conclusion

We considered the problem of constrained distributed online learning. Each agent only has access to its local constraints and objective function, and the aim is to coordinate the actions among the agents such that the resulting trajectories are feasible and optimal for the team as a whole. We showed that a distributed online version of the saddle point algorithm achieves global fit, regret and network disagreement bounded by functions whose growth rate is bounded by $\sqrt{T}$. The latter result suggests vanishing constraint violation, optimality and network agreement in average as time evolves. We evaluate the performance of the algorithm for a team of robots driving through an urban environment to perform real time texture classification for the purpose of terrain recognition.

Appendix A Appendices

A-A Proof of Lemma 1

Let us start by considering the time derivative of the energy function defined in (22). Using the chain rule yields

[TABLE]

Substituting the time derivatives by those given by the Distributed Online Saddle Point dynamics (20)-(21) in the previous expression yields

[TABLE]

Since both ${\mathbf{x}}_{i}(t)$ and $\tilde{{\mathbf{x}}}_{i}$ belong to the convex set ${\mathcal{X}}$, the inner product between ${\mathbf{x}}_{i}(t)-\tilde{{\mathbf{x}}}_{i}$ and the projected vector can be upper bounded by the the product with the field without the projection (cf., Lemma 1 [20])

[TABLE]

Summing across agents on both sides of the previous expression allows us to upper bound the first term summation in (58)

[TABLE]

In addition, since the Lagrangian is a convex function with respect to ${\mathbf{x}}_{i}(t)$ we can further upper bound the sum of inner products by the difference between the Lagrangian evaluated at $\tilde{{\mathbf{x}}}_{i}$ and ${\mathbf{x}}_{i}(t)$. Proceeding in this way for all $i\in{\mathcal{V}}$ yields

[TABLE]

Likewise, for the multipliers the following relationships hold by virtue of [20, Lemma 1]

[TABLE]

and

[TABLE]

Because the Lagrangian is linear with respect to $\bm{\lambda}$ and $\bm{\mu}$ (cf., (15)) we can write the above inner products as differences of the Lagrangian evaluated at $\bm{\lambda}_{i}(t)$ and $\bar{\bm{\lambda}}_{i}$ and as differences of the Lagrangian evaluated at $\bm{\mu}_{i}(t)$ and $\bar{\bm{\mu}}_{i}$

[TABLE]

Substituting the expressions (61) and (64) in (58) reduces to

[TABLE]

Observe that the second and third terms on the right hand side of the previous expression cancel out for all $t$. Hence, the previous bound reduces to

[TABLE]

Rearranging the terms in the previous inequality and integrating both sides from $t=0$ until the time horizon $t=T$ yields

[TABLE]

By virtue of the Fundamental Theorem of Calculus, the right hand side of the previous expression reduces to the difference between ${V}_{\tilde{{\mathbf{x}}},\tilde{\bm{\lambda}},\tilde{\bm{\mu}}}({\mathbf{x}}(t),\bm{\lambda}(t),\bm{\mu}(t))$ evaluated at time $T$ and [math]. The proof if completed by observing that for any $T$, ${V}_{\tilde{{\mathbf{x}}},\tilde{\bm{\lambda}},\tilde{\bm{\mu}}}({\mathbf{x}}(T),\bm{\lambda}(T),\bm{\mu}(T))$ is non-negative and thus, the right hand side can be upper bounded by the energy function evaluated at $t=0$.