A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates

TL;DR

This paper introduces a decentralized proximal-gradient algorithm that allows agents to use independent step-sizes, achieves linear convergence under strong convexity, and separates convergence rate dependencies on objectives and network topology.

Contribution

The proposed algorithm enables network-independent step-sizes and separates convergence rate dependencies, improving upon previous methods like PG-EXTRA.

Findings

Agents can use uncoordinated step-sizes independent of network topology.

Linear convergence is achievable without non-smooth terms under strong convexity.

Convergence rate depends separately on objective functions and network structure.

Abstract

This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a previous algorithm, PG-EXTRA \cite{shi2015proximal}, but has a few advantages. First of all, agents use uncoordinated step-sizes, and the stable upper bounds on step-sizes are independent of network topologies. The step-sizes depend on local objective functions, and they can be as large as those of the gradient descent. Secondly, for the special case without non-smooth terms, linear convergence can be achieved under the strong convexity assumption. The dependence of the convergence rate on the objective functions and the network are separated, and the convergence rate of the new algorithm is as good as one of the two convergence rates that match the typical rates for the general gradient descent and the consensus averaging. We provide numerical experiments to demonstrate the efficacy of the introduced algorithm and validate our theoretical discoveries.

Figures

Tables

Table 1. TABLE I: Summary of algorithmic parameters used in EXTRA and NIDS. The step size bound on α 𝛼 \alpha for EXTRA comes from reference [ 47 ] which improves that given in reference  [ 36 ] .

Method	$\alpha$ or ${\alpha }_i$	$c$
EXTRA (${\lambda }_n\left(𝐖\right)$ given)		–
EXTRA (${\lambda }_n\left(𝐖\right)$ not given)		–
NIDS (${\lambda }_n\left(𝐖\right)$ given)		$c\le 1/\left(\left(1-{\lambda }_n\left(𝐖\right)\right){\max }_i{\alpha }_i\right)$
NIDS (${\lambda }_n\left(𝐖\right)$ not given)		$c\le 1/\left(2{\max }_i{\alpha }_i\right)$

Table 2. TABLE II: Summary of a few relevant algorithms. Here, μ 𝜇 \mu (or μ g subscript 𝜇 𝑔 \mu_{g} or μ ¯ ¯ 𝜇 \bar{\mu} ) is the strong convexity constant of the objective function (or that of a modified objective function); L 𝐿 L (or L ¯ ¯ 𝐿 \bar{L} ) is the Lipschitz constant of the objective gradient (or its modified version). Also σ = 1 − λ n ​ ( 𝐖 ) 1 − λ 2 ​ ( 𝐖 ) 𝜎 1 subscript 𝜆 𝑛 𝐖 1 subscript 𝜆 2 𝐖 \sigma=\frac{1-\lambda_{n}({\mathbf{W}})}{1-\lambda_{2}({\mathbf{W}})} is considered as the condition number of the network, which scales at the order of O ​ ( n 2 ) 𝑂 superscript 𝑛 2 O(n^{2}) in the worst case scenario. We omit “ O ​ ( ⋅ ) 𝑂 ⋅ O(\cdot) ” in “Orders of step-size bounds” and “Scalability” for brevity. Note quantities involving K 𝐾 K only hold for a finite K 𝐾 K .

Algorithm	Support	Orders of step-size bounds	Uncoordinated	Scalability	Rate
	prox. operators	(strongly convex; convex)	step-size	(strongly convex)	(convex)
EXTRA
[36, 1]	Yes	$\frac{{\mu }_g}{L^2}$; $\frac{1}{L}$	No	${\left(\frac{L}{{\mu }_g}\right)}^2$	$o\left(\frac{1}{k}\right)$
Aug-DGM
[38]	No	small enough	Yes	–	converges
Harness
[40]	No	$\frac{\mu }{L^2{\sigma }^2}$; $\frac{1}{L{\sigma }^2}$	No	${\left(\frac{L}{\mu }\sigma \right)}^2$	$O\left(\frac{1}{k}\right)$
Acc-DNGD		$\frac{{\left(\sigma -1\right)}^3}{{\sigma }^{6}L}{\left(\frac{\mu }{L}\right)}^{3/7}$;
[52]	No	$\frac{\min \{{\left(1-{\sigma }^{-1}\right)}^2,{\left(\sigma \right)}^{-3}\}}{L{k}^{0.6}}$	No	$\frac{{\sigma }^3}{{\left(\sigma -1\right)}^{1.5}}{\left(\frac{L}{\mu }\right)}^{5/7}$	$O\left(\frac{1}{k^{1.4}}\right)$
DIGing
[41, 42]	No	$\min \{\frac{{\overline{\mu }}^{0.5}}{\sigma \left(\sigma -1\right)L^{1.5}{n}^{0.5}},\frac{1}{\overline{L}}\}$; –	Yes	$\max \{\frac{L}{\overline{\mu }},\frac{{\left(\sigma -1\right)}^2{n}^{0.5}{L}^{1.5}+{\sigma }^2{\overline{\mu }}^{1.5}}{{\overline{\mu }}^{1.5}}\}$	–
Optimal
[53, 54]	No	$\mu$; $\frac{L\sigma }{K^2}$	No	${\left(\frac{L}{\mu }\sigma \right)}^{0.5}$	$O\left(\frac{1}{K^2}\right)$
NIDS	Yes	$\frac{1}{L}$; $\frac{1}{L}$	Yes	$\max \{\frac{L}{\mu },\sigma \}$	$o\left(\frac{1}{k}\right)$

Equations

$$\underset{x\in {\mathbb{R}}^p}{\mathrm{minimize}}\bar{f}(x):=\frac{1}{n}\sum _{i=1}^n(s_i(x)+r_i(x)),$$

$${\mathbf{prox}}_{\lambda r_i}(y)=\underset{x\in {\mathbb{R}}^p}{\mathrm{argmin}}\lambda r_i(x)+\frac{1}{2}\Vert x-y{\Vert }^2,$$

$$f(\mathbf{x}):=\frac{1}{n}\sum _{i=1}^n(s_i(x_i)+r_i(x_i)),$$

$$\mathbf{x}:=\left(\begin{array}{ccc}-& x_1^{\top }& -\\ -& x_2^{\top }& -\\ & ⋮& \\ -& x_n^{\top }& -\end{array}\right)\in {\mathbb{R}}^{n\times p}.$$

$$s(\mathbf{x}):=\frac{1}{n}\sum _{i=1}^n{s}_i(x_i),r(\mathbf{x}):=\frac{1}{n}\sum _{i=1}^n{r}_i(x_i).$$

$$\nabla s(\mathbf{x}):=\left(\begin{array}{ccc}-& {\left(\nabla s_1(x_1)\right)}^{\top }& -\\ -& {\left(\nabla s_2(x_2)\right)}^{\top }& -\\ & ⋮& \\ -& {\left(\nabla s_n(x_n)\right)}^{\top }& -\end{array}\right)\in {\mathbb{R}}^{n\times p}.$$

$$\begin{array}{r}(\text{EXTRA}){\mathbf{x}}^{k+2}=\frac{\mathbf{I}+\mathbf{W}}{2}(2{\mathbf{x}}^{k+1}-{\mathbf{x}}^k)\\ -\alpha \nabla s({\mathbf{x}}^{k+1})+\alpha \nabla s({\mathbf{x}}^k),\end{array}$$

$$\begin{array}{r}(\text{Proposed NIDS}){\mathbf{x}}^{k+2}=\frac{\mathbf{I}+\mathbf{W}}{2}(2{\mathbf{x}}^{k+1}-{\mathbf{x}}^k\\ -\alpha \nabla s({\mathbf{x}}^{k+1})+\alpha \nabla s({\mathbf{x}}^k)).\end{array}$$

$$O\left(\max \left(\frac{L}{\mu },\frac{1-{\lambda }_n(\mathbf{W})}{1-{\lambda }_2(\mathbf{W})}\right)\right)\log \frac{1}{ϵ},$$

$$\underset{\mathbf{x}}{\mathrm{minimize}}s(\mathbf{x}),\text{s.t.}(\mathbf{I}-\mathbf{W}{)}^{1/2}\mathbf{x}=0,$$

$$\displaystyle\left[\begin{array}[]{cc}0&\textbf{{\L}}\\ -\textbf{{\L}}&0\end{array}\right]\left[\begin{array}[]{c}{\mathbf{x}}^{*}\\ {\mathbf{p}}^{*}\end{array}\right]=-\left[\begin{array}[]{c}\nabla s({\mathbf{x}}^{*})\\ 0\end{array}\right].$$

$$\displaystyle\left[\left[\begin{array}[]{cc}{1\over\alpha}{\mathbf{I}}&-\textbf{{\L}}\\ -\textbf{{\L}}&2\alpha{\mathbf{I}}\end{array}\right]+\left[\begin{array}[]{cc}0&\textbf{{\L}}\\ -\textbf{{\L}}&0\end{array}\right]\right]\left[\begin{array}[]{c}{\mathbf{x}}^{k+1}\\ {\mathbf{p}}^{k+1}\end{array}\right]$$

$$\displaystyle=\left[\begin{array}[]{cc}{1\over\alpha}{\mathbf{I}}&-\textbf{{\L}}\\ -\textbf{{\L}}&2\alpha{\mathbf{I}}\end{array}\right]\left[\begin{array}[]{c}{\mathbf{x}}^{k}\\ {\mathbf{p}}^{k}\end{array}\right]-\left[\begin{array}[]{c}\nabla s({\mathbf{x}}^{k})\\ 0\end{array}\right].$$

$${\mathbf{x}}^{k+1}=$$

$$\alpha {\mathbf{p}}^{k+1}=$$

$$\left[\begin{array}[]{cc}{1\over\alpha}{\mathbf{I}}&-\textbf{{\L}}\\ -\textbf{{\L}}&2\alpha{\mathbf{I}}\end{array}\right]\succcurlyeq\left[\begin{array}[]{cc}{L\over 2}{\mathbf{I}}&\mathbf{0}\\ \mathbf{0}&\mathbf{0}\end{array}\right],$$

$$\displaystyle\left[\left[\begin{array}[]{cc}{1\over\alpha}{\mathbf{I}}&0\\ 0&\alpha({\mathbf{I}}+{\mathbf{W}})\end{array}\right]+\left[\begin{array}[]{cc}0&\textbf{{\L}}\\ -\textbf{{\L}}&0\end{array}\right]\right]\left[\begin{array}[]{c}{\mathbf{x}}^{k+1}\\ {\mathbf{p}}^{k+1}\end{array}\right]$$

$$=$$

$$2\alpha {\mathbf{p}}^{k+1}=$$

$${\mathbf{x}}^{k+1}=$$

$$z_i^1$$

$$x_i^1$$

$$z_i^{k+1}=z_i^k-x_i^k+\sum _{j={\mathcal{N}}_i\cup \{i\}}{\tilde{w}}_{ij}(2x_j^k-x_j^{k-1}$$

$$-{\alpha }_j\nabla s_j\left(x_j^k\right)+{\alpha }_j\nabla s_j\left(x_j^{k-1}\right)),$$

$$x_i^{k+1}=\underset{x\in {\mathbb{R}}^p}{\mathrm{argmin}}{\alpha }_i{r}_i(x)+\frac{1}{2}\Vert x-z_i^{k+1}{\Vert }^2.$$

$$⟨\mathbf{x}-\mathbf{y},\nabla s(\mathbf{x})-\nabla s(\mathbf{y})⟩\ge {\Vert \nabla s\left(\mathbf{x}\right)-\nabla s\left(\mathbf{y}\right)\Vert }_{{\mathbf{L}}^{-1}}^2,$$

$$\begin{array}{rl}{\mathbf{z}}^{k+1}=& {\mathbf{z}}^k-{\mathbf{x}}^k+\tilde{\mathbf{W}}(2{\mathbf{x}}^k-{\mathbf{x}}^{k-1}\\ & -\Lambda \nabla s({\mathbf{x}}^k)+\Lambda \nabla s({\mathbf{x}}^{k-1})),\end{array}$$

$${\mathbf{x}}^{k+1}=$$

$$\begin{array}{rl}{\mathbf{z}}^{k+1}=& {\mathbf{z}}^k-{\mathbf{x}}^k+\frac{\mathbf{I}+\mathbf{W}}{2}(2{\mathbf{x}}^k-{\mathbf{x}}^{k-1}\\ & -\alpha \nabla s({\mathbf{x}}^k)+\alpha \nabla s({\mathbf{x}}^{k-1})),\end{array}$$

$${\mathbf{x}}^{k+1}=$$

$$\begin{array}{rl}{\mathbf{z}}^{k+1}=& {\mathbf{z}}^k-{\mathbf{x}}^k+\frac{\mathbf{I}+\mathbf{W}}{2}(2{\mathbf{x}}^k-{\mathbf{x}}^{k-1})\\ & -\alpha \nabla s({\mathbf{x}}^k)+\alpha \nabla s({\mathbf{x}}^{k-1}),\end{array}$$

$${\mathbf{x}}^{k+1}=$$

$${\mathbf{d}}^k:={\Lambda }^{-1}({\mathbf{x}}^{k-1}-{\mathbf{z}}^k)-\nabla s\left({\mathbf{x}}^{k-1}\right).$$

$${\mathbf{d}}^{k+1}$$

$$=$$

Full text

I Introduction

This paper focuses on the following decentralized optimization problem:

[TABLE]

where $s_{i}:\mathbb{R}^{p}\rightarrow\mathbb{R}$ and $r_{i}:\mathbb{R}^{p}\rightarrow\mathbb{R}\cup\{+\infty\}$ are two lower semi-continuous proper convex functions held privately by agent $i$ to encode the agent’s objective. We assume that one function (e.g., without loss of generality, $s_{i}$) is differentiable and has a Lipschitz continuous gradient with parameter $L>0$, and the other function $r_{i}$ is proximable, i.e., its proximal mapping

[TABLE]

has a closed-form solution or can be computed easily. Examples of $s_{i}$ include linear functions, quadratic functions, and logistic functions, while $r_{i}$ could be the $\ell_{1}$ norm, 1D total variation, or indicator functions of simple convex sets. In addition, we assume that the agents are connected through a fixed bi-directional communication network. Every agent in the network wants to obtain an optimal solution of (1) while it can only receive/send nonsensitive messages111We believe that agent $i$’s instantaneous estimation on the optimal solution is not a piece of sensitive information but $s_{i}$ and $r_{i}$ are. from/to its immediate neighbors.

Specific problems of form (1) that require a decentralized computing architecture have appeared in various areas including networked multi-vehicle coordination, distributed information processing and decision making in sensor networks, as well as distributed estimation and learning. Some examples include distributed average consensus [2, 3, 4], distributed spectrum sensing [5], information control [6, 7], power systems control [8, 9], statistical inference and learning [10, 11, 12]. In general, decentralized optimization fits the scenarios where the data is collected and/or stored in a distributed network, a fusion center is either inapplicable or unaffordable, and/or computing is required to be performed in a distributed but collaborative manner by multiple agents or by network designers.

I-A Literature Review

The study on distributed algorithms dates back to the early 1980s [13, 14]. Since then, due to the emergence of large-scale networks, decentralized (optimization) algorithms, as a special type of distributed algorithms for solving problem (1), have received attention. Many efforts have been made on star networks with one master agent and multiple slave agents [15, 16]. This scheme is “centralized” due to the use of a “master” agent. It may suffer a single point of failure and may violate the privacy requirement in certain applications. In this paper, we focus on solving (1) in a decentralized fashion, where no “master” agent is used.

Incremental algorithms [17, 18, 19, 20, 21, 22] can solve (1) without the need of a “master” agent and it is based on a directed ring network. To handle general (possibly time-varying) networks, the distributed sub-gradient algorithm was proposed in [23]. This algorithm and its variants [24, 25] are intuitive and simple but usually slow due to the diminishing step-size that is needed to obtain a consensual and optimal solution, even if the objective functions are differentiable and strongly convex. With a fixed step-size, these distributed methods can be fast, but they only converge to a neighborhood of the solution set which depends on the step-size. This phenomenon creates an exactness-speed dilemma [26].

A class of distributed approaches that bypass this dilemma is based on introducing the Lagrangian dual. The resulting algorithms include distributed dual decomposition [27] and decentralized alternating direction method of multipliers (ADMM) [28]. The decentralized ADMM and its proximal-gradient variant can employ a fixed step-size to achieve $O(1/k)$ rate under general convexity assumptions [29, 30, 31]. Under the strong convexity assumption, the decentralized ADMM has linear convergence for time-invariant undirected graphs [32]. There exist some other distributed methods that do not (explicitly) use dual variables but can still converge to an optimal consensual solution with fixed step-sizes. In particular, works in [33, 34] employ multi-consensus inner loops, Nesterov’s acceleration, and/or the adapt-then-combine (ATC) strategy. Under the assumption that the objectives have bounded and Lipschitz continuous gradients222This means that the nonsmooth terms $r_{i}$’s are absent. Such assumption is much stronger than the one used for achieving the $O(1/k^{2})$ rate in Nesterov’s optimal gradient method [35]., the algorithm proposed in [34] has $O\left(\ln(k)/k^{2}\right)$ rate. References [36, 1] use a difference structure to cancel the steady state error in decentralized gradient descent [23, 26], thereby developing the algorithm EXTRA and its proximal-gradient variant PG-EXTRA. It converges at an $O(1/k)$ rate when the objective function in (1) is convex and has a linear convergence rate when the objective function is strongly convex and $r_{i}(x)=0$ for all $i$.

A number of recent works employed the so-called gradient tracking [37] to conquer different issues [38, 39, 40, 41, 42]. To be specific, works [38, 42] relax the step-size rule to allow uncoordinated step-sizes across agents. Paper [39] solves non-convex optimization problems. Paper [41] aims at achieving geometric convergence over time-varying graphs. Work [40] improves the convergence rate over EXTRA, and its formulation is the same as that in [42].

Another topic of interest is decentralized optimization over directed graphs [43, 44, 45, 46, 41], which is beyond the scope of this paper.

I-B Proposed Algorithm and its Advantages

To proceed, let us introduce some basic notation first. Agent $i$ holds a local variable $x_{i}\in\mathbb{R}^{p}$, and we denote $x_{i}^{k}$ as its value at the $k$-th iteration. Then, we introduce a new function that is the average of all the local functions with local variables as

[TABLE]

where

[TABLE]

If all local variables are identical, i.e., $x_{1}=\dots=x_{n}$, we say that ${\mathbf{x}}$ is consensual. In addition, we define

[TABLE]

We have $f({\mathbf{x}})=s({\mathbf{x}})+r({\mathbf{x}})$. The gradient of $s$ at ${\mathbf{x}}$ is given in the same way as ${\mathbf{x}}$ in (3) by

[TABLE]

22footnotetext: In the original EXTRA, two mixing matrices ${\mathbf{W}}$ and $\widetilde{{\mathbf{W}}}$ are used. For simplicity, we let $\widetilde{{\mathbf{W}}}=\frac{I+{\mathbf{W}}}{2}$ here.

By making a simple modification over PG-EXTRA [1], our proposed algorithm brings a big improvement in the speed and the dependency of convergence over networks. To better expose this simple modification, let us compare a special case of our proposed algorithm with EXTRA for the smooth case, i.e., $r({\mathbf{x}})=0$.

[TABLE]

Here, ${\mathbf{W}}\in\mathbb{R}^{n\times n}$ is a matrix that represents information exchange between neighboring agents (more details about this matrix are in Assumption 1) and $\alpha$ is the step-size. The only difference between EXTRA and the proposed algorithm is the information exchanged between the agents. EXTRA exchanges only the estimations $2{\mathbf{x}}^{k+1}-{\mathbf{x}}^{k}$, while the proposed algorithm exchanges the gradient adapted estimations, i.e., $2{\mathbf{x}}^{k+1}-{\mathbf{x}}^{k}-\alpha\nabla s({\mathbf{x}}^{k+1})+\alpha\nabla s({\mathbf{x}}^{k})$. Because of this small modification, the proposed algorithm has a Network InDependent Step-size, which will be explained later. Therefore we name the proposed algorithm NIDS and will use this abbreviation throughout the paper. For the nonsmooth case, more detailed comparison between PG-EXTRA and NIDS will be given in Section III.

A large and network independent step-size for NIDS: All works mentioned above either employ pessimistic step-sizes or have network dependent upper bounds on step-sizes. Furthermore, the step-sizes for the strongly convex case are more conservative. For example, the step-size used to achieve linear convergence rates for EXTRA in [36, 48] is in the order of $O(\mu/L^{2})$, where $\mu$ and $L$ are the strong convexity constant of $s({\mathbf{x}})$ and Lipschitz constant of $\nabla s({\mathbf{x}})$, respectively. As a contrast, the centralized gradient descent can choose a step-size in the order of $O(1/L)$. The upper bound of step-size for EXTRA was recently improved to $(5+3\lambda_{n}({\mathbf{W}}))/(4L)$ in [47]. We can choose $1/(2L)$ for any ${\mathbf{W}}$ satisfying Assumption 1. Another example of employing a constant step-size in distributed optimization is DIGing [41]. Although its ATC variant in [42] was shown to converge faster than DIGing, the step-size is still very conservative compared to $O(1/L)$. We will show that the step-size of NIDS can have the same upper bound $2/L$ as that of the centralized gradient descent. The achievable step-sizes of NIDS for $o(1/k)$ rate in the general convex case and the linear convergence rate in the strongly convex case are both in the order of $O(1/L)$. Furthermore, NIDS allows each agent to have an individual step-size. Each agent $i$ can choose a step-size $\alpha_{i}$ that is as large as $2/L_{i}$ on any connected network, where $L_{i}$ is the Lipschitz constant of $\nabla s_{i}(x)$ and $L_{i}\leq L$ for all $i$ ($L=\max_{i}L_{i}$). Apart from the step-sizes, to run NIDS, a common/public parameter $c$ is needed for the construction of $\widetilde{{\mathbf{W}}}$ (see (8) for the algorithm and $\widetilde{{\mathbf{W}}}$). This parameter $c$ can be chosen without any knowledge of the network (or the mixing matrix ${\mathbf{W}}$). For example, $c={1}/({2\max_{i}\alpha_{i}})$. Table I provides an overview of algorithmic configurations for EXTRA and NIDS. NIDS works as long as each agent can estimate its local functional parameter: No agent needs any other global information including the number of agents in the whole network except the largest step-size, if it is not the same for all agents.

In the line of research of optimization over heterogeneous networks, after the initial work [38] regarding uncoordinated step sizes, references [49] and [50] introduce and analyze a diffusion strategy with corrections that achieves an exact linear convergence with uncoordinated step sizes. This exact diffusion algorithm is still related to the Lagrangian method but can be considered as having incorporated a CTA structure. The CTA strategy can, similar to the ATC strategy, improve the convergence speed of certain consensus optimization algorithms (see [41, Remark 3] and [46, Section II.C]). However, the analysis in [50], though allowing step size mismatch across the network, does not take into consideration the heterogeneity of agents’ functional conditions. Furthermore, their upper bound for the step-size is in the order of $O(\mu/L^{2})$.

Sublinear convergence rate for the general case: Under the general convexity assumption, we show that NIDS has a convergence rate of $o(1/k)$, which is slightly better than the $O(1/k)$ rate of PG-EXTRA. Because the step-size of NIDS does not depend on the network topology and is much larger than that of PG-EXTRA, NIDS can be much faster than PG-EXTRA, as shown in the numerical experiments.

Linear convergence rate for the strongly convex case: Let us first define “scalability”. When the iterate $x^{k}$ of an algorithm converges to the optimal solution $x^{*}$ linearly, i.e., $\|x^{k}-x^{*}\|^{2}=O((1-1/S)^{k})$ with some positive constant $S$, we say that the algorithm needs to run $O(S)\log(1/\epsilon)$ iterations to reach $\epsilon$-accuracy. So we call $O(S)$ the scalability of the algorithm.

For the case where the non-smooth terms are absent and the functions $\{s_{i}\}_{i=1}^{n}$ are strongly convex, we show that NIDS achieves a linear convergence rate whose dependencies on the functions $\{s_{i}\}_{i=1}^{n}$ and the network topology are decoupled. To be specific, to reach $\epsilon$-accuracy, the number of iterations needed for NIDS is

[TABLE]

where $\lambda_{i}({\mathbf{W}})$ is the $i$th largest eigenvalue of ${\mathbf{W}}$. Both $\frac{L}{\mu}$ and $\frac{1-\lambda_{n}({\mathbf{W}})}{1-\lambda_{2}({\mathbf{W}})}$ are typical in the literature of optimization and average consensus, respectively. The value $\frac{L}{\mu}$, also called the condition number of the objective function, is aligned with the scalability of the standard gradient descent [35]. The value $\frac{1-\lambda_{n}({\mathbf{W}})}{1-\lambda_{2}({\mathbf{W}})}$ is understood as the condition number444When we choose ${\mathbf{W}}={\mathbf{I}}-\tau\textbf{\L}$ where Ł is the Laplacian of the underlying graph and $\tau$ is a positive tunable constant, we have $\frac{1-\lambda_{n}({\mathbf{W}})}{1-\lambda_{2}({\mathbf{W}})}=\frac{\lambda_{1}(\textbf{\L})}{\lambda_{n-1}(\textbf{\L})}$ which is the finite condition number of Ł. Note that $\lambda_{n}(\textbf{\L})=0$. of the network and aligned with the scalability of the simplest linear iterations for distributed averaging [51].

Separating the condition numbers of the objective function and the network provides a way to determine the bottleneck of NIDS for a specific problem and a given network. Therefore, the system designer might be able to smartly apply preconditioning on $\{s_{i}\}_{i=1}^{n}$ or improve the connectivity of the network to cost-effectively obtain a better convergence.

Summary and comparison of state-of-the-art algorithms: We list the properties of a few relevant algorithms in Table II. We let $\sigma:=\frac{1-\lambda_{n}({\mathbf{W}})}{1-\lambda_{2}({\mathbf{W}})}$. This quantity is directly affected by the network topology and how the matrix ${\mathbf{W}}$ is defined, thus is also related to the consensus ability of a network. When the network is fully connected (a complete graph), we can choose ${\mathbf{W}}$ so that $\lambda_{2}({\mathbf{W}})=\lambda_{n}({\mathbf{W}})=0$ and thus $\sigma=1$ (the best case); in general $\sigma\geq 1$ since $0<1-\lambda_{2}({\mathbf{W}})\leq 1-\lambda_{n}({\mathbf{W}})<2$; in the worst case, we have $\sigma\leq\frac{1}{1-\lambda_{2}({\mathbf{W}})}=O(n^{2})$ [4, Section 2.3]. We keep $\sigma$ in the bounds/rates of involved algorithms for a fair comparison instead of focusing on the worst case that often gives pessimistic/conservative results. We omit “$O(\cdot)$” in “Bounds of step-sizes” and “Scalabilities” for brevity and only compare the effect of functional properties ($\mu$ and $L$) and network properties ($\sigma$ and/or $n$). Before talking into details, let us clarify a few points. In EXTRA, $\mu_{g}$ is a quantity that is associated with the strong convexity of the original function $\bar{f}(x)$, so it covers a larger class of problems; In DIGing, $\bar{\mu}$ is the mean value of the strong convexity constants of local objectives; In Acc-DNGD, the step-size for the convex case contains $k$, the current number of iterations. Thus it represents a diminishing step-size sequence; In Optimal [53, 54], the total number of iterations $K$ is used to determine the step-size for the convex case. In addition, they apply to problems in which the objectives are dual friendly (see [54] for its definition). Note some types of objectives are suitable for gradient update, some are suitable for dual gradient update (dual friendly), and some are suitable for proximal update.

Finding the algorithm with the lowest per-iteration cost depends on the problem (functions). Apparently, our bounds on step-sizes and the corresponding scalability/rate are better than those given in EXTRA and Harness (see Table II). When $\sigma$ is close to $1$ (the graph is well connected), the step-size bound and scalability given in DIGing are the same as NIDS. However, when $\sigma$ is large, their result becomes rather conservative. Acc-DNGD and Optimal have improved the scalability/rate of gradient-based distributed optimization by employing Nesterov’s acceleration technique on primal and dual problems, respectively. For the convex case, our rate is worse than theirs because our algorithm does not employ Nesterov’s acceleration. For the primal distributed gradient method after acceleration [52], the scalability in $\sigma$ is still worse than our result. Algorithm Optimal achieves the optimal scalability/rate for distributed optimization. However, as we have mentioned above, their algorithms are dual based thus apply to a different class of problems. In addition, NIDS supports proximable non-smooth functions and uncoordinated step-sizes while these have not been considered in Acc-DNGD and Optimal. To sum up, we have reached the best possible performance of first-order algorithms for distributed optimization without acceleration. Further improving the performance by incorporating Nesterov’s techniques to our algorithm will be a future direction.

Finally, we note that, references [49, 50], appearing simultaneously with this work, also proposed (6b) to enlarge the step-size and use column stochastic matrices rather than symmetric doubly stochastic matrices. However, their algorithm only works for smooth problems, and their analysis seems to be restrictive and requires twice differentiability and strong convexity of $\{s_{i}\}_{i=1}^{n}$. The stepsize is also in the order of $\mu/L^{2}$ [48].

I-C Future Works

The capability of our algorithm using purely locally determined parameters increases its potential to be extended to dynamic networks with a time-varying number of nodes. Given such flexibility, we may use similar schemes to solve the decentralized empirical risk minimization problems. Furthermore, it also enhances the privacy of the agents through allowing each agent to perform their own optimization procedure without negotiation on any parameter.

By using Nesterov’s acceleration technique, reference [4] shows that the scalability of a new average consensus protocol can be improved to $O(n)$; when the nonsmooth terms $r_{i}$’s are absent, reference [53] shows that the scalability of a new dual based accelerated distributed gradient method can be improved to $O(\sqrt{\sigma L/\mu})$. One of our future work is exploring the convergence rates/scalability of the Nesterov’s accelerated version of our algorithm.

I-D Paper Organization

The rest of this paper is organized as follows. To facilitate the description of the technical ideas, the algorithms, and the analysis, we introduce additional notation in Subsection I-E. The intuition for the network-independent step-size is provided in Section II. In Section III, we introduce our algorithm NIDS and discuss its relation to some other existing algorithms. In Section IV, we first show that NIDS can be understood as an iterative algorithm for seeking a fixed point. Following this, we establish that NIDS converges at an $o(1/k)$ rate for the general convex case and a linear rate for the strongly convex case. Then, numerical simulations are given in Section V to corroborate our theoretical claims. Final remarks are given in Section VI.

I-E Notation

We use bold upper-case letters such as ${\mathbf{W}}$ to define matrices in $\mathbb{R}^{n\times n}$ and bold lower-case letters such as ${\mathbf{x}}$ and ${\mathbf{z}}$ to define matrices in $\mathbb{R}^{n\times p}$ (when $p=1$, they are vectors). Let ${\mathbf{1}}$ and $\mathbf{0}$ be matrices with all ones and zeros, respectively, and their dimensions are provided when necessary. For matrices ${\mathbf{x}},~{}{\mathbf{y}}\in\mathbb{R}^{n\times p}$, we define their inner product as $\langle{\mathbf{x}},{\mathbf{y}}\rangle={\mathrm{tr}}({\mathbf{x}}^{\top}{\mathbf{y}})$ and the norm as $\|{\mathbf{x}}\|=\sqrt{\langle{\mathbf{x}},{\mathbf{x}}\rangle}$. Additionally, by an abuse of notation, we define $\langle{\mathbf{x}},{\mathbf{y}}\rangle_{\mathbf{Q}}={\mathrm{tr}}({\mathbf{x}}^{\top}{\mathbf{Q}}{\mathbf{y}})$ and $\|{\mathbf{x}}\|_{\mathbf{Q}}^{2}=\langle{\mathbf{x}},{\mathbf{x}}\rangle_{\mathbf{Q}}$ for any given symmetric matrix ${\mathbf{Q}}\in\mathbb{R}^{n\times n}$. Note that $\langle\cdot,\cdot\rangle_{\mathbf{Q}}$ is an inner product defined in $\mathbb{R}^{n\times p}$ if and only if ${\mathbf{Q}}$ is positive definite. However, when ${\mathbf{Q}}$ is not positive definite, $\langle{\mathbf{x}},{\mathbf{y}}\rangle_{\mathbf{Q}}$ can still be an inner product defined in a subspace of $\mathbb{R}^{n\times p}$, see Lemma 3 for more details. We define the range of ${\mathbf{A}}\in\mathbb{R}^{n\times n}$ by ${\mathbf{range}}({\mathbf{A}}):=\{{\mathbf{x}}\in\mathbb{R}^{n\times p}:{\mathbf{x}}={\mathbf{A}}{\mathbf{y}},~{}{\mathbf{y}}\in\mathbb{R}^{n\times p}\}$. The largest eigenvalue of a symmetric matrix ${\mathbf{A}}$ is also denoted as $\lambda_{\max}({\mathbf{A}})$. For two symmetric matrices ${\mathbf{A}},{\mathbf{B}}\in\mathbb{R}^{n\times n}$, ${\mathbf{A}}\succ{\mathbf{B}}$ (or ${\mathbf{A}}\succcurlyeq{\mathbf{B}}$) means that ${\mathbf{A}}-{\mathbf{B}}$ is positive definite (or positive semidefinite). Moreover, we use ${\mathcal{N}}_{i}$ to represent the set of agents that can directly send messages to agent $i$.

II Intuition for Network-independent step-size

In this section, we provide an intuition for the network-independent step-size for NIDS with only the differentiable function $s$. The decentralized optimization problem is equivalent to

[TABLE]

where $({\mathbf{I}}-{\mathbf{W}})^{1/2}$ is the square root of ${\mathbf{I}}-{\mathbf{W}}$, and the constraint is the same as the consensual condition with the mixing matrix ${\mathbf{W}}$ given in Assumption 1. Denote $\textbf{{\L}}=({\mathbf{I}}-{\mathbf{W}})^{1/2}$. The corresponding optimality condition with the introduction of the dual variable ${\mathbf{p}}$ is

[TABLE]

EXTRA is equivalent to the Condat-Vu primal-dual algorithm [55, 47], and it can be further explained as a forward-backward splitting applied to the equation, i.e.,

[TABLE]

The update is

[TABLE]

It is equivalent to EXTRA after ${\mathbf{p}}$ is eliminated. In this case, the new metric is a full matrix, and therefore, the upper bound of the step-size $\alpha$ depends on the matrix Ł. To be more specific,

[TABLE]

which gives $\alpha\leq 2(1+\lambda_{\min}({\mathbf{W}}))/L$. A larger and optimal upper bound for the step-size of EXTRA is shown in [47] (See Table I), and it still depends on ${\mathbf{W}}$. However, we choose a block diagonal metric and have

[TABLE]

The update becomes

[TABLE]

which is equivalent to NIDS after ${\mathbf{p}}$ is eliminated. Because the new metric is block diagonal, and the nonexpansiveness of the forward step depends on the function only, i.e., $\alpha\leq 2/L$.

III Proposed Algorithm NIDS

In this section, we describe our proposed NIDS in Algorithm 1 for solving (1) in more details and explain the connections to other related methods.

The mixing matrix satisfies the following assumption, which comes from [1, 36].

Assumption 1** (Mixing matrix)**

The connected network ${\mathcal{G}}=\{{\mathcal{V}},{\mathcal{E}}\}$ consists of a set of agents ${\mathcal{V}}=\{1,2,\cdots,n\}$ and a set of undirected edges ${\mathcal{E}}$. An undirected edge $(i,j)\in{\mathcal{E}}$ means that there is a connection between agents $i$ and $j$ and both agents can exchange data. The mixing matrix ${\mathbf{W}}=[w_{ij}]\in\mathbb{R}^{n\times n}$ satisfies:

(Decentralized property).* If $i\neq j$ and $(i,j)\notin{\mathcal{E}}$, then $w_{ij}=0$;* 2. 2.

(Symmetry).* ${\mathbf{W}}={\mathbf{W}}^{T}$;* 3. 3.

(Null space property).* $\mathbf{Null}({\mathbf{I}}-{\mathbf{W}})=\mathbf{span}({\mathbf{1}}_{n\times 1})$;* 4. 4.

(Spectral property).* $2{\mathbf{I}}\succcurlyeq{\mathbf{W}}+{\mathbf{I}}\succ\mathbf{0}_{n\times n}$.*

Remark 1

Assumption 1 implies that the eigenvalues of ${\mathbf{W}}$ lie in $(-1,1]$ and the multiplicity of eigenvalue $1$ is one, i.e., $1=\lambda_{1}({\mathbf{W}})>\lambda_{2}({\mathbf{W}})\geq\cdots\geq\lambda_{n}({\mathbf{W}})>-1$. Item 3 of Assumption 1 shows that $({\mathbf{I}}-{\mathbf{W}}){\mathbf{1}}_{n\times 1}=\mathbf{0}$ and the orthogonal complement of $\mathbf{span}({\mathbf{1}}_{n\times 1})$ is the row space of ${\mathbf{I}}-{\mathbf{W}}$, which is also the column space of ${\mathbf{I}}-{\mathbf{W}}$ because of the symmetry of ${\mathbf{W}}$.

The functions $\{s_{i}\}_{i=1}^{n}$ and $\{r_{i}\}_{i=1}^{n}$ satisfy the following assumption.

Assumption 2

Functions $\{s_{i}(x)\}_{i=1}^{n}$ and $\{r_{i}(x)\}_{i=1}^{n}$ are lower semi-continuous proper convex, and $\{s_{i}(x)\}_{i=1}^{n}$ have Lipschitz continuous gradients with constants $\{L_{i}\}_{i=1}^{n}$, respectively. Thus, we have

[TABLE]

where ${\mathbf{L}}={\mathrm{Diag}}(L_{1},\cdots,L_{n})$ is the diagonal matrix with the Lipschitz constants [35].

Instead of using the same step-size for all the agents, we allow agent $i$ to choose its own step-size $\alpha_{i}$ and let $\Lambda={\mathrm{Diag}}(\alpha_{1},\cdots,\alpha_{n})\in\mathbb{R}^{n\times n}$. Then NIDS can be expressed as

[TABLE]

where $\widetilde{\mathbf{W}}={\mathbf{I}}-c\Lambda({\mathbf{I}}-{\mathbf{W}})$ and $c$ is chosen such that $\Lambda^{-1/2}\widetilde{\mathbf{W}}\Lambda^{1/2}={\mathbf{I}}-c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}\succcurlyeq\mathbf{0}$. This condition shows that the upper bound of the parameter $c$ depends on ${\mathbf{W}}$ and $\Lambda$. When the information about ${\mathbf{W}}$ is not given, we can just let $c=1/(2\max_{i}\alpha_{i})$ because $\lambda_{n}({\mathbf{W}})>-1$. To set such a parameter, a preprocessing step is needed to obtain the maximum. However, since the maximum can be easily computed in a connected network in no more than $n-1$ rounds of communication wherein each node repeatedly takes maximum of the values from neighbors, the cost of this preprocessing is essentially negligible compared to the worst-case running time of our optimization protocol.

If all agents choose the same step-size, i.e., $\Lambda=\alpha{\mathbf{I}}$, and we let $c=1/(2\alpha)$,  (8) becomes

[TABLE]

Remark 2

The update of PG-EXTRA is

[TABLE]

The only difference between NIDS and PG-EXTRA is that the mixing operation is further applied to the successive difference of the gradients $-\alpha\nabla s({\mathbf{x}}^{k})+\alpha\nabla s({\mathbf{x}}^{k-1})$ in NIDS.

When there is no function $r({\mathbf{x}})$, (8) becomes

[TABLE]

and it further reduces to (6b) when $\Lambda=\alpha{\mathbf{I}}$ and $c=1/(2\alpha)$. Note that, though (6b) appears in [49, 50], its convergence still needs a small step-size that also depends on the network topology and the strong convexity constant. In Theorem 1 of [50], the upper bound for the step-size is also $O(\mu/L^{2})$, which is the same as that of PG-EXTRA.

IV Convergence Analysis of NIDS

In order to show the convergence of NIDS, we also need the following assumption.

Assumption 3** (Solution existence)**

Problem (1) has at least one solution.

To simplify the analysis, we introduce a new sequence $\{{\mathbf{d}}^{k}\}_{k\geq 0}$ which is defined as

[TABLE]

Using the sequence $\{{\mathbf{x}}^{k}\}_{k\geq 0}$, we obtain a recursive (update) relation for $\{{\mathbf{d}}^{k}\}_{k\geq 0}$:

[TABLE]

where the second equality comes from the update of ${\mathbf{z}}^{k+1}$ in (8a) and the last one holds because of the definition of ${\mathbf{d}}^{k}$ in (11). Therefore, the iteration (8) is equivalent to, with the update order $({\mathbf{x}},{\mathbf{d}},{\mathbf{z}})$,

[TABLE]

in the sense that both (8) and (12) generate the same $\{{\mathbf{x}}^{k},{\mathbf{z}}^{k}\}_{k>0}$ sequence.

Because ${\mathbf{x}}^{k}$ is determined by ${\mathbf{z}}^{k}$ only and can be eliminated from the iteration, iteration (12) is essentially an operator for $({\mathbf{d}},{\mathbf{z}})$. Note that we have ${\mathbf{d}}^{1}={\Lambda^{-1}}({\mathbf{x}}^{0}-{\mathbf{z}}^{1})-\nabla s\left({\mathbf{x}}^{0}\right)=\mathbf{0}$ from Algorithm 1. Therefore, from the update of ${\mathbf{d}}^{k+1}$ in (12b), ${\mathbf{d}}^{k}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$ for all $k$. In fact, any ${\mathbf{z}}^{1}$ such that ${\mathbf{d}}^{1}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$ works for NIDS. The following two lemmas show the relation between fixed points of (12) and optimal solutions of (1). The proofs for all lemmas and propositions are included in the supplemental material.

Lemma 1** (Fixed point of (12))**

$({\mathbf{d}}^{*},{\mathbf{z}}^{*})$* is a fixed point of (12) if and only if there exists a subgradient ${\mathbf{q}}^{*}\in\partial r({\mathbf{x}}^{*})$ such that ${\mathbf{z}}^{*}={\mathbf{x}}^{*}+\Lambda{\mathbf{q}}^{*}$ and*

[TABLE]

Lemma 2** (Optimality condition)**

${\mathbf{x}}^{*}$* is consensual with $x^{*}_{1}=x^{*}_{2}=\cdots=x^{*}_{n}=x^{*}$ being an optimal solution of problem (1) if and only if there exists ${\mathbf{p}}^{*}$ and a subgradient ${\mathbf{q}}^{*}\in\partial r({\mathbf{x}}^{*})$ such that:*

[TABLE]

In addition, $({\mathbf{d}}^{*}=({\mathbf{I}}-{\mathbf{W}}){\mathbf{p}}^{*},{\mathbf{z}}^{*}={\mathbf{x}}^{*}+\Lambda{\mathbf{q}}^{*})$ is a fixed point of iteration (12).

Lemma 2 shows that we can find a fixed point of iteration (12) to obtain an optimal solution of problem (1). It also tells us that we need ${\mathbf{d}}^{*}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$ to get an optimal solution of problem (1). Therefore, we need ${\mathbf{d}}^{1}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$.

Lemma 3** (Norm over range space)**

For any symmetric positive semidefinite matrix ${\mathbf{A}}\in\mathbb{R}^{n\times n}$ with rank $r\leq n$, let $\lambda_{1}\geq\lambda_{2}\geq\dots\geq\lambda_{r}>0$ be its $r$ eigenvalues. Then ${\mathbf{range}}({\mathbf{A}})$ defined in Section I-E is a $rp$-dimensional subspace in $\mathbb{R}^{n\times p}$ and has a norm defined by $\|{\mathbf{x}}\|^{2}_{{\mathbf{A}}^{\dagger}}:=\langle{\mathbf{x}},{\mathbf{A}}^{\dagger}{\mathbf{x}}\rangle$, where ${\mathbf{A}}^{\dagger}$ is the pseudo inverse of ${\mathbf{A}}$. In addition, $\lambda_{1}^{-1}\|{\mathbf{x}}\|^{2}\leq\|{\mathbf{x}}\|^{2}_{{\mathbf{A}}^{\dagger}}\leq\lambda_{r}^{-1}\|{\mathbf{x}}\|^{2}$ for all ${\mathbf{x}}\in{\mathbf{range}}({\mathbf{A}})$.

Proposition 1

Let ${\mathbf{M}}=c^{-1}({\mathbf{I}}-{\mathbf{W}})^{\dagger}-\Lambda$ with ${\mathbf{I}}\succcurlyeq c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}\succcurlyeq\mathbf{0}$. Then $\|\cdot\|_{{\mathbf{M}}}$ is a norm defined for ${\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$.

The following lemma compares the distance to a fixed point of (12) for two consecutive iterates.

Lemma 4** (Fundamental inequality)**

Let $({\mathbf{d}}^{*},{\mathbf{z}}^{*})$ be a fixed point of iteration (12) with ${\mathbf{d}}^{*}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$. The update $({\mathbf{d}}^{k},{\mathbf{z}}^{k})\Rightarrow({\mathbf{d}}^{k+1},{\mathbf{z}}^{k+1})$ in (12) satisfies

[TABLE]

Proof:

From the update of ${\mathbf{z}}^{k+1}$ in (12c), we have

[TABLE]

where the second equality comes from (12b), (14b), and ${\mathbf{d}}^{k+1}-{\mathbf{d}}^{*}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$. From (12a), we have that

[TABLE]

Therefore, we have

[TABLE]

The inequality and the second equality comes from (17) and (16), respectively. The first and fourth equalities hold because of the update of ${\mathbf{z}}^{k+1}$ in (12c). Using $2\langle{\mathbf{a}},{\mathbf{b}}\rangle=\|{\mathbf{a}}+{\mathbf{b}}\|^{2}-\|{\mathbf{a}}\|^{2}-\|{\mathbf{b}}\|^{2}$ and rearranging the previous inequality give us that

[TABLE]

Therefore, (15) is obtained. ∎

IV-A Sublinear convergence of NIDS

As explained in Section II, NIDS is equivalent to the primal-dual algorithm [56] applied to problem

[TABLE]

where $\iota(\cdot)$ is the indicator function, which return 0 for $\mathbf{0}$ and $+\infty$ otherwise, with the metric matrix being

[TABLE]

We apply [56, Theorem 1] and obtain the following sublinear convergence result.

Theorem 1** (Sublinear rate)**

Let $({\mathbf{d}}^{k},{\mathbf{z}}^{k})$ be the sequence generated from NIDS in (12) with $\alpha_{i}<2/L_{i}$ for all $i$ and ${\mathbf{I}}\succcurlyeq c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$. We have

[TABLE]

Furthermore, $({\mathbf{d}}^{k},{\mathbf{z}}^{k})$ converges to a fixed point $(\bar{\mathbf{d}},\bar{\mathbf{z}})$ of iteration (12) and $\bar{\mathbf{d}}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$, if ${\mathbf{I}}\succ c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$.

Remark 3

Note the convergence in Theorem 1 is shown in ${\mathbf{z}}$ and ${\mathbf{d}}$. We will show the convergence in terms of (14). Recall that

[TABLE]

where ${\mathbf{q}}^{k}\in\partial r({\mathbf{x}}^{k})$. Therefore, $\|{\mathbf{z}}^{k+1}-{\mathbf{z}}^{k}\|_{\Lambda^{-1}}^{2}\rightarrow 0$ implies the convergence in terms of (14a).

Combining (12b) and (12c), we have

[TABLE]

Rearranging it gives

[TABLE]

Then we have

[TABLE]

where the second equality comes from ${\mathbf{d}}^{k+1}-{\mathbf{d}}^{k}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$. Thus $\|{\mathbf{z}}^{k+1}-{\mathbf{z}}^{k}\|_{\Lambda^{-1}}^{2}+\|{\mathbf{d}}^{k+1}-{\mathbf{d}}^{k}\|^{2}_{\mathbf{M}}\rightarrow 0$ implies the convergence in terms of (14b).

IV-B Linear convergence for special cases

In this subsection, we provide the linear convergence rate for the case when $r({\mathbf{x}})=0$, i.e., ${\mathbf{z}}^{k}={\mathbf{x}}^{k}$ in NIDS.

Theorem 2

If $\{s_{i}(x)\}_{i=1}^{n}$ are strongly convex with parameters $\{\mu_{i}\}_{i=1}^{n}$, then

[TABLE]

where ${\mathbf{S}}={\mathrm{Diag}}(\mu_{1},\cdots,\mu_{n})\in\mathbb{R}^{n\times n}$. Let $({\mathbf{d}}^{k},{\mathbf{x}}^{k})$ be the sequence generated from NIDS with $\alpha_{i}<2/L_{i}$ for all $i$ and ${\mathbf{I}}\succcurlyeq c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$. We define

[TABLE]

*and have *

[TABLE]

Proof:

From (15), we have

[TABLE]

For the two inner product terms, we have

[TABLE]

The first inequality comes from ${\mathbf{x}}^{k+1}={\mathbf{x}}^{k}-\Lambda\nabla s\left({\mathbf{x}}^{k}\right)-{\mathbf{d}}^{k+1}$, $\Lambda\nabla s\left({\mathbf{x}}^{*}\right)+{\mathbf{d}}^{*}=\mathbf{0}$, (7), and (20). Combing (23) and (32), we have

[TABLE]

Therefore,

[TABLE]

Since

[TABLE]

Let $\rho$ defined as (21), then we have (22). ∎

Remark 4

The condition ${\mathbf{I}}\succcurlyeq c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$ implies that $c\leq\lambda_{n-1}(\Lambda^{-1/2}({\mathbf{I}}-{\mathbf{W}})^{\dagger}\Lambda^{-1/2})$.

• •

If the agents across the whole network use an identical stepsize $\alpha$, that is, ${\Lambda}=\alpha{\mathbf{I}}$, then

[TABLE]

A concise but informative expression of the rate $\rho=\max\left(1-{\min_{i}\mu_{i}\over\max_{i}L_{i}},{\lambda_{2}({\mathbf{W}})-\lambda_{n}({\mathbf{W}})\over 1-\lambda_{n}({\mathbf{W}})}\right)$ can be obtained when we specifically choose $\alpha={1\over\max_{i}L_{i}}$ and $c={1\over(1-\lambda_{n}({\mathbf{W}}))\alpha}$. When $\lambda_{n}({\mathbf{W}})$ is not given, we choose $c={1/(2\alpha)}$ and obtain the scalability $\max\left({\max_{i}L_{i}\over\min_{i}\mu_{i}},{2\over 1-\lambda_{2}({\mathbf{W}})}\right)$. In this case, the network impact and the functional impact are decoupled.

• •

If we let ${\Lambda}={\mathbf{L}}^{-1}$ and $c=\lambda_{n-1}({\mathbf{L}}^{-1/2}({\mathbf{I}}-{\mathbf{W}})^{\dagger}{\mathbf{L}}^{-1/2})$, then the rate becomes

[TABLE]

*When $\lambda_{n}({\mathbf{W}})$ is not given, we choose $c=1/(2\max_{i}\alpha_{i})=\min_{i}L_{i}/2$ and obtain the scalability $\max\left(\max_{i}\frac{L_{i}}{\mu_{i}},\frac{\max_{i}L_{i}}{\min_{i}L_{i}}\cdot\frac{2}{1-\lambda_{2}({\mathbf{W}})}\right)$. In this case, the networking impact is coupled with the function factors, i.e., the smoothness heterogeneity $\frac{\max_{i}L_{i}}{\min_{i}L_{i}}$ is multiplied on the networking impact. While the other number depends on the functional condition numbers $\frac{L_{i}}{\mu_{i}}$’s only. *

Remark 5

*Theorem 2 separates the dependence of the linear convergence rate on the functions and the network structure. In our current scheme, all the agents perform information exchange and the proximal-gradient step once in each iteration. If the proximal-gradient step is expensive, this explicit rate formula can help us to decide whether the so-called multi-step consensus can help reducing the computational time. *

For the sake of simplicity, let us assume for this moment that all the agents have the same strong convexity constant $\mu$ and gradient Lipschitz continuity constant $L$. Suppose that the “$t$-step consensus” technique is employed, i.e., the mixing matrix ${\mathbf{W}}$ in our algorithm is replaced by ${\mathbf{W}}^{t}$, where $t$ is a positive integer. Then to reach $\epsilon$-accuracy, the number of iterations needed is

[TABLE]

When $L/\mu=1$ and step sizes are chosen as $\Lambda={\mathbf{L}}^{-1}$, it says that we should let $t\rightarrow+\infty$ if the graph is not a complete graph. Such theoretical result is correct in intuition since in this case, the centralized gradient descent only needs one step to reach optimal and the bottleneck in decentralized optimization is the network.

Suppose $t_{\max}$ is a reasonable upper bound on $t$, which is set by the system designer. It is difficult to explicitly find an optimal $t$. But with the above analysis as an evidence, we suggest that one choose $t=\min\left([\log_{\lambda_{2}({\mathbf{W}})}(1-\frac{\mu}{L})],t_{\max}\right)$ if $1-\frac{\mu}{L}>\lambda_{2}({\mathbf{W}})$; otherwise $t=1$. Here $[\cdot]$ gives the nearest integer.

If the bottleneck is on the functions, we can introduce a mapping $x=By$ and change the unknown variable from $x$ to $y$. E.g., if the function $s_{i}(x)$ is a composition of a convex function with a linear mapping, replacing $x$ using $y$ changes the linear mapping and the condition number $L/\mu$ of the function. When $B$ is diagonal, it is similar to the column normalization in machine learning applications. There are other possible ways for reducing the condition number of the functions. It is out of the scope of this work, and we leave this as future work.

V Numerical Experiments

In this section, we compare the performance of NIDS with several state-of-the-art algorithms for decentralized optimization. These methods are

• •

The EXTRA/PG-EXTRA (see (10));

• •

The DIGing-ATC [42]. For reference, the DIGing-ATC updates are provided as follows:

[TABLE]

• •

The accelerated distributed Nesterov gradient descent (Acc-DNGD-SC in [52]);

• •

The (dual friendly) optimal algorithm (OA) for distributed optimization (equation (7) in [54]).

Note there are two rounds of communication in each iteration of DIGing-ATC and Acc-DNGD-SC while there is only one round in that of EXTRA/NIDS/OA. For all the experiments, we first compute the exact solution ${\mathbf{x}}^{*}$ for (1) using the centralized (proximal) gradient descent. All networks are randomly generated with connectivity ratio $\tau$, where $\tau$ is defined as the number of actual edges divided by the total number of possible edges ${{n(n-1)}\over 2}$. We will report the specific $\tau$ used in each test. The mixing matrix ${\mathbf{W}}$ is always chosen with the Metropolis rule (see [57] and [36, Section 2.4]).

The experiments are carried in Matlab R2016b running on a laptop with Intel i7 CPU @ 2.60HZ, 16.0 GB of RAM, and Windows 10 operating system. The source codes for reproducing the numerical results can be accessed at https://github.com/mingyan08/NIDS.

V-A The strongly convex case with $r({\mathbf{x}})=0$

Consider the decentralized problem that solves for an unknown signal $x\in\mathbb{R}^{p}$. Each agent $i\in\{1,\cdots,n\}$ takes its own measurement via $y_{i}={\mathbf{M}}_{i}x+e_{i}$, where $y_{i}\in\mathbb{R}^{m_{i}}$ is the measurement vector, ${\mathbf{M}}_{i}\in\mathbb{R}^{m_{i}\times p}$ is the sensing matrix, and $e_{i}\in\mathbb{R}^{m_{i}}$ is the independent and identically distributed noise. To estimate $x$ collaboratively, we apply the decentralized algorithms to solve

[TABLE]

In order to ensure that each function ${1\over 2}\|{\mathbf{M}}_{i}x-y_{i}\|^{2}$ is strongly convex, we choose $m_{i}=60$ and $p=50$ and set the number of nodes $n=40$. For the first experiment, we choose ${\mathbf{M}}_{i}$ such that the Lipshchitz constant of $\nabla s_{i}$ satisfies $L_{i}=1$ and the strongly convex constant $\mu_{i}=0.5$ for all $i$. Based on Remark 4, we choose $\alpha={1/(\max_{i}L_{i})}=1$ and $c={1/(1-\lambda_{n}({\mathbf{W}}))}$ for NIDS. In addition, we choose $c={1/2}$ such that $\widetilde{{\mathbf{W}}}={{\mathbf{I}}+{\mathbf{W}}\over 2}$, which gives the same as that for EXTRA.

The comparison of these methods (NIDS with $c={1/((1-\lambda_{n}({\mathbf{W}}))\alpha)}$, NIDS with $c=1/2$, EXTRA, DIGing-ATC, Acc-DNGD-SC, and OA) is shown in Fig. 1 for two different networks with connectivity ratios $\tau=0.35$ (top) and $\tau=0.45$ (bottom), respectively. It shows better performance of NIDS in both choices of $c$ (corresponding to known ${\mathbf{W}}$ and unknown ${\mathbf{W}}$) than that of other algorithms. NIDS with $c={1/((1-\lambda_{n}({\mathbf{W}}))\alpha)}$ always takes less than half the number of iterations used by EXTRA to reach the same accuracy. In our experiment, DIGing-ATC appears to be sensitive to networks. Under a better connected network (see Fig. 1 bottom), DIGing-ATC can catch up with NIDS with $c={1/(2\alpha)}$. The theoretical step-size of Acc-DNGD-SC is too small due to a very small constant in the bound in [52], and the convergence of Acc-DNGD-DC under such theoretical step-size in our test is slow and uncompetitive. Thus we have carefully tuned its step-size. With the hand-optimized step-size, Acc-DNGD-SC can achieve a comparable performance as NIDS with $c={1/(2\alpha)}$. In the plots, we observe that OA is fast in terms of the number of iterations. However, in this case, the per-iteration cost of OA is relatively high since it requires solving a system of linear equations at each iteration (though factorization tricks may be used to save some computational time).

Next, we demonstrate the effort of uncoordinated/adaptive step-size. We construct the function with $\mu_{i}=0.02$ and $L_{i}=1$ for each node $i$. Then we change the $L_{i}$ values by multiplying the function by a constant.

We use the same mixing matrix for the following two experiments.

• •

We change half nodes. We randomly pick an even number node and multiply its function by $4$. For remaining even number nodes, we multiply their functions by a random integer 2 or 3.

• •

We change a quarter nodes. We randomly pick an node not in ${\mathcal{U}}=\{4,8,16,\dots,40\}$ and multiply its function by 10. Then for other nodes in ${\mathcal{U}}$, we multiply their functions by a random integer between 2 and 9.

We compare NIDS with adaptive step-size ($1/{L_{i}}$ for node $i$) and NIDS with same step-size $1/{\max_{i}L_{i}}$ in Fig. 2. We let $c=1/(2\max_{i}\alpha_{i})$, so no network information is needed. As shown in Fig. 2, NIDS with adaptive step-size converges faster than same step-size.

V-B The case with nonsmooth function $r({\mathbf{x}})$

In this subsection, we compare the performance of NIDS with PG-EXTRA [1] only since other methods in Section V-A, such as DIGing, can not be applied to this nonsmooth case. We consider a decentralized compressed sensing problem. Again, each agent $i\in\{1,\cdots,n\}$ takes its own measurement via $y_{i}={\mathbf{M}}_{i}x+e_{i}$, where $y_{i}\in\mathbb{R}^{m_{i}}$ is the measurement vector, ${\mathbf{M}}_{i}\in\mathbb{R}^{m_{i}\times p}$ is the sensing matrix, and $e_{i}\in\mathbb{R}^{m_{i}}$ is the independent and identically distributed noise. Here, $x$ is a sparse signal. The optimization problem is

[TABLE]

where the connectivity ratio of the network $\tau=0.1$. We normalize the problem to make sure that the Lipschitz constant satisfies $L_{i}=1$ for each node, we choose $m_{i}=3$ and $p=200$ and set the number of nodes $n=40$.

Fig. 3 shows that a larger step-size in NIDS leads to faster convergence. With step-size $1$, NIDS and PG-EXTRA converge at the same speed. But if we keep increasing the step-size, PG-EXTRA will diverge with step-size $1.4$ while the step-size of NIDS can be increased to $1.9$ maintaining convergence at a faster speed.

V-C An application in classification for healthcare data

We consider a decentralized sparse logistic regression problem to classify the colon-cancer data [58]. There are 62 samples, and each sample features 2,000 pieces of gene expressing information (numericalized and normalized [58]) and a binary outcome. The outcome can be normal/negative ($+1$) or tumor/positive ($-1$) and the data set contains $22$ normal and $40$ tumor colon tissue samples. We store the gene information in ${\mathbf{M}}_{i}\in\mathbb{R}^{1\times 2001}$ (one more dimension is augmented to take care of the linear offset/constant in the logit function model), and the outcome information is $y_{i}\in\{-1,1\}$, $i\in\mathcal{S}_{1}\bigcup\mathcal{S}_{2}$, where $\mathcal{S}_{1}$ serves for training while $\mathcal{S}_{2}$ serves for testing. Suppose we have a 50-node connected network where each node $i$ holds $1$ sample $({\mathbf{M}}_{i},y_{i})$ (the connected network is randomly generated and its connectivity ratio is set to $0.08$; the 50 in-network samples indexed by $\mathcal{S}_{1}$ are randomly drawn from the 62 samples). The decentralized logistic regression

[TABLE]

is then solved over the network to train a sparse linear classifier ${\mathbf{x}}^{*}$ for the outcome prediction of the remaining/future samples/data. In the optimization formulation, the $\ell_{2}$ norm term imposes strong convexity to $s$, while $\ell_{1}$ term promotes sparsity of the solution. Aside from the 50 samples used for training purpose, we randomly select 12 nodes from the 50 nodes to show the prediction performance of the remaining 12 samples in Fig. 4 left. The middle and right figures in Fig. 4 show how the consensus error $\left\|{\mathbf{x}}^{k}\right\|_{I-{\mathbf{W}}}$ and the sparsity of the average solution vector ${1\over 50}\sum_{i=1}^{50}x_{i}^{k}$ drops, respectively.

VI Conclusion

We proposed a novel decentralized consensus algorithm NIDS, whose step-size does not depend on the network structure. In NIDS, the step-size depends only on the objective function, and it can be as large as $2/L$, where $L$ is the Lipschitz constant of the gradient of the smooth function. We showed that NIDS converges at the $o(1/k)$ rate for the general convex case and at a linear rate for the strongly convex case. For the strongly convex case, we separated the linear convergence rate’s dependence on the objective function and the network. The separated convergence rates match the typical rates for the general gradient descent and the consensus averaging. Furthermore, every agent in the network can choose its own step-size independently by its own objective function. Numerical experiments validated the theoretical results and demonstrated better performance of NIDS over state-of-the-art algorithms. Because the step-size of NIDS does not depend on the network structure, there are many possible future extensions. One extension is to apply NIDS on dynamic networks where nodes can join and drop off.

Acknowledgement

We thank the anonymous reviewers for helpful comments and suggestions to improve the clarity of this paper.

Supplementary Material for “A Decentralized Proximal-Gradient Methodwith Network Independent Step-sizes andSeparated Convergence Rates”

VI-A Proof of Lemma 1

Lemma 1** (Fixed point of (12))**

$({\mathbf{d}}^{*},{\mathbf{z}}^{*})$* is a fixed point of (12) if and only if there exists a subgradient ${\mathbf{q}}^{*}\in\partial r({\mathbf{x}}^{*})$ such that ${\mathbf{z}}^{*}={\mathbf{x}}^{*}+\Lambda{\mathbf{q}}^{*}$ and*

[TABLE]

Proof:

“$\Rightarrow$” If $({\mathbf{d}}^{*},{\mathbf{z}}^{*})$ is a fixed point of (12), we have

[TABLE]

where the two equalities come from (12b) and (12c), respectively. Combining (12c) and (12a) gives

[TABLE]

where ${\mathbf{q}}^{*}\in\partial r({\mathbf{x}}^{*})$.

“$\Leftarrow$” In order to show that $({\mathbf{d}}^{*},{\mathbf{z}}^{*})$ is a fixed point of iteration (12), we just need to verify that $({\mathbf{d}}^{k+1},{\mathbf{z}}^{k+1})=({\mathbf{d}}^{*},{\mathbf{z}}^{*})$ if $({\mathbf{d}}^{k},{\mathbf{z}}^{k})=({\mathbf{d}}^{*},{\mathbf{z}}^{*})$. From (12a), we have ${\mathbf{x}}^{k}={\mathbf{x}}^{*}$, then

[TABLE]

Therefore, $({\mathbf{d}}^{*},{\mathbf{z}}^{*})$ is a fixed point of iteration (12). ∎

VI-B Proof of Lemma 2

Lemma 2** (Optimality condition)**

${\mathbf{x}}^{*}$* is consensual with $x^{*}_{1}=x^{*}_{2}=\cdots=x^{*}_{n}=x^{*}$ being an optimal solution of problem (1) if and only if there exists ${\mathbf{p}}^{*}$ and a subgradient ${\mathbf{q}}^{*}\in\partial r({\mathbf{x}}^{*})$ such that:*

[TABLE]

In addition, $({\mathbf{d}}^{*}=({\mathbf{I}}-{\mathbf{W}}){\mathbf{p}}^{*},{\mathbf{z}}^{*}={\mathbf{x}}^{*}+\Lambda{\mathbf{q}}^{*})$ is a fixed point of iteration (12).

Proof:

“$\Rightarrow$” Because ${\mathbf{x}}^{*}={\mathbf{1}}_{n\times 1}(x^{*})^{\top}$, we have $({\mathbf{I}}-{\mathbf{W}}){\mathbf{x}}^{*}=({\mathbf{I}}-{\mathbf{W}}){\mathbf{1}}_{n\times 1}(x^{*})^{\top}=\mathbf{0}_{n\times 1}(x^{*})^{\top}=\mathbf{0}$. The fact that $x^{*}$ is an optimal solution of problem (1) means there exists ${\mathbf{q}}^{*}\in\partial r({\mathbf{x}}^{*})$ such that $(\nabla s({\mathbf{x}}^{*})+{\mathbf{q}}^{*})^{\top}{\mathbf{1}}_{n\times 1}=\mathbf{0}$. That is to say all columns of $\nabla s({\mathbf{x}}^{*})+{\mathbf{q}}^{*}$ are orthogonal to ${\mathbf{1}}_{n\times 1}$. Therefore, Remark 1 shows the existence of ${\mathbf{p}}^{*}$ such that $({\mathbf{I}}-{\mathbf{W}}){\mathbf{p}}^{*}+\nabla s({\mathbf{x}}^{*})+{\mathbf{q}}^{*}=\mathbf{0}$.

“$\Leftarrow$” Equation (34b) shows that ${\mathbf{x}}^{*}$ is consensual because of item 3 of Assumption 1, i.e., ${\mathbf{x}}^{*}={\mathbf{1}}_{n\times 1}(x^{*})^{\top}$ for some $x^{*}$. From (34a), we have $\mathbf{0}=(({\mathbf{I}}-{\mathbf{W}}){\mathbf{p}}^{*}+\nabla s({\mathbf{x}}^{*})+{\mathbf{q}}^{*})^{\top}{\mathbf{1}}_{n\times 1}=({\mathbf{p}}^{*})^{\top}({\mathbf{I}}-{\mathbf{W}}){\mathbf{1}}_{n\times 1}+(\nabla s({\mathbf{x}}^{*})+{\mathbf{q}}^{*})^{\top}{\mathbf{1}}_{n\times 1}=(\nabla s({\mathbf{x}}^{*})+{\mathbf{q}}^{*})^{\top}{\mathbf{1}}_{n\times 1}$. Thus, $\mathbf{0}\in\sum_{i=1}^{n}(\nabla s_{i}(x^{*})+\partial r_{i}(x^{*}))$ because ${\mathbf{x}}^{*}$ is consensual. This completes the proof for the equivalence.

Lemma 1 shows that $({\mathbf{d}}^{*}=({\mathbf{I}}-{\mathbf{W}}){\mathbf{p}}^{*},{\mathbf{z}}^{*}={\mathbf{x}}^{*}+\Lambda{\mathbf{q}}^{*})$ is a fixed point of iteration (12). ∎

VI-C Proof of Lemma 3

Lemma 3** (Norm over range space)**

For any symmetric positive semidefinite matrix ${\mathbf{A}}\in\mathbb{R}^{n\times n}$ with rank $r\leq n$, let $\lambda_{1}\geq\lambda_{2}\geq\dots\geq\lambda_{r}>0$ be its $r$ eigenvalues. Then ${\mathbf{range}}({\mathbf{A}})$ defined in Section I-E is a $rp$-dimensional subspace in $\mathbb{R}^{n\times p}$ and has a norm defined by $\|{\mathbf{x}}\|^{2}_{{\mathbf{A}}^{\dagger}}:=\langle{\mathbf{x}},{\mathbf{A}}^{\dagger}{\mathbf{x}}\rangle$, where ${\mathbf{A}}^{\dagger}$ is the pseudo inverse of ${\mathbf{A}}$. In addition, $\lambda_{1}^{-1}\|{\mathbf{x}}\|^{2}\leq\|{\mathbf{x}}\|^{2}_{{\mathbf{A}}^{\dagger}}\leq\lambda_{r}^{-1}\|{\mathbf{x}}\|^{2}$ for all ${\mathbf{x}}\in{\mathbf{range}}({\mathbf{A}})$.

Proof:

Let ${\mathbf{A}}={\mathbf{U}}\Sigma{\mathbf{U}}^{\top}$, where $\Sigma={\mathrm{Diag}}(\lambda_{1},\lambda_{2},\cdots,\lambda_{r})$ and the columns of ${\mathbf{U}}$ are orthonormal eigenvectors for corresponding eigenvalues, i.e., ${\mathbf{U}}\in\mathbb{R}^{n\times r}$ and ${\mathbf{U}}^{\top}{\mathbf{U}}={\mathbf{I}}_{r\times r}$. Then ${\mathbf{A}}^{\dagger}={\mathbf{U}}\Sigma^{-1}{\mathbf{U}}^{\top}$, where $\Sigma^{-1}={\mathrm{Diag}}(\lambda_{1}^{-1},\lambda_{2}^{-1},\cdots,\lambda_{r}^{-1})$.

Letting ${\mathbf{x}}={\mathbf{A}}{\mathbf{y}}$, we have $\|{\mathbf{x}}\|^{2}=\langle{\mathbf{U}}\Sigma{\mathbf{U}}^{\top}{\mathbf{y}},{\mathbf{U}}\Sigma{\mathbf{U}}^{\top}{\mathbf{y}}\rangle=\langle\Sigma{\mathbf{U}}^{\top}{\mathbf{y}},\Sigma{\mathbf{U}}^{\top}{\mathbf{y}}\rangle=\|\Sigma{\mathbf{U}}^{\top}{\mathbf{y}}\|^{2}$. In addition,

[TABLE]

Therefore,

[TABLE]

which means that $\|\cdot\|^{2}_{{\mathbf{A}}^{\dagger}}=\langle\cdot,{\mathbf{A}}^{\dagger}\cdot\rangle$ is a norm for ${\mathbf{range}}({\mathbf{A}})$. ∎

VI-D *Proof of Proposition 1 *

Proposition 1

Let ${\mathbf{M}}=c^{-1}({\mathbf{I}}-{\mathbf{W}})^{\dagger}-\Lambda$ with $\Lambda$ being symmetric positive definite and ${\mathbf{I}}\succcurlyeq c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}\succcurlyeq\mathbf{0}$. Then $\|\cdot\|_{{\mathbf{M}}}$ is a norm defined for ${\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$.

Proof:

Rewrite the matrix ${\mathbf{M}}$ as

[TABLE]

For any ${\mathbf{x}}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$, we can find ${\mathbf{y}}\in\mathbb{R}^{n\times p}$ such that ${\mathbf{x}}=({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}{\mathbf{y}}$. Then

[TABLE]

We apply Lemma 3 on $\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$ and obtain the result. ∎

VI-E Proof of Theorem 1

Before proving Theorem 1, we present two lemmas. The first lemma shows that the distance to a fixed point of (12) is decreasing, and the second one show the distance between two iterations is decreasing.

Lemma 4** (A key inequality of descent)**

Let $({\mathbf{d}}^{*},{\mathbf{z}}^{*})$ be a fixed point of (12) and ${\mathbf{d}}^{*}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$. For the sequence $({\mathbf{d}}^{k},{\mathbf{z}}^{k})$ generated from NIDS in (12) with ${\mathbf{I}}\succcurlyeq c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$, we have

[TABLE]

Proof:

Young’s inequality and (7) give us

[TABLE]

Therefore, from (15), we have

[TABLE]

This completes the proof. ∎

Lemma 5** (Monotonicity of successive difference in a special norm)**

Let $({\mathbf{d}}^{k},{\mathbf{z}}^{k})$ be the sequence generated from NIDS in (12) with $\alpha_{i}<2/L_{i}$ for all $i$ and ${\mathbf{I}}\succcurlyeq c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$. Then the sequence $\left\{\|{\mathbf{z}}^{k+1}-{\mathbf{z}}^{k}\|^{2}_{\Lambda^{-1}}+\|{\mathbf{d}}^{k+1}-{\mathbf{d}}^{k}\|_{{\mathbf{M}}}^{2}\right\}_{k\geq 0}$ is monotonically nonincreasing.

Proof:

Similar to the proof for Lemma 4, we can show that

[TABLE]

Subtracting (LABEL:for:nonincreasing2) from (LABEL:for:nonincreasing1) on both sides, we have

[TABLE]

where the inequality comes from the Cauchy-Schwarz inequality. Then, the previous inequality, together with (39) and the Cauchy-Schwarz inequality, gives

[TABLE]

and consequently

[TABLE]

The three inequalities hold because of (39), (40), and the Cauchy-Schwarz inequality, respectively. The first and third equalities come from (12c). Rearranging the previous inequality, we obtain

[TABLE]

where the second and last inequalities come from (7) and $\Lambda<2{\mathbf{L}}^{-1}$, respectively. It completes the proof. ∎

Theorem 1** (Sublinear rate)**

Let $({\mathbf{d}}^{k},{\mathbf{z}}^{k})$ be the sequence generated from NIDS in (12) with $\alpha_{i}<2/L_{i}$ for all $i$ and ${\mathbf{I}}\succcurlyeq c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$. We have

[TABLE]

Furthermore, $({\mathbf{d}}^{k},{\mathbf{z}}^{k})$ converges to a fixed point $(\bar{\mathbf{d}},\bar{\mathbf{z}})$ of iteration (12) and $\bar{\mathbf{d}}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$, if ${\mathbf{I}}\succ c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$.

Proof:

Lemma 5 shows that $\left\{\|{\mathbf{z}}^{k+1}-{\mathbf{z}}^{k}\|^{2}_{\Lambda^{-1}}+\|{\mathbf{d}}^{k+1}-{\mathbf{d}}^{k}\|_{{\mathbf{M}}}^{2}\right\}_{k\geq 0}$ is monotonically nonincreasing. Summing up (4) from $1$ to $k$, we have

[TABLE]

Therefore, we have

[TABLE]

and [59, Lemma 1] gives us (41).

When ${\mathbf{I}}\succ c\Lambda^{1/2}({\mathbf{I}}-{\mathbf{W}})\Lambda^{1/2}$, inequality (4) shows that the sequence $({\mathbf{d}}^{k},{\mathbf{z}}^{k})$ is bounded, and there exists a convergent subsequence $({\mathbf{d}}^{k_{i}},{\mathbf{z}}^{k_{i}})$ such that $({\mathbf{d}}^{k_{i}},{\mathbf{z}}^{k_{i}})\rightarrow(\bar{\mathbf{d}},\bar{\mathbf{z}})$. Then (41) gives the convergence of $({\mathbf{d}}^{k_{i}+1},{\mathbf{z}}^{k_{i}+1})$. More specifically, $({\mathbf{d}}^{k_{i}+1},{\mathbf{z}}^{k_{i}+1})\rightarrow(\bar{\mathbf{d}},\bar{\mathbf{z}})$. Therefore $(\bar{\mathbf{d}},\bar{\mathbf{z}})$ is a fixed point of iteration (12). In addition, because ${\mathbf{d}}^{k}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$ for all $k$, we have $\bar{\mathbf{d}}\in{\mathbf{range}}({\mathbf{I}}-{\mathbf{W}})$. Finally Lemma 4 implies the convergence of $({\mathbf{d}}^{k},{\mathbf{z}}^{k})$ to $(\bar{\mathbf{d}},\bar{\mathbf{z}})$. ∎