Annealing for Distributed Global Optimization

TL;DR

This paper introduces an annealing-based distributed algorithm for nonconvex optimization in multi-agent networks, proving convergence to global optima despite network variability and local nonconvexity.

Contribution

It presents a novel consensus+innovations algorithm with annealing and noise decay, ensuring convergence to global minima in distributed nonconvex optimization.

Findings

Convergence to global optima is proven under certain network and cost conditions.

The algorithm achieves convergence in probability to the set of global minima.

The approach handles time-varying communication graphs and local nonconvex functions.

Abstract

The paper proves convergence to global optima for a class of distributed algorithms for nonconvex optimization in network-based multi-agent settings. Agents are permitted to communicate over a time-varying undirected graph. Each agent is assumed to possess a local objective function (assumed to be smooth, but possibly nonconvex). The paper considers algorithms for optimizing the sum function. A distributed algorithm of the consensus+innovations type is proposed which relies on first-order information at the agent level. Under appropriate conditions on network connectivity and the cost objective, convergence to the set of global optima is achieved by an annealing-type approach, with decaying Gaussian noise independently added into each agent's update step. It is shown that the proposed algorithm converges in probability to the set of global minima of the sum function.

Equations

$$U(x)=\frac{1}{N}\sum _{n=1}^N{U}_n(x)$$

$$U_n(\theta )={\mathbb{E}}_{D_n}[l_n(\theta ,z_n)],$$

$${\mathbf{x}}_n(t+1)=$$

$$-{\alpha }_t\left(\nabla U_n(x_n(t))+{\mathbf{\zeta }}_n(t)\right)+{\gamma }_t{\mathbf{w}}_n(t),$$

$$\pi (f)\dot{=}\int fd\pi ,$$

$${\mathbb{E}}_{t_0,z_0}[f(Z_t)]\dot{=}\mathbb{E}[f(Z_t)|Z_{t_0}=z_0].$$

$${\Omega }_n=\left\{l\in V|(n,l)\in E\right\}.$$

$${\mathbf{x}}_{t+1}={\mathbf{x}}_t$$

$$-{\alpha }_t\left(\nabla \overline{U}({\mathbf{x}}_t)+{\mathbf{\zeta }}_t\right)+{\gamma }_t{\mathbf{w}}_t,$$

$$\Vert \nabla U_n(\mathbf{x})-\nabla U_n(\acute{\mathbf{x}})\Vert \le L\Vert \mathbf{x}-\acute{\mathbf{x}}\Vert$$

$$\underset{\mathbf{x}\in {\mathbb{R}}^d}{\sup }\Vert \nabla U_n(\mathbf{x})-\nabla U(\mathbf{x})\Vert <\infty ,\forall n.$$

$${\mathcal{H}}_t=\sigma \left({\mathbf{x}}_0,L_0,\cdots ,L_{t-1},{\mathbf{\zeta }}_0,\cdots ,{\mathbf{\zeta }}_{t-1},{\mathbf{w}}_0,\cdots ,{\mathbf{w}}_{t-1}\right).$$

$$\mathbb{E}[{\mathbf{\zeta }}_t|{\mathcal{H}}_t]=0\text{mbox}and\mathbb{E}[\Vert {\mathbf{\zeta }}_t{\Vert }^2|{\mathcal{H}}_t]<C_1$$

$$\displaystyle\alpha_{t}=\frac{c_{\alpha}}{t},~{}~{}\beta_{t}=\frac{c_{\beta}}{t^{\tau_{\beta}}},~{}~{}\gamma_{t}=\frac{c_{\gamma}}{t^{1/2}\sqrt{\log\log t}},~{}~{}~{}\mbox{for $t$ large},$$

$$\mathbb{P}\left((t+1{)}^{-1/2-\delta }\Vert {\mathbf{\zeta }}_t\Vert >\epsilon \right)$$

$$\le \frac{C_1}{{\epsilon }^2(t+1{)}^{1+2\delta }}.$$

$$\mathbb{P}\left((t+1{)}^{-1/2-\delta }\Vert {\mathbf{\zeta }}_t\Vert >\epsilon \text{mbox}i.o.\right)=0,$$

$${\mathbf{z}}_{t+1}\le \left(1-r_1(t)\right){\mathbf{z}}_t+r_2(t)V_t\left(1+J_t\right).$$

$$\frac{a_1}{(t+1{)}^{{\delta }_1}}\le \mathbb{E}\left[r_1(t)|{\mathcal{H}}_t\right]\le 1$$

$$0\le {\delta }_0<{\delta }_2-{\delta }_1-\frac{1}{2+{\epsilon }_1},$$

$$\displaystyle\mathcal{C}\dot{=}\left\{\mathbf{z}\in\mathbb{R}^{Nd}~{}:~{}\mathbf{z}=\mathbf{1}_{N}\otimes\mathbf{a}~{}\mbox{for some $\mathbf{a}\in\mathbb{R}^{d}$}\right\},$$

$$={\lambda }_2\left(\mathbb{E}[L_t]\right)={\lambda }_2(\overline{L})>0,$$

$$\Vert \left(I_{Nd}-{\beta }_t{L}_t\otimes I_d\right){\mathbf{z}}_t\Vert \le \left(1-r_t\right)\Vert {\mathbf{z}}_t\Vert$$

$$\mathbb{E}\left[r_t|{\mathcal{H}}_t\right]\ge \frac{c_r}{(t+1{)}^{{\tau }_{\beta }}}\text{mbox}a.s.$$

$$\mathbb{P}\left(\underset{t\to \infty }{\lim }(t+1{)}^{\tau }\Vert {\mathbf{x}}_n(t)-{\overline{\mathbf{x}}}_t\Vert =0\right)=1,\forall n,$$

$${\overline{\mathbf{x}}}_{t+1}={\overline{\mathbf{x}}}_t-{\alpha }_t\left(\frac{1}{N}\sum _{n=1}^N\nabla U_n({\mathbf{x}}_n(t))+{\overline{\mathbf{\zeta }}}_t\right)+{\gamma }_t{\overline{\mathbf{w}}}_t,$$

$${\overline{\mathbf{\zeta }}}_t=\frac{1}{N}\sum _{n=1}^N{\mathbf{\zeta }}_n(t),\text{mbox}and{\overline{\mathbf{w}}}_t=\frac{1}{N}\sum _{n=1}^N{\mathbf{w}}_n(t).$$

$${\mathcal{P}}_{Nd}{\breve{\mathbf{x}}}_t=0,\forall t,$$

$${\breve{\mathbf{x}}}_{t+1}$$

$$\displaystyle-\alpha_{t}\underbrace{\begin{bmatrix}\nabla U_{1}(\mathbf{x}_{1}(t))-\frac{1}{N}\sum_{l=1}^{N}\nabla U_{l}(\mathbf{x}_{l}(t))\\ \nabla U_{2}(\mathbf{x}_{2}(t))-\frac{1}{N}\sum_{l=1}^{N}\nabla U_{l}(\mathbf{x}_{l}(t))\\ \vdots\\ \vdots\\ \nabla U_{N}(\mathbf{x}_{N}(t))-\frac{1}{N}\sum_{l=1}^{N}\nabla U_{l}(\mathbf{x}_{l}(t))\end{bmatrix}}_{\mathrm{\mbox{$T_{1}$}}}$$

$$\displaystyle-\alpha_{t}\underbrace{\left(\boldsymbol{\zeta}_{t}-\mathbf{1}_{N}\otimes\overline{\boldsymbol{\zeta}}_{t}\right)}_{\mathrm{\mbox{$T_{2}$}}}+\gamma_{t}\underbrace{\left(\mathbf{w}_{t}-\mathbf{1}_{N}\otimes\overline{\mathbf{w}}_{t}\right)}_{\mathrm{\mbox{$T_{3}$}}}$$

$$T_1^n\doteq \nabla U_n({\mathbf{x}}_n(t))-\frac{1}{N}\sum _{l=1}^N\nabla U_l({\mathbf{x}}_l(t)),$$

$$T_1^n$$

Full text

I Introduction

In this paper we consider a class of algorithms for nonconvex optimization in distributed multi-agent systems and prove convergence to the set of global optima. Recent years have seen a surge in research interest in nonconvex optimization, motivated, to a large degree, by emerging applications in machine learning and artificial intelligence. The majority of research in this area has focused on centralized computing frameworks in which memory and processing resources are either shared or coordinated by a central mechanism [1, 2, 3, 4, 5, 6, 7, 8, 9].

With the advent of the internet of things (IoT) and low-latency 5G communication networks, there is a growing trend towards storing and processing data at the “edge” of the network (e.g., directly on IoT devices) rather than processing data in the cloud. This necessitates algorithms that are able to operate robustly in adhoc networked environments without centralized coordination. Beyond applications in IoT, distributed algorithms for non-convex optimization also play an important role in other domains, including power systems [10], sensor networks [11], unmanned aerial vehicles [12], and wireless communications [13].

This paper considers the following distributed computation framework: A group of $N$ agents (or nodes) communicates over a (possibly random, possibly sparse) undirected communication graph $G$. Each agent has a local objective function $U_{n}:\mathbb{R}^{d}\to\mathbb{R}$. We are interested in distributed algorithms that optimize the sum function

[TABLE]

using only local neighborhood information exchange between agents and without any centralized coordination.

As an example, in the context of distributed risk minimization or probably approximately correct (PAC) learning, e.g., [14], the $U_{n}(\cdot)$’s may correspond to (expected) risk

[TABLE]

where $l_{n}(\cdot,\cdot)$ is the local loss function at agent $n$ and $D_{n}$ is the local data distribution. The agents are interested in learning a common “hypothesis,” parameterized by $\theta$, using their collective data.

Distributed optimization algorithms have been studied extensively when the objective functions are convex [15, 16, 17, 18, 19, 20]. Not so when the objective is non-convex. The majority of current work in this area focuses on demonstrating convergence of distributed algorithms to critical points of $U$ (not necessarily to minima, local or global).

This motivates us to consider a class of distributed algorithms for computing the global optima of (1). Our algorithms take the form:

[TABLE]

$n=1,\ldots,N$, where ${\bf x}_{n}(t)\in\mathbb{R}^{d}$ is the state of agent $n$ at iteration $t\geq 0$, $\Omega_{n}(t)$ denotes the set of agents neighboring agent $n$ at time $t$ (per the communication graph), $\{\alpha_{t}\}$ and $\{\beta_{t}\}$ are sequences of decaying weight parameters, $\{\gamma_{t}\}$ is a sequence of decaying annealing weights, $\zeta_{n}$(t) is a $d$-dimensional random variable (representing gradient noise), and ${\bf w}_{n}(t)$ is a $d$-dimensional Gaussian noise (introduced for annealing). The algorithm is distributed since in (2) each agent only knows its local function $U_{n}(\cdot)$ and accesses information on the state of neighboring agents.

Algorithm (2) may be viewed as a distributed consensus + innovations algorithm [21]. The algorithm consists of the consensus term, $-\beta_{t}\sum_{l\in\Omega_{n}(t)}\left(\mathbf{x}_{n}(t)-\mathbf{x}_{l}(t)\right)$, that encourages agreement among agents, and the innovation term, $-\alpha_{t}\left(\nabla U_{n}(x_{n}(t))+\boldsymbol{\zeta}_{n}(t)\right)$, that encourages each agent to follow the gradient descent direction of their local objective function (with $\boldsymbol{\zeta}_{t}$ being zero-mean gradient noise). Finally, the term $\gamma_{t}\mathbf{w}_{n}(t)$ is an annealing term that injects decaying Gaussian noise into the dynamics to destabilize local minima and saddle points. By appropriately controlling the decay rates of the parameter sequences, one can balance the various objectives of reaching consensus among agents, reaching a critical point of (1), and destabilizing local minima and saddle points (see Assumption 6).

Our main contribution is the following: We show that, under appropriate assumptions (outlined below), the distributed algorithm (2) converges in probability to the set of global minima of (1). More precisely, it will be shown that (i) agents reach consensus, almost surely (a.s.), i.e., $\lim_{t\to\infty}\|{\bf x}_{n}(t)-{\bf x}_{\ell}(t)\|=0$ for each $n,\ell=1,\ldots,N$, a.s., and (ii) for each agent $n$, ${\bf x}_{n}(t)$ converges in probability to the set of global minima of $U(\cdot)$. A precise statement of the main result is given in Theorem 2 at the end of Section IV.

Theorem 2 is proved under Assumptions 1–11. Assumptions 1–2 and 7–11 concern the agents’ objective functions, Assumption 3 concerns the time-varying communication graph, Assumptions 4–5 concern the gradient annealing noise, and Assumption 6 concerns the weight parameter sequences.

Related Work. Work on distributed optimization with convex objectives has been studied extensively; for an overview of the expansive literature in this field we refer readers to [15, 16, 17, 18, 19, 20] and references therein.

The topic of distributed algorithms for non-convex optimization is a subject of more recent research focus. We briefly summarize related contributions here. Reference [13] considers an algorithm for nonconvex optimization (possibly constrained) over an undirected communication graph and shows convergence to KKT points. Relevant applications to wireless adhoc networks are discussed. Reference [22] considers a distributed primal dual algorithm for nonconvex optimization. The primal dual algorithm solves an approximation to the original nonconvex problem. Reference [23] analyzes the alternating direction penalty method and method of multipliers in nonconvex problems and demonstrates convergence to primal feasible points under mild assumptions. Reference [24] considers a push-sum algorithm for distributed nonconvex optimization on time-varying directed graphs and demonstrates convergence to first-order stationary points. [25] considers a distributed algorithm for nonconvex optimization with smooth objective and possibly non-smooth regularizer and demonstrates convergence to stationary solutions. Our work differs from these primarily in that we study distributed algorithms for global optimization of a nonconvex function.

The key feature of this approach is the incorporation of decaying Gaussian noise that allows the algorithm to escape local minima. Such techniques were explored in [8] and later studied and successfully applied in various centralized settings; e.g., [26, 27, 28, 29, 30] and references therein. On the other hand, consensus + innovations techniques, such as those used in [21, 31], are used in distributed settings. In this paper we prove global optimal convergence for consensus + innovations techniques, with an appropriate annealing schedule, in nonconvex optimization. This results in a distributed equivalent of the centralized result in [8].

As noted in [32], the analysis techniques developed to study consensus + innovations algorithms contributes to the general theory of mixed-time-scale stochastic approximation (SA) algorithms, e.g., [33]. In such algorithms, the right-hand side of the stochastic approximation difference equation contains two potentials decaying at different rates. The work [8] studies mixed-time scale SA algorithms in the context of simulated annealing. In [8], the term that serves a role analogous to our innovations potential is assumed to converge asymptotically to a Martingale difference process. A key element of our analysis here is to characterize the rate at which the innovation potential converges to a Martingale difference sequence in order to apply the results of [8].

Organization. The remainder of the paper is organized as follows. Section I-A introduces relevant notation. Section II formally introduces our distributed algorithm. Section III presents the assumptions used in our main result and some intermediate results, and it reviews a classical result in global optimization (Theorem 1) that will be used in the proof of our main result. Section IV proves the main result (Theorem 2).

I-A Notation

The set of reals is denoted by $\mathbb{R}$, whereas $\mathbb{R}_{+}$ denotes the non-negative reals. For $a,b\in\mathbb{R}$, we will use the notations $a\vee b$ and $a\wedge b$ to denote the maximum and minimum of $a$ and $b$ respectively. We denote the $k$-dimensional Euclidean space by $\mathbb{R}^{k}$. The set of $k\times k$ real matrices is denoted by $\mathbb{R}^{k\times k}$. The $k\times k$ identity matrix is denoted by $I_{k}$, while $\mathbf{1}_{k}$ and $\mathbf{0}_{k}$ denote respectively the column vector of ones and zeros in $\mathbb{R}^{k}$. Often the symbol [math] is used to denote the $k\times p$ zero matrix, the dimensions being clear from the context. The operator $\left\|\cdot\right\|$ applied to a vector denotes the standard Euclidean $\mathcal{L}_{2}$ norm, while applied to matrices it denotes the induced $\mathcal{L}_{2}$ norm, which is equivalent to the matrix spectral radius for symmetric matrices. The notation $A\otimes B$ is used for the Kronecker product of two matrices $A$ and $B$. We say that a function $f$ is of class $C^{k}$, $k\geq 1$, if $f$ is $k$-times continuously differentiable.

Given a set of elements in $z_{1},\ldots,z_{N}$ belonging to some Euclidean space, we let $\boldsymbol{\operatorname{Vec}}(\{z_{i}\}_{i=1}^{N})$ denote the vector stacking these elements. To simplify notation, we sometimes suppress the interior brackets when the meaning is clear.

We assume there exists a rich enough probability space to carry out the constructions of the random objects defined in the paper. Unless stated otherwise, all (in)equalities involving random objects are to be interpreted almost surely (a.s.). We denote by $\mathbb{P}$ and $\mathbb{E}$ probability and expectation respectively. Given a measure $\pi$ on $\mathbb{R}^{k}$ and a (measurable) function $f:\mathbb{R}^{k}\mapsto\mathbb{R}$, we let

[TABLE]

whenever the integral exists. For a stochastic process $\{Z_{t}\}$ and a function $f$, we let

[TABLE]

Spectral graph theory: The inter-agent communication topology may be described by an undirected graph $G=(V,E)$, with $V=\left[1\cdots N\right]$ and $E$ the set of agents (nodes) and communication links (edges), respectively. The unordered pair $(n,l)\in E$ if there exists an edge between nodes $n$ and $l$. We consider simple graphs, i.e., graphs devoid of self-loops and multiple edges. A graph is connected if there exists a path111A path between nodes $n$ and $l$ of length $m$ is a sequence $(n=i_{0},i_{1},\cdots,i_{m}=l)$ of vertices, such that $(i_{k},i_{k+1})\in E\>\forall~{}0\leq k\leq m-1$., between each pair of nodes. The neighborhood of node $n$ is

[TABLE]

Node $n$ has degree $d_{n}=|\Omega_{n}|$ (the number of edges with $n$ as one end point). The structure of the graph can be described by the symmetric $N\times N$ adjacency matrix, $A=\left[A_{nl}\right]$, $A_{nl}=1$, if $(n,l)\in E$, $A_{nl}=0$, otherwise. Let the degree matrix be the diagonal matrix $D=\mbox{diag}\left(d_{1}\cdots d_{N}\right)$. The positive semidefinite matrix $L=D-A$ is the graph Laplacian matrix. The eigenvalues of $L$ can be ordered as $0=\lambda_{1}(L)\leq\lambda_{2}(L)\leq\cdots\leq\lambda_{N}(L)$, the eigenvector corresponding to $\lambda_{1}(L)$ being $(1/\sqrt{N})\mathbf{1}_{N}$. The multiplicity of the zero eigenvalue equals the number of connected components of the network; for a connected graph, $\lambda_{2}(L)>0$. This second eigenvalue is the algebraic connectivity or the Fiedler value of the network; see [34] for detailed treatment of graphs and their spectral theory.

II Algorithm

Consider $N$ agents connected over a time-varying graph, with $L_{t}$ denoting the graph Laplacian at time $t$. Let $U_{n}:\mathbb{R}^{d}\to\mathbb{R}$, $n=1,\ldots,N$ denote the objective function of agent $n$. Let $U:\mathbb{R}^{d}\to\mathbb{R}$ be as defined in (1).

The agents update their states in a distributed fashion according to (2) for all $t\geq 0$ with deterministic initial conditions $\mathbf{x}_{n}(0)\in\mathbb{R}^{d}$, $n=1,\cdots,N$. In (2), $\boldsymbol{\zeta}_{n}(t)$ denotes gradient noise and $\mathbf{w}_{n}(t)$ denotes a standard normal vector (introduced for annealing). In vector form, the update in (2) may be written as:

[TABLE]

where $\mathbf{x}_{t}=\boldsymbol{\operatorname{Vec}}(\mathbf{x}_{n}(t))$, $\overline{U}(\mathbf{x}_{t})=\boldsymbol{\operatorname{Vec}}(U_{n}(\mathbf{x}_{n}(t)))$, $\boldsymbol{\zeta}_{t}=\boldsymbol{\operatorname{Vec}}(\boldsymbol{\zeta}_{n}(t))$, $\mathbf{w}_{t}=\boldsymbol{\operatorname{Vec}}(\mathbf{w}_{n}(t))$, and $L_{t}$ denotes the (stochastic) undirected graph Laplacian.

Remark 1**.**

In empirical risk minimization, agents optimize an empirical risk function using collected data, rather than optimizing the expected risk. In such problems, it is common to use stochastic gradient descent (SGD) techniques that mitigate computational burden by handling the data in batches. We note that our framework readily handles such SGD techniques as the $\boldsymbol{\zeta}_{t}$ term can model independent gradient noise.

III Intermediate Results

This section presents some intermediate results. In Section III-A, we begin by presenting several technical lemmas. Subsequently, in Section III-B we will use these technical lemmas to prove that the algorithm (7) obtains asymptotic consensus (see Lemma 4). Finally, in Section III-C, we briefly review classical results in global optimization that will be used in the proof of our main result.

III-A Technical Results

We begin by making the following assumptions.

Assumption 1**.**

The functions $U_{n}(\cdot)$ are $C^{2}$ with Lipschitz continuous gradients, i.e., there exists $L>0$ such that

[TABLE]

for all $n$.

Assumption 2**.**

The functions $U_{n}(\cdot)$ satisfy the following bounded gradient-dissimilarity condition:

[TABLE]

Denote by $\{\mathcal{H}_{t}\}$ the natural filtration corresponding to the update process (2), i.e., for all $t$, the $\sigma$-algebra $\mathcal{H}_{t}$ is given by

[TABLE]

Assumption 3**.**

The $\{\mathcal{H}_{t+1}\}$-adapted sequence of undirected graph Laplacians $\{L_{t}\}$ are independent and identically distributed (i.i.d.), with $L_{t}$ being independent of $\mathcal{H}_{t}$ for each $t$, and are connected in the mean, i.e., $\lambda_{2}(\bar{L})>0$ where $\bar{L}=\mathbb{E}[L_{t}]$.

Assumption 4**.**

The sequence $\{\boldsymbol{\zeta}_{t}\}$ is $\{\mathcal{H}_{t+1}\}$-adapted and there exists a constant $C_{1}>0$ such that

[TABLE]

for all $t\geq 0$.

Assumption 5**.**

For each $n$, the sequence $\{\mathbf{w}_{n}(t)\}$ is a sequence of i.i.d. $d$-dimensional standard Gaussian vectors with covariance $I_{d}$ and with $\mathbf{w}_{n}(t)$ being independent of $\mathcal{H}_{t}$ for all $t$. Further, the sequences $\{\mathbf{w}_{n}(t)\}$ and $\{\mathbf{w}_{l}(t)\}$ are mutually independent for each pair $(n,l)$ with $n\neq l$.

Assumption 6**.**

The sequences $\{\alpha_{t}\}$, $\{\beta_{t}\}$, and $\{\gamma_{t}\}$ satisfy

[TABLE]

where $c_{\alpha},c_{\beta},c_{\gamma}>0$ and $\tau_{\beta}\in(0,1/2)$.

The following lemma characterizes the decay rate of scaled gradient noise.

Lemma 1**.**

Let Assumption 4 hold. Then, for every $\delta>0$, we have that $(t+1)^{-1/2-\delta}\|\boldsymbol{\zeta}_{t}\|\rightarrow 0$ a.s. as $t\rightarrow\infty$.

Proof.

Fix $\varepsilon>0$ and note that, by Assumption 4,

[TABLE]

Since $\delta>0$, the term on the R.H.S. of (15) is summable, and by the Borel-Cantelli lemma we may conclude that

[TABLE]

where i.o. means infinitely often. Since $\varepsilon>0$ is arbitrary, the desired assertion follows. ∎

The following two technical results from [31] will be useful (see also [21]).

Lemma 2** (Lemma 4.3 in [31]).**

Let $\{\mathbf{z}_{t}\}$ be an $\mathbb{R}_{+}$ valued $\{\mathcal{H}_{t}\}$ adapted process that satisfies

[TABLE]

In the above, $\{r_{1}(t)\}$ is an $\{\mathcal{H}_{t+1}\}$ adapted process, such that for all $t$, $r_{1}(t)$ satisfies $0\leq r_{1}(t)\leq 1$ and

[TABLE]

with $a_{1}>0$ and $0\leq\delta_{1}<1$. The sequence $\{r_{2}(t)\}$ is deterministic, $\mathbb{R}_{+}$ valued and satisfies $r_{2}(t)\leq a_{2}/(t+1)^{\delta_{2}}$ with $a_{2}>0$ and $\delta_{2}>0$. Further, let $\{V_{t}\}$ and $\{J_{t}\}$ be $\mathbb{R}_{+}$ valued $\{\mathcal{H}_{t+1}\}$ adapted processes with $\sup_{t\geq 0}\|V_{t}\|<\infty$ a.s. The process $\{J_{t}\}$ is i.i.d. with $J_{t}$ independent of $\mathcal{H}_{t}$ for each $t$ and satisfies the moment condition $\mathbb{E}\left[\left\|J_{t}\right\|^{2+\varepsilon_{1}}\right]<\kappa<\infty$ for some $\varepsilon_{1}>0$ and a constant $\kappa>0$. Then, for every $\delta_{0}$ such that

[TABLE]

we have $(t+1)^{\delta_{0}}\mathbf{z}_{t}\rightarrow 0$ a.s. as $t\rightarrow\infty$.

Remark 2**.**

Note that Lemma 4.3 in [31] assumes $V_{t}$ above is $\{\mathcal{H}_{t}\}$ adapted, but the proof in [31] uses a pathwise analysis and can be readily adjusted to the case of $\{\mathcal{H}_{t+1}\}$ adapted $V_{t}$.

In $\mathbb{R}^{Nd}$, denote by $\mathcal{C}$ the consensus subspace,

[TABLE]

and denote by $\mathcal{C}^{\perp}$ its orthogonal subspace in $\mathbb{R}^{Nd}$.

Lemma 3** (Lemma 4.4 in [31]).**

Let $\{\mathbf{z}_{t}\}$ be an $\mathbb{R}^{Nd}$ valued $\{\mathcal{H}_{t}\}$ adapted process such that $\mathbf{z}_{t}\in\mathcal{C}^{\perp}$ for all $t$. Also, let $\{L_{t}\}$ be an i.i.d. sequence of Laplacian matrices as in Assumption 3 that satisfies

[TABLE]

with $L_{t}$ being $\mathcal{H}_{t+1}$ adapted and independent of $\mathcal{H}_{t}$ for all $t$. Then there exists a measurable $\{\mathcal{H}_{t+1}\}$ adapted $\mathbb{R}_{+}$ valued process $\{r_{t}\}$ (depending on $\{\mathbf{z}_{t}\}$ and $\{L_{t}\}$) and a constant $c_{r}>0$, such that $0\leq r_{t}\leq 1$ a.s. and

[TABLE]

with

[TABLE]

for all $t$ large enough, where the weight sequence $\{\beta_{t}\}$ and $\tau_{\beta}$ are defined in Assumption 6.

See [31] for a detailed discussion of the necessity of the various technicalities involved in the statement of Lemma 3.

III-B Consensus

The following lemma shows that, a.s., the algorithm (7) obtains consensus asymptotically.

Lemma 4** (Convergence to Consensus Subspace).**

Let Assumptions 1-6 hold. Let $\{x_{t}\}$ satisfy (7) with arbitrary initial condition. Then, for every $\tau\in[0,1/2-\tau_{\beta})$, we have that

[TABLE]

where $\overline{\mathbf{x}}_{t}$ is the network-averaged process, $\overline{\mathbf{x}}_{t}=(1/N)\sum_{l=1}^{N}\mathbf{x}_{l}(t)$.

Proof.

Noting that $\left(\mathbf{1}_{N}\otimes I_{d}\right)^{\top}\left(L_{t}\otimes I_{d}\right)=\mathbf{0}$ (by the properties of the undirected Laplacian), we have by (7),

[TABLE]

where

[TABLE]

Denote by $\{\breve{\mathbf{x}}_{t}\}$ the process, $\breve{\mathbf{x}}_{t}=\mathbf{x}_{t}-\mathbf{1}_{N}\otimes\overline{\mathbf{x}}_{t}$, for all $t\geq 0$, and note that

[TABLE]

since $\breve{\mathbf{x}}_{t}\in\mathcal{C}^{\perp}$, where recall $\mathcal{C}^{\perp}$ is the orthogonal complement of the consensus subspace (see (20)) and $\mathcal{P}_{Nd}=(1/N)\left(\mathbf{1}_{N}\otimes I_{d}\right)\left(\mathbf{1}_{N}\otimes I_{d}\right)^{\top}$.

By (7) and (25) we have

[TABLE]

for all $t\geq 0$. (For convenience, we suppress the time index on the $T_{i}$ terms.) Now, consider the $n$-th component of the term $T_{1}$,

[TABLE]

and note that $T_{1}^{n}$ may be decomposed as

[TABLE]

For the second term on the R.H.S. of (31), note that, by Assumption 2, there exists a constant $c_{1}>0$ such that

[TABLE]

Finally, by the Lipschitz continuity of the gradients (see Assumption 1), we have, for a constant $c_{2}>0$ large enough,

[TABLE]

Hence, there exist constants $c_{3},c_{4}>0$ such that

[TABLE]

For the term $T_{2}$ in (28), consider $\delta\in(0,1/2)$ arbitrarily small. Consider the process $\{R_{t}\}$, defined as $R_{t}=(t+1)^{-1/2-\delta}\left\|\boldsymbol{\zeta}_{t}-\mathbf{1}_{N}\overline{\boldsymbol{\zeta}}_{t}\right\|$ for all $t$, and note that by Lemma 1 we have $R_{t}\rightarrow 0$ as $t\rightarrow\infty$ a.s. Since $t^{-1}\leq 2(t+1)^{-1}$ for all $t>0$, we have (see Assumption 6)

[TABLE]

Similarly, note that,

[TABLE]

Noting that $\|T_{3}\|$ has moments of all order (by the Gaussianity of the $\mathbf{w}_{t}$’s), by (37)-(38) we conclude that there exist $\mathbb{R}_{+}$-valued $\{\mathcal{H}_{t+1}\}$-adapted processes $\{V_{t}\}$ and $\{J_{t}\}$ such that

[TABLE]

with $\{V_{t}\}$ being bounded a.s. and $\{J_{t}\}$ possessing moments of all orders.

Since $\breve{\mathbf{x}}_{t}\in\mathcal{C}^{\perp}$ for all $t\geq 0$, by Lemma 3 there exists a $\{\mathcal{H}_{t+1}\}$ adapted $\mathbb{R}_{+}$ valued process $\{r_{t}\}$ and a constant $c_{5}>0$ such that $0\leq r_{t}\leq 1$ a.s. and

[TABLE]

with

[TABLE]

for all $t$ large enough.

Thus, by (28), (36), (40), and (41) we obtain

[TABLE]

for $t$ large. Denote by $\{\overline{r}_{t}\}$ the process given by, $\overline{r}_{t}=r_{t}-\alpha_{t}$ for all $t$, and note that, since $\tau_{\beta}<1$, by (42) there exists a constant $c_{6}>0$ such that

[TABLE]

for all $t$ large enough. Noting that $\alpha_{t}=c_{\alpha}t^{-1}\leq 2(t+1)^{-1/2+\delta}$ for all $t$ large, by (43) we have

[TABLE]

for $t$ large and a constant $c_{7}>0$ sufficiently large. By (45) and the above development, the recursion in (46) clearly falls under the purview of Lemma 2, and we conclude that (by taking $\delta_{2}=1/2-\delta$ and $\delta_{1}=\tau_{\beta}$ in Lemma 2) for all $\tau$ and $\varepsilon_{1}>0$ such that

[TABLE]

we have $(t+1)^{\tau}\breve{\mathbf{x}}_{t}\rightarrow 0$ a.s. as $t\rightarrow\infty$. By taking $\varepsilon_{1}\rightarrow\infty$ (since $J_{t}$ has moments of all orders) and $\delta\rightarrow 0$, we conclude that $(t+1)^{\tau}\breve{\mathbf{x}}_{t}\rightarrow 0$ a.s. as $t\rightarrow\infty$ for all $\tau\in(0,1/2-\tau_{\beta})$. ∎

III-C Classical Results: Recursive Algorithms for Global Optimization

We will now briefly review classical results on global optimization from [8] that will be used in the proof of our main result.

Consider the following stochastic recursion in $\mathbb{R}^{d}$:

[TABLE]

where $U:\mathbb{R}^{d}\to\mathbb{R}_{+}$, $\{\boldsymbol{\xi}_{t}\}$ is a sequence of $\mathbb{R}^{d}$-valued random variables, $\{\mathbf{w}_{t}\}$ is a sequence of independent $d$-dimensional Gaussian random variables with mean zero and covariance $I_{d}$, and

[TABLE]

where $A,B>0$ are constants.

Consider the following assumptions on $U(\cdot)$, the gradient field $\nabla U(\cdot)$, and noise $\boldsymbol{\xi}_{t}$:

Assumption 7**.**

$U:\mathbb{R}^{d}\to\mathbb{R}$* is a $C^{2}$ function such that*

1. (i)

$\min U(x)=0$, 2. (ii)

$U(x)\to\infty$* and $|\nabla U(x)|\to\infty$ as $|x|\to\infty$,* 3. (iii)

$\inf(|\nabla U(x)|^{2}-\Delta U(x))>-\infty$.

We note that Within the context of PAC learning, the assumption (i) above corresponds to the “realizability” assumption, i.e., there exists a true (but unknown) hypothesis that accurately represents that data.

Assumption 8**.**

For $\varepsilon>0$ let

[TABLE]

[TABLE]

$U$* is such that $\pi^{\varepsilon}$ has a weak limit $\pi$ as $\varepsilon\to 0$.*

We note that $\pi$ is constructed so as to place mass 1 on the set of global minima of $U$. A discussion of simple conditions ensuring the existence of such a $\pi$ can be found in [35].

Assumption 9**.**

$\liminf_{|x|\to\infty}\langle\frac{\nabla U(x)}{|\nabla U(x)|},\frac{x}{|x|}\rangle\geq C(d)$, $C(d)=\left(\frac{4d-4}{4d-3}\right)^{\frac{1}{2}}$.

Assumption 10**.**

$\liminf_{|x|\to\infty}\frac{|\nabla U(x)|}{|x|}>0$**

Assumption 11**.**

$\limsup_{|x|\to\infty}\frac{|\nabla U(x)|}{|x|}<\infty$**

Let $\{\mathcal{G}_{t}\}$ be the natural filtration generated by (48); that is, $\mathcal{G}_{t}$, $t\geq 0$ is given by

[TABLE]

Assumption 12**.**

There exists $C_{1}>0$ such that

[TABLE]

with $\gamma_{1}>-1$ and $\gamma_{2}>0$.

Note that, in contrast to Assumption 4, Assumption 12 assumes the conditional mean may be non-zero (but decaying).

Finally, let $C_{0}$ be the constant as defined after (2.3) in [8].

The following result on the convergence of the stochastic process (48) was obtained in [8].

Theorem 1** (Theorem 4 in [8]).**

Let $\{\mathbf{z}_{t}\}$ be given by (48). Suppose that Assumptions 7–12 hold and assume $A$ and $B$ in (49) satisfy $B/A>C_{0}$. Then, for any bounded continuous function $f:\mathbb{R}^{d}\to\mathbb{R}$, we have that

[TABLE]

IV Main Results

We will now prove the main result of the paper. We shall proceed as follows. We will first study the behavior of the $\mathbb{R}^{d}$-valued networked averaged process

[TABLE]

Using Theorem 1, we will show that $\bar{\bf x}_{t}$ converges to the set of global minima of $U(\cdot)$ (see Lemma 5). After proving Lemma 5 we will present Theorem 2, which is the main result of the paper. Theorem 2 follows as a straightforward consequence of Lemmas 4 and 5.

Note that, taking the average on both sides of (2) we obtain,

[TABLE]

where

[TABLE]

and $\overline{\boldsymbol{\zeta}}_{t}$ and $\overline{\mathbf{w}}_{t}$ are given in (26).

The following lemma shows that the networked-averaged process $\{\overline{\bf x}_{t}\}$ converges to the set of global minima of $U(\cdot)$.

Lemma 5**.**

Let $\{{\bf x}_{t}\}$ satisfy the recursion (7) and let $\{\overline{\bf x}_{t}\}$ be given by (51), with initial condition $\mathbb{\bar{x}_{0}}\in\mathbb{R}^{d}$. Let Assumptions 3–6 hold and Assume $U(\cdot)$ satisfies Assumptions 1–2 and 7–11. Further, suppose that $c_{\alpha}$ and $c_{\gamma}$ in Assumption 6 satisfy, $c_{\gamma}^{2}/c_{\alpha}>C_{0}$, where $C_{0}$ is defined after Assumption 12. Then, for any bounded continuous function $f:\mathbb{R}^{d}\to\mathbb{R}$, we have that

[TABLE]

Proof.

The result will be proven by showing that the $\{\overline{\mathbf{x}}_{t}\}$ in (52) falls under the purview of Theorem 1, and, in particular, that Assumption 12 is satisfied. To do this, the key technical issue lies in handling the process $\{\overline{R}_{t}\}$. Specifically, we must restate the a.s. convergence obtained in Lemma 4 in terms of conditional expectations as required by Assumption 12.

Fix $\tau\in(0,\frac{1}{2}-\tau_{\beta})$ and let $\delta>0$ be arbitrary. Since, by Lemma 4, $t^{\tau}\|\mathbf{x}_{n}(t)-\overline{\mathbf{x}}_{t}\|\rightarrow 0$ as $t\rightarrow 0$ a.s. for all $n$, by Egorov’s theorem there exists a constant $R_{\delta}>0$ such that

[TABLE]

Note that, by Assumption 1,

[TABLE]

Now consider the $\mathcal{H}_{t}$-adapted process $\{\overline{R}^{\delta}_{t}\}$, given by

[TABLE]

for all $t\geq 0$. Note that, by construction,

[TABLE]

Now consider the stochastic process $\{\overline{\mathbf{x}}^{\delta}_{t}\}$ evolving as

[TABLE]

with initial condition $\overline{\mathbf{x}}_{0}^{\delta}=\frac{1}{N}\sum_{n=1}^{N}\mathbf{x}_{n}(0)$. It is readily seen that $\{\overline{\mathbf{x}}_{t}^{\delta}\}$ is $\mathcal{H}_{t}$-adapted and the processes $\{\overline{\mathbf{x}}_{t}^{\delta}\}$ and $\{\overline{\mathbf{x}}_{t}\}$ agree, for all $t\geq 0$, on the event $\left\{\sup_{t\geq 0}t^{\tau}\left\|\overline{R}_{t}\right\|\leq LR_{\delta}\right\}$, so that

[TABLE]

Let $\overline{\boldsymbol{\xi}}_{t}=\overline{\boldsymbol{\zeta}}_{t}+\overline{R}^{\delta}_{t}$ for all $t\geq 0$, and denote by $\mathcal{F}^{\delta}_{t}$ the $\sigma$-algebra

[TABLE]

Note that $\mathcal{F}_{t}^{\delta}\subset\mathcal{H}_{t}$ for all $t\geq 0$.

By Assumption 4 and (60) we have that, almost surely,

[TABLE]

and, by the parallelogram law,

[TABLE]

It is now straightforward to see that the process $\{\overline{\mathbf{x}}^{\delta}_{t}\}$ falls under the purview of Theorem 1. In particular, letting by taking $\gamma_{1}=0$ and $\gamma_{2}=\tau>0$ and letting $\mathcal{G}_{t}=\mathcal{F}_{t}^{\delta}$ we see that Assumption 12 is satisfied. We conclude that

[TABLE]

Note that, by (62),

[TABLE]

Hence, by (68), we have

[TABLE]

Since $\delta>0$ is arbitrary, we conclude that

[TABLE]

∎

We now state the main result concerning convergence of the agent estimates $\{\mathbf{x}_{n}(t)\}$ to the set of global minima of $U(\cdot)$.

Theorem 2**.**

Let $\{{\bf x}_{t}\}$ satisfy the recursion (7) with initial condition $\mathbb{x_{0}}$. Let Assumptions 3–6 hold and assume $U(\cdot)$ satisfies Assumptions 1–2 and 7–11. Further, suppose that $c_{\alpha}$ and $c_{\gamma}$ in Assumption 6 satisfy $c_{\gamma}^{2}/c_{\alpha}>C_{0}$, where $C_{0}$ is defined after Assumption 12. Then, for any bounded continuous function $f:\mathbb{R}^{d}\to\mathbb{R}$ and for all $n=1,\ldots,N$, we have that

[TABLE]

In the above theorem, we recall that we use the conventions (4)– (5), and that $\pi$ is a probability measure placing mass 1 on the set of global minima of $U(\cdot)$ as constructed in Assumption 8. The proof of the theorem follows below.

Proof.

The proof follows immediately from Lemma 4 and Lemma 5. In particular, by Lemma 4, we have that $\|\mathbf{x}_{n}(t)-\overline{\mathbf{x}}_{t}\|\rightarrow 0$ as $t\rightarrow\infty$ a.s. (by taking $\tau=0$ in Lemma 4) and the desired assertion follows by noting that the above a.s. convergence implies that the processes $\{\mathbf{x}_{n}(t)\}$ and $\{\overline{\mathbf{x}}_{t}\}$ have the same weak limits. ∎

V Conclusions

The paper proves, for a distributed consensus + innovations algorithm, convergence to the set of global optima in a distributed nonconvex optimization problems. Each agent only has information about the gradient of its personal objective function and the current state of neighboring agents. Convergence to a global minimum is achieved by means of decaying (annealing) noise injected into each agent’s update step. The paper proved convergence (in probability) of the algorithm to the set of global minima of the sum objective.