Asynchronous Delay-Aware Accelerated Proximal Coordinate Descent for Nonconvex Nonsmooth Problems

TL;DR

This paper introduces an accelerated asynchronous proximal coordinate descent method tailored for nonconvex and nonsmooth optimization problems, ensuring convergence and demonstrating practical speed improvements.

Contribution

It extends APCD to an accelerated version for nonconvex nonsmooth problems with delay-awareness, providing convergence guarantees for both bounded and unbounded delays.

Findings

Ensures convergence to critical points under delays

Achieves linear and sublinear convergence rates

Demonstrates practical efficiency through numerical experiments

Abstract

Nonconvex and nonsmooth problems have recently attracted considerable attention in machine learning. However, developing efficient methods for the nonconvex and nonsmooth optimization problems with certain performance guarantee remains a challenge. Proximal coordinate descent (PCD) has been widely used for solving optimization problems, but the knowledge of PCD methods in the nonconvex setting is very limited. On the other hand, the asynchronous proximal coordinate descent (APCD) recently have received much attention in order to solve large-scale problems. However, the accelerated variants of APCD algorithms are rarely studied. In this paper, we extend APCD method to the accelerated algorithm (AAPCD) for nonsmooth and nonconvex problems that satisfies the sufficient descent property, by comparing between the function values at proximal update and a linear extrapolated point using a delay-aware momentum value. To the best of our knowledge, we are the first to provide stochastic and deterministic accelerated extension of APCD algorithms for general nonconvex and nonsmooth problems ensuring that for both bounded delays and unbounded delays every limit point is a critical point. By leveraging Kurdyka-Lojasiewicz property, we will show linear and sublinear convergence rates for the deterministic AAPCD with bounded delays. Numerical results demonstrate the practical efficiency of our algorithm in speed.

Figures

Equations

$$\underset{x\in {\mathbb{R}}^m}{\min }F(x)=f(x)+g(x)$$

$$g(x)=\sum _{j=1}^m{g}_j(x_j)$$

$${\text{Prox}}_{\eta g_j}(y)=\text{arg}\underset{x\in {\mathbb{R}}^m}{\min }\frac{1}{2\eta }{\Vert x-y\Vert }^2+g_j(x_j)$$

$${\hat{y}}^k=y^k-\sum _{h\in I(k)}(y^{h+1}-y^h)$$

$${\varphi }^{\prime }(f(u)-f(\hat{u})){\text{dist}}_{\partial f(u)}(0)\ge 1$$

$$G(x^k):=G(x^k,y^k,\dots ,y^{k-\tau })=F(x^k)+{\xi }_k$$

$${\xi }_k:=\frac{L^2\tau }{2C}\sum _{h=k-\tau +1}^k(h-k+\tau ){\Vert y^h-y^{h-1}\Vert }^2$$

$$\begin{array}{rl}\mathbb{E}[& G(x^{k+1})]\le \mathbb{E}[G(y^k)]\\ & -\left(\frac{1}{2\eta }-\frac{L}{2}-L\tau (1+{\beta }_k)\right)\mathbb{E}{\Vert x^{k+1}-y^k\Vert }^2.\end{array}$$

$$b_1=\frac{2(\frac{1}{\eta }+L{)}^2(K+1)+2L^2{T}_1(1+\beta )+2L^{2}T(1+{\beta }^{\prime \prime })}{\left(\frac{1}{2\eta }-\frac{L}{2}-L\tau (1+\beta )\right)}$$

$$G(x^k):=G(x^k,y^k,\dots ,y^0)=F(x^k)+{\xi }_k$$

$${\xi }_k:=\frac{L^2}{2C}\sum _{h=1}^k{c}_{k-h}{\Vert y^h-y^{h-1}\Vert }^2.$$

$$\begin{array}{rl}\mathbb{E}[& G(x^{k+1})]\le \mathbb{E}[G(y^k)]\\ & -\left(\frac{1}{2\eta }-\frac{L}{2}-L(1+{\beta }_k)\sqrt{c_0}\right)\mathbb{E}{\Vert x^{k+1}-y^k\Vert }^2.\end{array}$$

$$b_1=\frac{(\frac{2}{{\eta }^2}+4L^2)+4L^2+4L^2{c}_0(1+\beta )}{\frac{1}{2\eta }-\frac{L}{2}-L(1+\beta )\sqrt{c_0}}.$$

$$G(x^k):=F(x^k)+{\xi }_k$$

$${\xi }_k:=\frac{L^2}{2C}\sum _{h=1}^{\infty }{\delta }_{k-h}{\Vert y^h-y^{h-1}\Vert }^2$$

$$\begin{array}{rl}G(& x^{k+1})\le G(y^k)\\ & -\left(\frac{1}{2\eta }-\frac{L}{2}-\sqrt{{\delta }_0{\mu }_{d_k}}L(1+{\beta }_k)\right){\Vert x^{k+1}-y^k\Vert }^2\end{array}$$

$$b_1=\frac{2(\frac{1}{\eta }+L{)}^2+3(1+\beta {)}^2{L}^2{T}_1+2(1+{\beta }^{\prime \prime }{)}^2{L}^{2}T}{(\frac{1}{c}-1)\frac{L}{2}}$$

$$r_k=\tilde{G}(y^k)-F^{\ast }.$$

$$\begin{split}b_{1}=&\frac{3}{(\frac{1}{c}-1)\frac{L}{2}}\bigg{(}(\frac{1}{\eta}+L)^{2}+(1+\beta)^{2}L^{2}T_{1}\\ &+(1+\beta^{\prime\prime})^{2}L^{2}T+2(1+\beta)^{2}{L^{2}\mu_{T}}\delta_{0}\bigg{)}\end{split}$$

$$g(x)=\lambda \sum _{j=1}^m\min (|x_j|,\theta ),$$

$$\begin{array}{rl}⟨x_{j_k}^{k+1}-y_{j_k}^k,{\nabla }_{j_k}f({\hat{y}}^k)⟩+& \frac{1}{2\eta }{\Vert x_{j_k}^{k+1}-y_{j_k}^k\Vert }^2+g_{j_k}(x_{j_k}^{k+1})\\ & \le g_{j_k}(y_{j_k}^k).\end{array}$$

$$f(x^{k+1})\le f(y^k)+⟨x_{j_k}^{k+1}-y_{j_k}^k,{\nabla }_{j_k}f(y^k)⟩+\frac{L}{2}{\Vert x_{j_k}^{k+1}-y_{j_k}^k\Vert }^2.$$

$$\begin{array}{rl}f(& x^{k+1})+\sum _{j=1}^m{g}_j(x_j^{k+1})\le f(y^k)+\sum _{j=1}^m{g}_j(y_j^k)\\ & +\frac{L}{2}{\Vert x^{k+1}-y^k\Vert }^2+⟨x_{j_k}^{k+1}-y_{j_k}^k,{\nabla }_{j_k}f(y^k)-{\nabla }_{j_k}f({\hat{y}}^k)⟩-\frac{1}{2\eta }{\Vert x^{k+1}-y^k\Vert }^2\end{array}$$

$$\begin{array}{rl}F(x^{k+1})& \le F(y^k)+\frac{L}{2}{\Vert x^{k+1}-y^k\Vert }^2\\ & +⟨x_{j_k}^{k+1}-y_{j_k}^k,{\nabla }_{j_k}f(y^k)-{\nabla }_{j_k}f({\hat{y}}^k)⟩-\frac{1}{2\eta }{\Vert x^{k+1}-y^k\Vert }^2.\end{array}$$

$$\begin{array}{rl}& ⟨x_{j_k}^{k+1}-y_{i_k}^k,{\nabla }_{j_k}f(y^k)-{\nabla }_{j_k}f({\hat{y}}^k)⟩\\ & \stackrel{a)}{\le }L\Vert x_{j_k}^{k+1}-y_{j_k}^k\Vert \Vert y^k-{\hat{y}}^k\Vert \\ & \stackrel{b)}{\le }\frac{L^2}{2C}{\Vert y^k-{\hat{y}}^k\Vert }^2+\frac{C}{2}{\Vert x^{k+1}-y^k\Vert }^2\end{array}$$

$$\begin{array}{rl}& {\mathbb{E}}_{d_k}⟨x_{j_k}^{k+1}-y_{j_k}^k,{\nabla }_{j_k}f(y^k)-{\nabla }_{i_k}f({\hat{y}}^k)|{\mathcal{F}}_k⟩\\ & \le \frac{L^2}{2C}{\mathbb{E}}_{d_k}\left[{\Vert y^k-{\hat{y}}^k\Vert }^2|{\mathcal{F}}_k\right]+\frac{C}{2}{\mathbb{E}}_{d_k}{\Vert x^{k+1}-y^k\Vert }^2\\ & \stackrel{a)}{\le }\frac{L^2\tau }{2C}\sum _{h=k-\tau +1}^k{\Vert y^h-y^{h-1}\Vert }^2+\frac{C}{2}{\Vert x^{k+1}-y^k\Vert }^2\\ & =\left(\frac{L^2\tau }{2C}\sum _{h=k-\tau +1}^k(h-k+\tau ){\Vert y^h-y^{h-1}\Vert }^2-\frac{L^2\tau }{2C}\sum _{h=k+1-\tau +1}^{k+1}(h-(k+1)+\tau ){\Vert y^h-y^{h-1}\Vert }^2\right)\\ & +\frac{L^2{\tau }^2}{2C}{\Vert y^{k+1}-y^k\Vert }^2+\frac{C}{2}{\Vert x^{k+1}-y^k\Vert }^2\end{array}$$

$$\begin{array}{rl}\mathbb{E}\Vert y^{h+1}-y^h\Vert & =(1+{\beta }_h)\mathbb{E}\Vert x^{h+1}-y^h\Vert \text{for }h\in {{\Gamma }^c}_k^r,{\beta }_h>0\\ \mathbb{E}\Vert y^{h+1}-y^h\Vert & =\mathbb{E}\Vert x^{h+1}-y^h\Vert \text{for }h\in {{\Gamma }^0}_k^r\\ \mathbb{E}\Vert y^{h+1}-y^h\Vert & =(1+{\beta }_h)\mathbb{E}\Vert x^{h+1}-y^h\Vert \text{for }h\in {\Gamma }_k^r,{\beta }_h<0.\end{array}$$

$$\begin{array}{rl}\mathbb{E}[{\xi }_{k+1}]& \le \mathbb{E}[{\xi }_k]-\mathbb{E}⟨x_{j_k}^{k+1}-y_{j_k}^k,{\nabla }_{j_k}f(y^k)-{\nabla }_{j_k}f({\hat{y}}^k)⟩\\ & +\frac{L^2{\tau }^2}{2C}\mathbb{E}{\Vert y^{k+1}-y^k\Vert }^2+\frac{C}{2}\mathbb{E}{\Vert x^{k+1}-y^k\Vert }^2\\ & \le \mathbb{E}[{\xi }_k]-\mathbb{E}⟨x_{j_k}^{k+1}-y_{j_k}^k,{\nabla }_{j_k}f(y^k)-{\nabla }_{j_k}f({\hat{y}}^k)⟩\\ & +\left(\frac{L^2{\tau }^2(1+{\beta }_k{)}^2}{2C}+\frac{C}{2}\right)\mathbb{E}{\Vert x^{k+1}-y^k\Vert }^2.\end{array}$$

$$\begin{array}{rl}\mathbb{E}[F(x^{k+1})+{\xi }_{k+1}]+& \frac{1}{2\eta }\mathbb{E}{\Vert x^{k+1}-y^k\Vert }^2\\ \le \mathbb{E}[F(y^k)+{\xi }_k]+& \frac{L}{2}\mathbb{E}{\Vert x^{k+1}-y^k\Vert }^2+\left(\frac{L^2{\tau }^2(1+{\beta }_k{)}^2}{2C}+\frac{C}{2}\right)\mathbb{E}{\Vert x^{k+1}-y^k\Vert }^2.\end{array}$$

$$\begin{array}{rl}\mathbb{E}[F(x^{k+1})+{\xi }_{k+1}& ]\\ \le \mathbb{E}[F(y^k)+& {\xi }_k]-\left(\frac{1}{2\eta }-\frac{L}{2}-\frac{L^2{\tau }^2(1+{\beta }_k{)}^2}{2C}-\frac{C}{2}\right)\mathbb{E}{\Vert x^{k+1}-y^k\Vert }^2.\end{array}$$

Full text

Introduction

For many machine learning and data mining applications, efficiently solving the optimization problem with nonsmooth regularization is important. In this paper, we focus on the following composite optimization problem of machine learning model with nonsmooth regularization term as

[TABLE]

where $f:\mathbb{R}^{m}\to\mathbb{R}$ captures the empire risk which is smooth and possibly nonconvex, and $g:\mathbb{R}^{m}\to\mathbb{R}$, corresponding to the regularization term, reduces to a finite-sum

[TABLE]

where each $g_{j}$ can be nonconvex.

Many problems on (1) correspond to convex model that can be efficiently optimized by first order algorithm, in particular accelerated proximal gradient (APG) methods which is proven to be efficient for the class of convex functions. However, many real applications require the problems to be nonconvex. The nonconvexity might originate either from function $f(x)$ or the regularization function. This type of problems is popular in machine learning, for example, sparse logistic regression (?), and sparse multi-class classification (?). On the other hand regarding the nonsmooth regularization terms, proximal gradient methods often address solving optimization problems with nonsmoothness. The proximal operator is defined as following

[TABLE]

where $\eta>0$, and $\left\lVert\cdot\right\rVert$ is $l_{2}$-norm. If the proximal operator does not have an analytic solution, an algorithm should be used to solve the proximal operator which might be inexact. In this paper we consider only algorithms which use exact proximal mapping.

While the new algorithms for problem (1) provide both good theoretical convergence and empirical performances, the investigations on them were mainly conducted in the sequential setting. In the current big data era, we need to design algorithms to deal with very large scale problems ($m$ is large). In this case, we need to eliminate sequential updates which usually take too much costly idle time. This necessitates parallel computation which will not use synchronization to wait for all others and share their updates. Recently asynchronous parallelization have received huge successes due to its potential to vastly speed up algorithms (?; ?). We design and analyze an asynchronous parallel implementations of the accelerated proximal coordinate descent algorithms with bounded and unbounded delays for nonconvex nonsmooth problems, which is not well studied in the literature, to the best of our knowledge.

Contributions

The main contributions of this paper are summarized as follows. We first propose the basic stochastic and deterministic variants of asynchronous accelerated proximal coordinate descent algorithm for nonconvex problems. By construction of Lyapunov functions, we show that the limit points of the sequences generated by AAPCD are critical points of the problem (1) for both bounded delays and unbounded delays. This is one of the first convergence results for a method with acceleration which alleviates the bottleneck of unbounded delays for nonsmooth nonconvex functions. The convergence studies for AAPCD, through a novel perspective, characterize the stepsize based on the momentum parameter. This fills the void in previous analyses such as (?; ?), where the effect of the exact value of the momentum parameter on the acceleration of convergence were not observed. As the stability of the algorithm is highly affected by asynchronism, by allowing negative momentum for high staleness values we will show the reduction in the objective function will be increased significantly and accelerates convergence. In particular, we characterize the momentum parameter in the sense that increasing the stepsize would involve decreasing of the momentum parameter, while it will provide comparable asymptotic convergence in terms of the violation of first-order optimality conditions. We will show that by requiring momentum, a fixed stepsize could be chosen for unbounded delays.

By leveraging different cases of Kurdyka-Łojasiewicz property of the objective function, we establish the linear and sub-linear convergence rates of the function value sequence generated by the deterministic AAPCD with bounded deterministic delays and they match the synchronous results. In all the cases investigated in this paper, the independence assumption between blocks and delays is avoided.

We provide numerical experiments to demonstrate the performance of our stochastic AAPCD algorithm on various large-scale real-world datasets. The results outperforms other asynchronous stochastic algorithms reported in literature such as ASCD (?) and AASCD (?). It also shows that AAPCD can achieve good speedup on large-scale real-world datasets and provide significantly faster convergence to a reasonable accuracy than competing options, while still providing favorable asymptotic accuracy.

Related Works

Proximal Gradient Algorithms: Proximal gradient methods for nonsmooth regularization are among the most important methods for solving composite optimization problems. There have been accelerated exact proximal gradient variants. Specifically, for convex problems, the authors in (?) displayed basic accelerated proximal gradient (APG) method which extends Nesterov’s accelerated methods for solving single smooth convex function (?). They proved that APG displays the non-asymptotic convergence rate $O(\frac{1}{k^{2}})$, where $k$ is the number of iterations.

For extensions to nonconvex settings, (?) studied the condition that only the regularization term could be nonconvex, and proved the convergence rate of APG method. (?) established the convergence of proximal method when $f(x)$ and $g(x)$ could be nonconvex. (?) focused on first-order algorithms and by exploiting KL property they proved that APG algorithm can converge to a stationary point in different rates. Recently, in (?) and (?) several accelerated proximal methods were studied, and sublinear and linear rates under different cases of the KL property for nonconvex problems were provided.

In addition to the above proximal gradient methods, several stochastic optimization methods were developed for solving composite problems see, e.g., proximal stochastic coordinate descent prox-SCD (?), prox-SVRG (?), prox-SAGA (?), prox-SDCA (?). Under the assumption that the regularization term is block separable, (?) developed a randomized block-coordinate descent method. An accelerated variant of this method is studied in (?). All these stochastic methods require convexity of $f$, or even stronger assumptions.

For nonconvex problems, (?) generalized an accelerated SGD method to solve nonconvex but smooth minimization problems. Stochastic variance reduction methods for nonconvex problems were investigated in (?; ?). Furthermore, proximal variance reduction methods for general nonconvex, nonsmooth problems are proposed in (?; ?). Then, (?) proposed a block stochastic gradient method for nonconvex and nonsmooth problems.

Asynchronous Coordinate Descent: The asynchronous computation is much more efficient than the synchronous computation. More recently, asynchronous parallel methods have been successfully applied to accelerate many optimization algorithms including stochastic coordinate descent (?). We briefly review the works which are closely related to ours as follows. ASCD can provide linear and sublinear convergence rates (?; ?). Similar results were established for asynchronous SGD (?), and stochastic variance reduction algorithms (?; ?). A study of ASCD for unbounded delays has been performed in (?), however the results are restricted only to Lipschitz differentiable functions. Some asynchronous algorithms particularly outperform conventional ones. In (?), authors integrated momentum acceleration and variance reduction techniques to accelerate asynchronous SGD. Several accelerated schemes for asynchronous coordinate descent and SVRG using momentum compensation techniques were proposed in (?). Recently, (?) analyzed an asynchronous accelerated block coordinate descent algorithm with optimal complexity which converges linearly to a solution for strongly convex functions.

However, to the best of our knowledge, there is no study on the asynchronous parallel versions of accelerated proximal coordinate descent algorithms for nonconvex nonsmooth objective functions.

Preliminaries and Assumptions

We describe our asynchronous accelerated proximal coordinate descent for nonconvex problems in Algorithm 1. Compared to the regular proximal coordinate descent step, AAPCD takes an extra linear extrapolation step depending on the value of the current ages of $\hat{y}^{k}$, which is called also delay and denoted by $d_{k}$. In order to compute the delay $d_{k}$, we use a scalar counter to denote the weights at iteration $k$, starting from $k=0$, and with each update we increment the counter by one. We allow each worker to record the iteration $i$ when reading the weights and we let $k$ to denote the iteration when the same worker updating the weights. Then the actual delay $d_{k}$ is $d_{k}=k-i$. If delay is greater than the threshold $T_{1}$, we consider adding negative momentum to extrapolate a new iterate. We further show that adding such a momentum for large delays have the effect of decreasing Lyapunov function over iterations. For acceleration, AAPCD only accepts the new extrapolated iterate when the objective function value is sufficiently decreased. It is important to note that the threshold $T_{1}$ can adaptively change during the iterations. From practical point of view there is a need to know how to select the parameter $T_{1}$. We will address this question later when we present the analyses of convergence. It will be shown that accumulation points of sequences generated by AAPCD will converge to stationary points of $F$. In the step 5 of Algorithm 1, at iteration $k$, the block gradient $\nabla_{j_{k}}f$ is computed at the delay iterate $\hat{y}^{k}$, which is assumed to be some earlier state of $y^{k}$ in the shared memory with the delay $d_{k}$. The delay iterate $\hat{y}^{k}$ can be formulated as

[TABLE]

where $I(k)\in\{k-1,\ldots,k-d_{k}\}$ is a subset of previous iterations. From the proximal update for AAPCD, we have $x^{k+1}_{j}=y^{k}_{j}$ for $j\neq j_{k}$. We also assume $\beta=\max_{k}\{\beta_{k}\geq 0\}$ and $\beta^{\prime}=\max_{k}\{\beta_{k}<0\}$. We let $\Gamma_{k}^{r}$ be the set of iterations from $k$ to $r$ with $d_{k}>T_{1}$ and $y^{k+1}=v^{k}$, ${\Gamma^{c}}_{k}^{r}$ denote the set of iterations from $k$ to $r$ with $d_{k}\leq T_{1}$ and $y^{k+1}=v^{k}$, and ${\Gamma^{0}}_{k}^{r}$ denote the set of iterations from $k$ to $r$ with $y^{k+1}=x^{k+1}$.

By studying different cases of KL property we will show that AAPCD will decrease the function value properly at the initial point. For the deterministic AAPCD with deterministic bounded staleness, we prove the linear and sublinear convergence rate by exploiting different cases of KL property.

In the following we first introduce some tools for analyzing asynchronous algorithms, and then describe the assumptions on the problem (1) that we assume in this paper.

For analysis of the stochastic algorithm, we let $\mathcal{F}_{k}$ denote the sigma algebra generated by $\{y^{0},\ldots,y^{k}\}$ . We denote the total expectation by $\mathbb{E}$ and the expectation over the stochastic variable $d_{k}$ by $\mathbb{E}_{d_{k}}$. Function $g(x)$ is lower semicontinuous at point $x_{0}$ if $\lim\inf_{x\to x_{0}}g(x)\geq g(x_{0})$. Throughout this paper, we assume each $g_{j}$ in problem (1) is lower semicontinuous. A point $x\in\mathbb{R}^{m}$ is said a critical point of function $F$ if $0\in\partial F(x)$. The following Uniformized KL property is a powerful tool to analyze the first order descent algorithms.

Definition 1** (Uniformized KL Property).**

A function $f:\mathbb{R}^{m}\to(-\infty,\infty]$ is said to satisfy the Uniformized KL property if for every compact set $\Omega\subset\text{dom}~{}\partial f$ on which $f$ is constant, there exists $\epsilon,\gamma\in(0,+\infty]$ and $\phi\in\Phi_{\gamma}$, such that for all $\hat{u}\in\Omega$ and all $u\in\{u\in\mathbb{R}^{m}:\text{dist}_{\Omega}(u)<\epsilon\}\cap\{u\in\mathbb{R}^{m}:f(\hat{u})<f(u)<f(\hat{u})+\gamma\}$, the following inequality holds

[TABLE]

where $\Phi_{\gamma}$ stands for a class of function $\phi:[0,\gamma)\to\mathbb{R}^{+}$ satisfying: (1) $\phi$ is concave and $C^{1}$ on $(0,\gamma)$; (2) $\phi$ is continuous at [math], $\phi(0)=0$; and (3) $\phi^{\prime}(x)>0$, for all $x\in(0,\gamma)$.

By (?, Lemma 6), if function $f$ is lower semicontinuous and satisfies KL property at every point of $\Omega$, then it satisfies the Uniformized KL property. All semi-algebraic functions satisfy the KL property. Specially, the desingularising function $\phi(t)$ of semi-algebraic functions can be chosen to take the form $\phi(t)=\frac{C}{\theta}t^{\theta}$ with $\theta\in(0,1]$. In particular, typical semi-algebraic functions include real polynomial functions, $\left\lVert x\right\rVert_{p}$ with $p\geq 0$, rank, etc.

We make the following assumptions on the problem (1) in this paper.

Assumption 1**.**

Function $f$ and each $g_{j}$ are proper and lower semicontinuous; $\inf_{x\in\mathbb{R}^{m}}F(x)>-\infty$; the sublevel set $\{x\in\mathbb{R}^{d}:F(x)\leq\alpha\}$ is bounded for all $\alpha\in\mathbb{R}$.

Assumption 2**.**

Function $f$ is continuously differentiable and the gradient $\nabla f$ is $L$-Lipschitz continuous.

To prove the limit points of $\{y^{k}\}$ generated by AAPCD are stationary points, we need a new assumption:

Assumption 3**.**

For AAPCD, it is assumed that there exists $K\in\mathbb{N}$ such that for all $k\in\mathbb{N}$, we have $\{1,\ldots,m\}\subseteq\{j_{k+1},\ldots,j_{k+K}\}$.

The goal of our paper is to provide a comprehensive analysis for AAPCD for both bounded and unbounded delays to justify the overall advantages of AAPCD.

AAPCD with Bounded Delays

In this section we analyze the convergence of Algorithms 1 for bounded delays, i.e., we assume $d_{k}\leq\tau$ for all $k$ and for a fixed number $\tau$. Define the Lyapunov function $G$ as

[TABLE]

where the sequence $\{\xi_{k}\}_{k\in\mathbb{N}}$, defined by

[TABLE]

with $C>0$ is a constant to be determined later. In the lemma below, we present an inequality which states for a proper stepsize, AAPCD can provide sufficient descent in our Lyapunov function.

Lemma 1**.**

Suppose Assumption 2 hold. Given $\eta>0$, we have

[TABLE]

We characterize the convergence of AAPCD. Our first result characterizes the behavior of the limit points of the sequence generated by AAPCD. Based on the lemma, we show that the sequence $\{y^{k}\}$ generated by AAPCD approaches critical points of the general nonconvex problem (1).

Theorem 1**.**

Let Assumptions 1-3 hold for the problem (1). Then with stepsize $\eta<\frac{1}{L+2LT_{1}(1+\beta)}$, and the momentum $-1<\beta_{k}<\frac{1}{L\tau}(\frac{1}{2\eta}-\frac{L}{2})-1$ the sequence $\{y^{k}\}$ generated by AAPCD satisfies

$\{y^{k}\}$* is an almost surely bounded sequence and $\mathbb{E}\left\lVert y^{k+1}-y^{k}\right\rVert\to 0$.* 2. 2.

The set of limit points of $\{y^{k}\}$ forms a compact set, on which function $F$ is a constant $F^{*}$ and the sequences $\{F(y^{k})\}$ and $\{G(y^{k})\}$ converge to $F^{*}$. 3. 3.

All the limit points of $\{y^{k}\}$ are critical points of $F$, and $\mathbb{E}[\text{dist}_{\partial F(y^{k})}(0)]={o}(\frac{1}{\sqrt{k}})$.

Remark 1**.**

The connectedness and compactness of the set $\Omega$ of the limit points of $\{y^{k}\}$ is implied from $E\left\lVert y^{k+1}-y^{k}\right\rVert\to 0$. Theorem 1 also states that the objective function on $\Omega$ containing the critical points remains constant.

Remark 2**.**

Equation (4) shows that the selection of negative $\beta_{k}$ for substantial staleness values would increase Lyapunov function reduction over an iteration. In the light of the bounds for the momentum term $\beta_{k}$ in Theorem 1, we could realize an estimation of an upper bound for the threshold $T_{1}$ in AAPCD algorithm. The staleness bound $T_{1}$ should be large enough to allow positive $\beta_{k}$. For example if $\beta=\frac{1}{2}$, then we should have, $\frac{1}{L\tau}(\frac{1}{2\eta}-\frac{L}{2})-1\geq\frac{1}{2}$. Thus, by choosing $\eta=\frac{1}{L+4LT_{1}(1+\beta)}$, we obtain $T_{1}\geq\frac{3\tau}{4(1+\beta)}=\frac{\tau}{2}$.

The compact set $\Omega$ satisfies the requirements of the Uniformized KL property, and hence can be utilized to show the decrease of function values, depending on a certain exponent $\theta$ defined below.

Theorem 2**.**

Let the conditions of Theorem 1 hold. Suppose that $F$ satisfies the Uniformized KL property with desingularising function $\phi$ of the form $\phi(t)=\frac{e}{\theta}t^{\theta}$. Let $F(x)=F^{*}$ for all of the limit points of $\{x^{k}\}$ in AAPCD, and denote $r_{k}=F(x^{k})-F^{*}$. Then the sequence $\{r_{k}\}$ for $k$ large enough satisfies

If $\theta=1$, and $x_{0}$ is chosen such that $r_{0}<\frac{1}{b_{1}e^{2}}$, then $r_{k}$ reduces to zero in finite steps; 2. 2.

If $\theta=\frac{1}{2}$, then $\mathbb{E}[r_{k+1}]\leq\frac{b_{1}e^{2}}{1+b_{1}e^{2}}\mathbb{E}[r_{0}]$,

where

[TABLE]

with $\beta^{\prime\prime}=\max\{\beta^{\prime},0\}$.

Remark 3**.**

As $\beta^{\prime\prime}\leq 0$, the contribution of the delays greater than $T_{1}$ in the factor $b_{1}$, i.e., $2L^{2}T(1+\beta^{\prime\prime})$ decreases, which indicates acceleration is possible with negative momentum term.

AAPCD with Unbounded Delays

In this section, we allow the delay $d_{k}$ to be an unbounded stochastic variable, and extremely large delays in our algorithm are permitted. Depending on some limitations on the distribution of $d_{k}$, we can still prove convergence. For unbounded delay analysis, one approach is to consider a new bound for the distribution of the end-behavior of $d_{k}$ to decay sufficiently fast as the iterations progress.

We emulate this solution in the following. In particular, we define fixed parameters $p_{j}$ related to probabilities of the delay such that $p_{j}\geq P(j(k)=j)$, for all $k$, and $c_{k}:=\sum_{t=1}^{\infty}t(t+k)p_{t+k}$ with $\sum_{k=0}^{\infty}c_{k}<\infty$. For instance, we note that if ${p_{j}}$ have the probability distributions with decay bound $p_{j}={\mathcal{O}}(j^{-t})$, $t>4$, then $\sum_{k=0}^{\infty}c_{k}$ is finite.

We define a more involved Lyapunov function $G$ as

[TABLE]

where to simplify the presentation, we define $\xi_{k}$ which encompasses all terms

[TABLE]

where $\frac{1}{C}>0$ is a contraction rate to be defined later.

Lemma 2**.**

Under Assumption 1, for any $\eta>0$, we have

[TABLE]

Now we characterize the behavior of the limit points of the sequence generated by AAPCD with unbounded delays.

Theorem 3**.**

Let Assumptions 1-3 hold for the problem (1). Then with stepsize $\eta<\frac{1}{L+2L\sqrt{c_{T_{1}}}(1+\beta)}$ and momentum $-1<\beta_{k}<\frac{1}{L\sqrt{c_{0}}}(\frac{1}{2\eta}-\frac{L}{2})-1$, the sequence $\{y^{k}\}$ generated by AAPCD satisfies

$\{y^{k}\}$* is an almost surely bounded sequence and $\mathbb{E}[\xi_{k}]\to 0$.* 2. 2.

The set of limit points of $\{y^{k}\}$ forms a compact set, on which the functions $F$ is a constant $F^{*}$ and $\{F(y^{k})\}$ and $\{G(y^{k})\}$ converge to $F^{*}$. 3. 3.

All the limit points of $\{y^{k}\}$ are critical points of $F$.

Remark 4**.**

Lemma 2 shows that the selection of negative $\beta_{k}$ for delays greater than $T_{1}$ would decrease Lyapunov function substantially over an iteration. The bounds for $\beta_{k}$ in Theorem 3 imply an estimation of a lower bound for $c_{T_{1}}$. For example if $\beta=\frac{1}{2}$, then, we should have $\frac{1}{L\tau}(\frac{1}{2\eta}-\frac{L}{2})-1\geq\frac{1}{2}$. Hence, by selecting $\eta=\frac{1}{L+4L\sqrt{c_{T_{1}}}(1+\beta)}$, we obtain $c_{T_{1}}\geq\frac{9c_{0}}{16(1+\beta)^{2}}=\frac{c_{0}}{4}$.

Now by applying the Uniformized KL property we show Algorithm 1 decreases the objective value below that of $F(x_{0})$.

Theorem 4**.**

Let the conditions of Theorem 3 hold and $F$ satisfies the Uniformized KL property and the desingularising function has the form of $\phi(t)=\frac{e}{\theta}t^{\theta}$ with $e>0$. We denote $r_{k}=F(y^{k})-F^{*}$, where $F^{*}$ is the function value on the set of limit points of $\{y^{k}\}$. Then for $k$ large enough the sequence $\{r_{k}\}$ satisfies

If $\theta=1$, and $x_{0}$ is chosen such that $r_{0}<\frac{1}{b_{1}e^{2}}$ then $r_{k}$ reduces to zero in finite steps; 2. 2.

If $\theta=\frac{1}{2}$, then $\mathbb{E}[r_{k+1}]\leq\frac{b_{1}e^{2}}{1+b_{1}e^{2}}\mathbb{E}[r_{0}]$,

where

[TABLE]

Deterministic AAPCD

In this section, we consider deterministic unbounded delays. Specifically, deterministic AAPCD is presented in Algorithm 2. The stochastic and deterministic AAPCD differ only on how the current coordinates are selected at each iteration. For this purpose, we assume the delay variable $d_{k}$ is deterministic, which allow extremely large delays in our algorithm. We will prove that a subsequence of points $\{y^{k}\}$ generated by deterministic AAPCD converges to a stationary point. Using KL property we will see that if $x^{0}$ is not a stationary point, Algorithm 2 decreases the objective value below that of $F(x^{0})$. We also prove the rate of convergence for the deterministic algorithm with deterministic bounded delay by exploiting KL property, which is unavailable in the stochastic setting for the Lyapunov function.

As recommended in (?), we set a sequence $\{\epsilon_{i}\}_{i\geq 0}$ and define the Lyapunov function $G$ which encompasses all terms to control unbounded delays

[TABLE]

where to simplify the presentation, we define

[TABLE]

with $\delta_{i}=\sum_{j=i}^{\infty}\epsilon_{j}$ such that $\sum_{j=0}^{\infty}\delta_{j}<\infty$ and $C>0$ to be determined later.

Lemma 3**.**

Let Assumption 2 hold. For any $\eta>0$, we have

[TABLE]

where $\mu_{d_{k}}=\sum_{h=0}^{d_{k}-1}\frac{1}{\epsilon_{h}}$.

For any $T\geq\lim\inf d_{k}$ which can be arbitrarily large, let $S_{T}$ be the subsequence of $\mathbb{N}$ where the current delay is less than $T$. We will show the points $x^{k}$, $k\in S_{T}$, have convergence guarantees. The following theorem for unbounded deterministic delay is parallel to Theorem 3.

Theorem 5**.**

Suppose that Assumptions 1-3 hold. Then with stepsize $\eta=\frac{c}{L+2\sqrt{{\delta_{0}}{\mu_{T_{1}}}}L(1+\beta)}$ for $c\in(0,1)$, and momentum $-1<\beta_{k}<{\frac{\sqrt{\mu_{T_{1}}}}{c\sqrt{\mu_{d_{k}}}}}(1+\beta)-1$, we have,

$\{y^{k}\}$* is a bounded sequence and $\xi_{k}\to 0$.* 2. 2.

The function $F$ is constant on the set of limit points of $\{y^{k}\}$ and the sequences $\{F(y^{k})\}$ and $\{G(y^{k})\}$ converge to it. 3. 3.

For any subsequence $S_{T}$ generated by the deterministic AAPCD, all the limit points of $\{y^{k}\}_{k\in S_{T}}$ are critical points of $F$.

Remark 5**.**

Lemma 3 shows that the use of momentum for delayed gradient might gain no performance and have negative effects. Hence, to compensate this issue, we allow the selection of negative $\beta_{k}$ for high staleness values to maximize the reduction of the Lyapunov function over an iteration. By taking the bounds in Theorem 5 for the momentum term $\beta_{k}$ in to consideration, we could present an upper bound estimate for the threshold $T_{1}$ in AAPCD. The delay bound $T_{1}$ should be large enough to allow positive $\beta_{k}$. For example if we choose $\beta=1$, then we should have, ${\frac{\sqrt{\mu_{T_{1}}}}{c\sqrt{\mu_{d_{k}}}}}(1+\beta)-1\geq 1$. Therefore, $T_{1}$ must be large enough such that ${\mu_{T_{1}}}\geq\frac{4c^{2}\mu_{d_{k}}}{(1+\beta)^{2}}=c^{2}\mu_{d_{k}}$, for all $k$.

It is important to note that although Theorem 5 shows a fixed step size works for deterministic AAPCD, however, in return the upper bound for momentum is adaptive to the current delay.

In the following theorem, it turns out that a subsequence of Algorithm 2 can decrease the function value at $x_{0}$, depending on the parameter $\theta$ defined below.

Theorem 6**.**

Let conditions of Theorem 5 hold and that $F$ satisfies the Uniformized KL property and the desingularising function has the form $\phi(s)=\frac{e}{\theta}t^{\theta}$, where $\theta\in(0,1]$ and $e>0$. Let $F(x)=F^{*}$ for all $x\in\Omega$ (the set of limit points), and denote $r_{k}=F(y^{k})-F^{*}$. Then the sequence $\{r_{k}\}_{k\in S_{T}}$ for $k$ large enough satisfies

If $\theta=1$, and $x_{0}$ is chosen such that $r_{0}<\frac{1}{b_{1}c^{2}}$ then $r_{k}$ reduces to zero in finite steps; 2. 2.

If $\theta\in[\frac{1}{2},1)$, then for $k$ large enough $r_{k}\leq\frac{b_{1}e^{2}}{1+b_{1}e^{2}}r_{0}$; 3. 3.

If $\theta\in(0,\frac{1}{2})$, then $r_{k}\leq\left(\frac{1}{b_{2}{({1-2\theta})}+r_{{0}}^{2\theta-1}}\right)^{\frac{1}{1-2\theta}}$

where

[TABLE]

with $\beta^{\prime\prime}=\max\{\beta^{\prime},0\}$ and $b_{2}=\min(\frac{1}{b_{1}e^{2}R},\frac{r_{0}^{2\theta-1}(R^{\frac{2\theta-1}{2\theta-2}}-1)}{1-2\theta})$ for a fixed number $R\in(1,\infty)$.

For the deterministic AAPCD with deterministic bounded delay $T$, we define $\epsilon_{i}=0$ for $i>T$ and we let $\tilde{G}(x^{k},y^{k},\ldots,y^{k-T})$ denote the corresponding Lyapunov function. In the following $\tilde{G}(x)$ refers to $\tilde{G}(x,x,\ldots,x)$. We let $\Omega$ denote the set of stationary points of $F$. Since $\xi_{k}\to 0$, by Theorem 5, $\tilde{G}$ is constant on $\Omega$. We can derive convergence rates for

[TABLE]

Theorem 7**.**

Assume the conditions of Theorem 6, but only $\tilde{G}$ satisfies the Uniformized KL property and the desingularising function has the form $\phi(s)=\frac{e}{\theta}t^{\theta}$, where $\theta\in(0,1]$ and $e>0$. Then if the delay is bounded by $T$, the sequence $\{r_{k}\}$ for $k$ large enough satisfies

If $\theta=1$, then $r_{k}$ reduces to zero in finite steps; 2. 2.

If $\theta\in[\frac{1}{2},1)$, then $r_{k}\leq\left(\frac{b_{1}e^{2}}{1+b_{1}e^{2}}\right)^{\lfloor\frac{k-k_{1}}{T+K}\rfloor}r_{k_{1}}$ for $k_{1}$ large enough; 3. 3.

If $\theta\in(0,\frac{1}{2})$, then $r_{k}\leq\left(\frac{1}{{\lfloor\frac{k-k_{0}}{T+K}\rfloor}b_{2}{({1-2\theta})}+r_{{0}}^{2\theta-1}}\right)^{\frac{1}{1-2\theta}}$,

where

[TABLE]

with $\beta^{\prime\prime}=\max\{\beta^{\prime},0\}$ and $b_{2}=\min(\frac{1}{b_{1}e^{2}R},\frac{r_{0}^{2\theta-1}(R^{\frac{2\theta-1}{2\theta-2}}-1)}{1-2\theta})$ for a fixed number $R\in(1,\infty)$.

The convergence rates in Theorem 7 match the results from (?), but they need the independence assumption between blocks and delays. If $T=0$ we obtain a synchronous version of the accelerated coordinate descent, and hence Theorem 7 implies the same rates as given in (?) for nonconvex functions.

Remark 6**.**

The characterization of the factor $b_{1}$ in Theorems 6 and 7 is noticeable in a particular way that the delays greater than $T_{1}$ contribute to this factor. Since $\beta^{\prime\prime}\leq 0$, it shows that applying negative momentum for high delay values could efficiently decrease the value of $b_{1}$ which results in acceleration.

Remark 7**.**

The KL property of $F$ is not necessarily sufficient to ensure that the Lyapunov function $G$ satisfies the KL property. However, since $G-F$ is semi-algebraic and the class of semi-algebraic functions is closed under addition, it shows that $G$ is semi-algebraic, which implies that $G$ is a KL function.

Numerical Results

In this section we test the efficiency of the asynchronous stochastic proximal coordinate descent algorithm with momentum acceleration. We performed binary classifications on the benchmark dataset rcv1. Following the practices in (?), we consider the logistic loss function with nonconvex regularization,

[TABLE]

with $\lambda=0.0001$, $\theta=0.1\lambda$ and the zero vector as starting point. Figure 1 demonstrates the speedups of our algorithm. AAPCD has significant linear speedup on a parallel platform with shared memory compared to its sequential counterpart.

We conduct experiments for comparing AAPCD with other asynchronous algorithms: ASCD (?), an synchronous version of doubly stochastic proximal algorithm (DSPG) (?), AASCD (?). ASCD and DSPG did not utilize the momentum acceleration techniques. AASCD is an asynchronous accelerated variant of ASCD but only for convex and strongly convex functions. For all experiments we set the number of local workers to $32$. We set $\lambda=0.0001$, $\theta=0.1\lambda$. For AAPCD, we set $\eta=0.08$, $\beta=-0.08$ for negative momentum, $\beta=0.8$ for positive momentum and threshold $T_{1}=0.9\tau$. All blocks are of size $1000$. We set the stepsize for ASCD with $\eta=0.06$. In AASCD we set $\eta=0.09$, with momentum value $\theta_{1}=0.8$. For DSPG, the stepsize is $\eta=0.03$ and mini-batch size is $200$. All algorithms are terminated when the number of iterations exceeds $100$. Note that we use the best tuned parameters for each method which is obtained over a refined grid to attain the best performance. Figure 2 shows the convergence of the objective function with respect to CPU time and the number of iterations.

Towards the end AAPCD decreases rapidly and needs much fewer iterations and less computing time than ASCD and AASCD to reach the same objective function values. This means that our AAPCD algorithm is very efficient and attains the best performance. Moreover AAPCD obtains a much smaller objective value by order of magnitudes compared with other algorithms. For saving space, we leave another experiment for Sigmoid loss in the supplementary materials.

Figure 3 shows AAPCD by only applying nonnegative momentum values which is slower than AAPCD, showing that linear extrapolation using negative momentum $\beta$ for large delays is significantly useful.

In summary our experimental results validate that AAPCD can indeed accelerate the convergence in practice.

Conclusion

In this paper, we have studied the stochastic and deterministic asynchronous parallelization of coordinate descent algorithm with momentum acceleration for efficiently solving nonconvex nonsmooth problems. We have shown that every limit point is a critical point and proved the convergence rates for deterministic AAPCD with bounded delay. We verified the advantages of our method through numerical experiments.

Overall speaking, these asynchronous proximal algorithms can be highly efficient when being used to solve large scale nonconvex nonsmooth problems. As for future work, an extension of this study might develop the analysis in this paper to inexact proximal methods. We also plan to investigate the asynchronous parallelization of more algorithms for nonconvex nonsmooth programming for solving more complicated models.

Proof of Lemma 1

Proof.

Since $x^{k+1}_{j_{k}}=\text{Prox}_{j_{k},\eta g_{j_{k}}}(y^{k}-\eta\nabla_{j_{k}}f(\hat{y}^{k}))$, we have

[TABLE]

As $f$ is $L$-Lipschitz smooth,

[TABLE]

Combining with (1), we obtain

[TABLE]

where we used $x^{k+1}_{j}=y^{k}_{j}$ for $j\neq j_{k}$. This is equivalent to,

[TABLE]

For the cross term we have

[TABLE]

where $a)$ is by the Lipschitz of $\nabla f$, $b)$ by the Cauchy-Schwarz inequality. By taking expectation over $d_{k}$, the following sequence of inequalities is true for any $C>0$:

[TABLE]

where $a)$ is due the triangle inequality and $d_{k}\leq\tau$. The linear extrapolation step for the momentum acceleration in Algorithm 1 yields

[TABLE]

Thus, by taking total expectation on both sides of (5) we have

[TABLE]

Therefore, we have

[TABLE]

Hence, we can derive

[TABLE]

By choosing $C={L\tau(1+\beta)}$, we obtain

[TABLE]

and the result follows from the definition of $G(x^{k+1})$. ∎

Proof of Theorem 1

Proof.

Applying Lemma 1, we obtain that

[TABLE]

Since $\eta<\frac{1}{L+2LT_{1}(1+\beta)}$ and $-1<\beta_{k}<\frac{1}{L\tau}(\frac{1}{2\eta}-\frac{L}{2})-1$, it follows that $\mathbb{E}[G(x^{k+1})]\leq\mathbb{E}[G(y^{k})]$. Moreover, the update rule of AAPCD guarantees that $F(y^{k+1})\leq F(x^{k+1})$. In summary, for all $k$ the following inequality holds:

[TABLE]

Hence, from (11), we obtain

[TABLE]

From (13) it is seen that $\mathbb{E}\left\lVert x^{k+1}-y^{k}\right\rVert^{2}$ is summable (telescoping sum). Thus, $\mathbb{E}\left\lVert x^{k+1}-y^{k}\right\rVert\to 0$ and consequently using (6) we have $\mathbb{E}\left\lVert y^{k+1}-y^{k}\right\rVert\to 0$, which means $\mathbb{E}[\xi_{k}]\to 0$. Combing further (12) with the fact that $F(x^{k}),F(y^{k})\geq\inf F>-\infty$ for all $k$ and $\mathbb{E}\left\lVert y^{k+1}-y^{k}\right\rVert\to 0$, we conclude that $\{G(x^{k})\},\{G(y^{k})\}$ converge to the same limit $F^{*}$, i.e.,

[TABLE]

On the other hand, by induction we conclude from equation (12) that for all $k$

[TABLE]

Combining with Assumption 1 that $F$ has bounded sublevel set, we conclude that $\{x^{k}\}$ and $\{y^{k}\}$ are almost surely bounded and thus have bounded limit points.

Since $\mathbb{E}\left\lVert x^{k+1}-y^{k}\right\rVert\to 0$, and hence ${x^{k+1}}$ and ${y^{k}}$ share the same set of limit points denoted by $\Omega$. We let $l(k,j)\in\mathbb{N}$ be the last time coordinate $j$ was updated:

[TABLE]

On the other hand, by optimality condition of the proximal gradient step of AAPCD, we obtain that

[TABLE]

and for $j\neq j_{k}$,

[TABLE]

We have

[TABLE]

where $a)$ is from (15) and (16), $b)$ by by Lipschitz of $\nabla f$, $c)$ by applying the triangle inequality and $d)$ from Assumption 3 and the assumption of bounded staleness. The right hand terms converge to 0 and hence $\mathbb{E}\left\lVert q^{k+1}_{j}\right\rVert\to 0$. Since $\mathbb{E}\left\lVert x^{k+1}-y^{k}\right\rVert^{2}$ is summable, it implies that $\mathbb{E}\left\lVert x^{k+1}-y^{k}\right\rVert^{2}=o(\frac{1}{k+1})$. Therefore we have $\mathbb{E}[\text{dist}_{\partial F(y^{k+1})}(0)]\leq\mathbb{E}\left\lVert q^{k+1}_{j}\right\rVert=o(\frac{1}{\sqrt{k+1}})$. Consider any limit point $x\in\Omega$, and subsequences say $x^{k_{t}}\to x$, $y^{k_{t}}\to x$. By the definition of the proximal map, the proximal gradient step of AAPCD implies that

[TABLE]

Taking $\lim\sup$ on both sides and note that $x^{k+1}-y^{k}\to 0$, $y^{k_{t}}\to x$, we obtain that $\lim\sup_{t\to\infty}g_{j}(x^{k_{t}+1}_{j})\leq g_{j}(x_{j})$. Since $g$ is lower semicontinuous and $x^{k+1}\to x$, it follows that $\lim\sup_{t\to\infty}g_{j}(x^{k_{t}+1}_{j})\geq g_{j}(x_{j})$. Combining both inequalities, we conclude that $\lim_{t\to\infty}g_{j}(x^{k_{t}+1}_{j})=g_{j}(x_{j})$. Note that the continuity of $f$ yields $\lim_{t\to\infty}f(x^{k_{t}})=f(x)$, we then conclude that $\lim_{t\to\infty}F(x^{k_{t}})=F(x)$, and $\lim_{t\to\infty}G(x^{k_{t}})=F(x)$, since $\xi_{k}\to 0$. By (14) we have $\lim_{t\to\infty}F(x^{k_{t}})=F^{*}$, hence

[TABLE]

Thus by (19), $F$ remains constant on the compact set $\Omega$ (the set $\Omega$ is closed and bounded in $\mathbb{R}^{m}$). To this end, we have shown $x^{k_{t}}\to x$, $G(x^{k_{t}})\to F^{*}$. Further, we proved $q_{k_{t}}\in\partial F(x^{k_{t}})$ converges zero. We conclude that $0\in\partial F(x)$ for all $x\in\Omega$. ∎

Proof of Theorem 2

Proof.

Throughout the proof we assume that $r_{k}\neq 0$ for all $k$ because otherwise the conclusions hold trivially. From (13) we have

[TABLE]

By summing this inequality over $h=h-\tau-K,\ldots,k$ iterations we obtain

[TABLE]

Moreover, equations (15) and (17) imply that

[TABLE]

where $a)$ follows from Lipschitz of $\nabla f$, $b)$ is by the triangle inequality, $c)$ from Assumption 3, the assumption of bounded delays and by (6). We have shown in Theorem 1 that $F(x^{k})\to F^{*}$, and it is also clear that $\text{dist}_{\Omega}(x^{k})\to 0$. Thus, for any $\epsilon,\gamma>0$ there is $k_{0}$ such that for all $k\geq k_{0}$, we have

[TABLE]

Since $\Omega$ is compact and $F$ is constant on it, the Uniformized KL property implies that for all $k\geq k_{0}$

[TABLE]

Recall that $r_{k+1}:=F(y^{k+1})-F^{*}$. Then equation (24) is equivalent to

[TABLE]

where in the last inequality we used $r_{k+1}\leq F(x^{k+1})-F^{*}$ and $\phi^{\prime}(t)=et^{\theta-1}$ is nonincreasing. By taking expectation on both sides of this equation and using (22), we obtain

[TABLE]

The second inequality is by (21), and the equality is from the definition of $b_{1}$. Thus by using $F(x^{k-K-\tau})\leq F(x^{0})$, we have

[TABLE]

Part $1:$ Suppose that $\theta=1$, then for all $k$, we have $\mathbb{E}[r_{0}-r_{k+1}]\geq\frac{1}{b_{1}e^{2}}>0$, which cannot hold because $r_{0}<\frac{1}{b_{1}e^{2}}$. Thus, $\{G(y^{k})\}$ must converge in finitely many steps, which is by Theorem 1 is the stationary point of $F$.

Part $2:$ Suppose that $\theta=\frac{1}{2}$. We have

[TABLE]

which yields the result. ∎

Proof of Lemma 2

Proof.

Since $x^{k+1}_{j_{k}}=\text{Prox}_{j_{k},\eta g_{j_{k}}}(y^{k}-\eta\nabla_{j_{k}}f(\hat{y}^{k}))$, we have

[TABLE]

As $f$ is $L$-Lipschitz smooth,

[TABLE]

Combining with (29), we obtain

[TABLE]

where we used $x^{k+1}_{j}=y^{k}_{j}$ for $j\neq j_{k}$. Therefore,

[TABLE]

Hence,

[TABLE]

where $a)$ is by the Lipschitz of $\nabla f$ and $b)$ is by Cauchy-Schwarz inequality. We bound the expectation of $\left\lVert y^{k}-\hat{y}^{k}\right\rVert^{2}$ over the delay. In particular, the following sequence of inequalities is true for any $C>0$:

[TABLE]

where in $a)$, we used $c_{0}=\sum_{t=1}^{\infty}t^{2}p_{t}$, in $b)$, we switched the order of summation in the double sum, and c) uses $c_{k-h}:=\sum_{t=1}^{\infty}t(t-h+k)p_{t-h+k}$. Taking total expectation on the equation above, we obtain

[TABLE]

where $d)$ follows from (6). Thus, we have

[TABLE]

which is substituted into (31) to yield

[TABLE]

Thus, we get

[TABLE]

Finally, by choosing $C=L(1+\beta)\sqrt{{c_{0}}}$, we have

[TABLE]

In particular for all $k\in\mathbb{N}$, we have $G(y^{k})=F(y^{k})+\xi_{k}$ so (6) follows. ∎

Proof of Theorem 3

Proof.

From Lemma 2 we have

[TABLE]

Since $\eta<\frac{1}{L+2L{\sqrt{c_{T_{1}}}}(1+\beta)}$ and by the upper bound for $\beta_{k}$, it follows that $\mathbb{E}[G(x^{k+1})]\leq\mathbb{E}[G(y^{k})]$. Moreover, the update rule of AAPCD guarantees that $F(y^{k+1})\leq F(x^{k+1})$ and so we have $\mathbb{E}[G(y^{k+1})]\leq\mathbb{E}[G(x^{k+1})]$. In summary, for all $k$ the following inequality holds:

[TABLE]

Thus, from (39) we obtain

[TABLE]

Hence $\mathbb{E}\left\lVert x^{k+1}-y^{k}\right\rVert^{2}$ is summable and $\mathbb{E}\left\lVert x^{k+1}-y^{k}\right\rVert^{2}$ converges to zero. Further, by the fact $\sum_{h=1}^{\infty}\mathbb{E}\left\lVert y^{h}-y^{h-1}\right\rVert^{2}<\infty$ and using $\sum_{h=1}^{\infty}c_{h}<\infty$, the series $S_{k}=\sum_{h=1}^{k}c_{k-h}\mathbb{E}\left\lVert y^{h}-y^{h-1}\right\rVert^{2}$ converges to zero, i.e., $\mathbb{E}[\xi_{k}]\to 0$. Combing further with the fact that $F(x^{k}),F(y^{k})\geq\inf F>-\infty$ for all $k$ and (40), we conclude that $\{F(x^{k})\},\{F(y^{k})\}$ converge to the same limit $F^{*}$, i.e.,

[TABLE]

Since $\mathbb{E}\left\lVert y^{k}-x^{k+1}\right\rVert\to 0$, and hence ${x^{k}}$ and ${y^{k}}$ share the same set of limit points which is denoted by $\Omega$. Similar to the analysis of AAPCD with bounded delay, using the optimality condition of the proximal step of AAPCD, we obtain that

[TABLE]

and for $j\neq j_{k}$,

[TABLE]

By the Assumption 3, we can derive

[TABLE]

where $a)$ is by Lipschitz of $\nabla f$, $b)$ by the triangle inequality, $c)$ from (33) and $d)$ by using a telescoping sum. As $k\to\infty$, this right term converges to zero and therefore we have $\mathbb{E}\left\lVert q_{j}^{k+1}\right\rVert^{2}\to 0$. On the other hand, by induction we conclude from equation (42) that for all $k$

[TABLE]

Combining with Assumption 1 that $F$ has bounded sublevel set, we conclude that $\{x^{k}\}$ and $\{y^{k}\}$ are almost surely bounded and thus have bounded limit points. We assume that $\{x^{k}\}$ and $\{y^{k}\}$ are bounded and thus have bounded limit points. We fix any limit point $x\in\Omega$, say $x^{k_{t}}\to x$, $y^{k_{t}}\to x$. Note that the continuity of $f$ yields $\lim_{t\to\infty}f(x^{k_{t}})=f(x)$. Moreover, by the definition of the proximal map, the proximal gradient step of AAPCD implies that

[TABLE]

Hence, we have

[TABLE]

where the last inequality is by the triangle inequality. Taking $\lim\sup$ on both sides and note that $x^{k_{t}}-y^{k_{t}}\to 0$, $y^{k_{t}}\to x$, we obtain that $\lim\sup_{t\to\infty}g_{j}(x^{{k_{t}}}_{j})\leq g_{j}(x)$. Since $g_{j}$ is lower semicontinuous and $x^{k_{t}}\to x$, it follows that $\lim\sup_{t\to\infty}g_{j}(x^{k_{t}}_{j})\geq g_{j}(x)$. Combining both inequalities we conclude that $\lim_{t\to\infty}g_{j}(x^{k_{t}}_{j})=g_{j}(x)$. Hence we have that $\lim_{t\to\infty}F(x^{k_{t}})=F(x)$. Since $\lim_{t\to\infty}F(x^{k_{t}})=F^{*}$ by equation (42), we get

[TABLE]

Hence, $F$ remains constant on the compact set $\Omega$. Since $\lim\mathbb{E}[\xi_{k}]\to 0$, we have, $\mathbb{E}[G(x^{k_{t}})]\to F(x)$ and thus $\mathbb{E}[G(x^{k_{t}})]\to F^{*}$. We have shown $x^{k_{t}}\to x$, $F(x^{k_{t}})\to F(x)$ and that $q_{k_{t}}\in\partial F(x^{k_{t}})$ converges zero. Altogether, we have $0\in\partial F(x)$ for all $x\in\Omega$. ∎

Proof of Theorem 4

Proof.

Throughout the proof we assume that $r_{k}\neq 0$ for all $k$ because otherwise the algorithm terminates and the conclusions hold trivially. Lemma 2 yields that

[TABLE]

Therefore, from (40) we obtain

[TABLE]

By summing this inequality over $h=k-K,\ldots,k$ iterations we obtain

[TABLE]

We have shown in Theorem 3 that $F(x^{k})\to F^{*}$, and it is also clear that $\text{dist}_{\Omega}(x^{k})\to 0$. Thus, for any $\epsilon,\gamma>0$ there is $k_{0}$ such that for all $k\geq k_{0}$, we have

[TABLE]

Since $\Omega$ is compact and $F$ is constant on it, we can apply KL property. The Uniformized KL property implies that for all $k\geq k_{0}$

[TABLE]

Moreover, equations (43) and (44) imply that

[TABLE]

where $a)$ follows from (53), $b)$ by $r_{k+1}\leq F(x^{k+1})-F^{*}$ and the fact that $\phi^{\prime}$ is nonincreasing, $c)$ from the Lipschitz of $\nabla f$ and $d)$ from the triangle inequality. We have that $\phi^{\prime}(t)={e}t^{\theta-1}$. Thus the above equation becomes

[TABLE]

By taking total expectation on both sides of this equation, and following the derivations similar to that of (45), we have

[TABLE]

where the second last inequality is by the definition of $b_{1}$ and (51). Thus from the above inequality we have

[TABLE]

Part $1:$ Suppose that $\theta=1$, then for all $k$, we have $\mathbb{E}[r_{0}-r_{k+1}]\geq\frac{1}{b_{1}e^{2}}>0$, which cannot hold because $\mathbb{E}[r_{k+1}]\geq 0$ and $r_{0}<\frac{1}{b_{1}e^{2}}$. Thus, $\{F(y^{k})\}$ must converge in finitely many steps, which is by Theorem 3 the stationary point of $F$.

Part $2:$ Suppose that $\theta=\frac{1}{2}$. We have from (57)

[TABLE]

which yields the result. ∎

Proof of Lemma 3

Proof.

Since $x^{k+1}_{j_{k}}=\text{Prox}_{j_{k},\eta g_{j_{k}}}(y^{k}-\eta\nabla_{j_{k}}f(\hat{y}^{k}))$, we have

[TABLE]

As $f$ is $L$-Lipschitz smooth,

[TABLE]

Combining with (59), we obtain

[TABLE]

where we used $x^{k+1}_{j}=y^{k}_{j}$ for $j\neq j_{k}$. This is equivalent to,

[TABLE]

For the cross term we have

[TABLE]

where $a)$ is by the Lipschitz of $\nabla f$, $b)$ by the Cauchy-Schwarz inequality. By taking expectation over $d_{k}$, the following sequence of inequalities is true for any $C>0$:

[TABLE]

where $a)$ is due the triangle inequality and $d_{k}\leq\tau$. The linear extrapolation step for the momentum acceleration in Algorithm 1 yields

[TABLE]

Thus, by taking total expectation on both sides of (63) we have

[TABLE]

Therefore, we have

[TABLE]

Hence, we can derive

[TABLE]

By choosing $C={L\tau(1+\beta)}$, we obtain

[TABLE]

and the result follows from the definition of $G(x^{k+1})$. ∎

Proof of Theorem 5

Proof.

Applying Lemma 3 with $x=x^{k}$, $y=y^{k}$, we obtain that

[TABLE]

Since $\eta=\frac{c}{L+2L\sqrt{{\delta_{0}}{\mu_{T_{1}}}}(1+\beta)}$ and $-1<\beta_{k}<{\frac{\sqrt{\mu_{T_{1}}}}{c\sqrt{\mu_{d_{k}}}}}(1+\beta)-1$, it follows that $G(x^{k+1})\leq G(y^{k})$. Moreover, the update rule of the deterministic AAPCD guarantees that $F(y^{k+1})\leq F(x^{k+1})$ and hence we have $G(y^{k+1})\leq G(x^{k+1})$. In summary, for all $k$ the following inequality holds:

[TABLE]

Hence from (69) we have

[TABLE]

This equation shows $\left\lVert x^{h}-y^{h-1}\right\rVert^{2}$ is summable. Thus, we have $\lim_{k}\left\lVert x^{k}-y^{k-1}\right\rVert^{2}=0$. Since $\sum_{h=1}^{\infty}\delta_{h}<\infty$, the series $A_{k}=\sum_{h=1}^{k}\delta_{k-h}\left\lVert y^{h}-y^{h-1}\right\rVert^{2}$ converges to zero, i.e., $\xi_{k}\to 0$. Combing further with the fact that $F(x^{k}),F(y^{k})\geq\inf F>-\infty$ for all $k$, we conclude that $\{G(x^{k})\},\{G(y^{k})\}$ converge to the same limit $F^{*}$, i.e.,

[TABLE]

On the other hand, by induction we conclude from equation (70) that for all $k$

[TABLE]

Combining with Assumption 1 that $F$ has bounded sublevel set, we conclude that $\{x^{k}\}$ and $\{y^{k}\}$ are bounded and thus have bounded limit points.

Since $\left\lVert y^{k}-x^{k+1}\right\rVert\to 0$, ${x^{k}}$ and ${y^{k}}$ share the same set of limit points $\Omega$ which is compact in $\mathbb{R}^{m}$. We fix any limit point $x\in\Omega$, say $x^{k_{t}+1}\to x$, $y^{k_{t}}\to x$. Note that the continuity of $f$ yields $\lim_{t\to\infty}f(x^{k_{t}})=f(x)$. By the definition of $x^{k+1}_{j}$ as a proximal point, we have

[TABLE]

Taking $\lim\sup$ on both sides and note that $x^{k_{t}+1}-y^{k_{t}}\to 0$, $y^{k_{t}}\to x$, we obtain that $\lim\sup_{t\to\infty}g_{j}(x^{k_{t}+1}_{j})\leq g_{j}(x_{j})$. Since $g$ is lower semicontinuous and $x^{k_{t}}\to x$, it follows that $\lim\sup_{t\to\infty}g_{j}(x^{k_{t}+1}_{j})\geq g_{j}(x_{j})$. Combining both inequalities, we conclude that $\lim_{t\to\infty}g_{j}(x^{k_{t}+1}_{j})=g_{j}(x_{j})$. We then obtain $\lim_{t\to\infty}F(x^{k_{t}})=F(x)$. Since $\lim_{t\to\infty}F(x^{k_{t}})=F^{*}$ by equation (70), we have

[TABLE]

Thus, $F$ remains constant on the set of limit points $\Omega$. By optimality condition of the proximal gradient step of AAPCD, we obtain that

[TABLE]

and for $j\neq j_{k}$,

[TABLE]

We have also

[TABLE]

where $\left\lVert\frac{\partial y^{k+1}}{\partial x^{k+1}}\right\rVert\leq(1+\beta_{k})$. For $k+1\in S_{T}$ we have

[TABLE]

where $a)$ follows from (75) and (76), $b)$ by the Lipschitz of $\nabla f$, $c)$ is by the triangle inequality, $d)$ from triangular inequality and and the fact that $k+1\in S_{T}$ and $e)$ is due to Assumption 3. Thus, from this we have $\left\lVert q_{j}^{k+1}\right\rVert\to 0$. We have shown $x^{k_{t}}\to x$, $F(x^{k_{t}})\to F(x)$ and that $\partial F(x^{k_{t}})$ converges to [math]. Therefore, $0\in\partial F(x)$ for all $x\in\Omega$. ∎

Proof of Theorem 6

Proof.

From equation (71) we have

[TABLE]

Recall $\eta=\frac{c}{L+2L\sqrt{{\delta_{0}}{\mu_{T_{1}}}}(1+\beta)}$ for $c\in(0,1)$, and $-1<\beta_{k}<{\frac{\sqrt{\mu_{T_{1}}}}{c\sqrt{\mu_{d_{k}}}}}(1+\beta)-1$ implies

[TABLE]

By summing the above inequality over ${k+1-T-K},\ldots,k$ iterations we obtain

[TABLE]

We have shown in Theorem 5 that $F(x^{k})\to F^{*}$, and it is also clear that $\text{dist}_{\Omega}(x^{k})\to 0$. Thus, for any $\epsilon,\gamma>0$ there is $k_{0}$ such that for all $k\geq k_{0}$, we have

[TABLE]

Since $\Omega$ is compact and $F$ is constant on it, the Uniformized KL property implies that for all $k\geq k_{0}$

[TABLE]

Recall that ${r_{k}:=F(y^{k})-F^{*}}$. Then, we have

[TABLE]

where $a)$ follows from (83), $b)$ is due to $r_{k+1}\leq F(x^{k+1})-F^{*}$ and the fact that $\phi^{\prime}$ is nonincreasing, $c)$ from (78), $d)$ is a direct computation using (81) and $\beta^{\prime\prime}=\max\{\beta^{\prime},0\}$, $e)$ is also a result of the definition of $b_{1}$ and $f)$ by $G(y^{k+1-T-K})\leq F(y^{0})$ and $F(y^{k+1})\leq G(y^{k+1})$. We have that $\phi^{\prime}(t)={e}t^{\theta-1}$. Thus the above inequality becomes

[TABLE]

Part $1:$ Suppose that $\theta=1$, then for all $k$, we have $r_{0}-r_{k+1}\geq\frac{1}{b_{1}e^{2}}>0$, which cannot hold because $r_{0}<\frac{1}{b_{1}e^{2}}$. Thus, $\{F(y^{k})\}$ must converge in finitely many steps, which is by Theorem 5 the stationary point of $F$.

In the following we assume that $r_{k}\neq 0$ for all $k\in S_{T}$ because otherwise the algorithm terminates.

Part $2:$ Suppose that $\theta\in[\frac{1}{2},1)$. We select $k_{1}$ large enough such that $r_{k+1}^{2-2\theta}\geq r_{k+1}$, for all $k\geq k_{1}$. Then for all $k\geq k_{1}$, and $k+1\in S_{T}$ we have

[TABLE]

Part $3:$ Suppose that $\theta\in(0,\frac{1}{2})$. Let $h(s):=s^{2\theta-2}$. Then from (85) we find that

[TABLE]

Let $R\in(1,\infty)$ be a fixed number. We consider two cases:

Case 1: Let $\frac{h(r_{k+1})}{h(r_{0})}\leq R$. Then we have

[TABLE]

Case 2: Let $\frac{h(r_{k+1})}{h(r_{0})}\geq R$. Then, since $h(r_{k+1})\geq h(r_{0})R$, we have $r_{k+1}^{2\theta-1}\geq r_{0}^{2\theta-1}R^{\frac{2\theta-1}{2\theta-2}}$. Therefore, we obtain

[TABLE]

Thus, we can deduce that

[TABLE]

Combining equations (88) and (89) yields

[TABLE]

Hence we have

[TABLE]

where $b_{2}=\min(\frac{1}{b_{1}e^{2}R},\frac{r_{0}^{2\theta-1}(R^{\frac{2\theta-1}{2\theta-2}}-1)}{1-2\theta})$. Thus, we have the following bound for all $k+1\in S_{T}$

[TABLE]

∎

Proof of Theorem 7

Proof.

We have shown in Theorem 5 that $\tilde{G}(x^{k})\to F^{*}$, and it is also clear that $\text{dist}_{\Omega}(x^{k})\to 0$. Furthermore, similar to the proof of Theorem 5, we can show the elements of $\Omega$ are the critical points of $\tilde{G}$. Thus, for any $\epsilon,\gamma>0$ there is $k_{0}$ such that for all $k\geq k_{0}$, we have

[TABLE]

Since $\Omega$ is compact and $\tilde{G}$ is constant on it, the Uniformized KL property implies that for all $k\geq k_{0}$

[TABLE]

Recall that ${r_{k}:=\tilde{G}(y^{k})-F^{*}}$. Then, we have

[TABLE]

where $a)$ follows from (94), $b)$ is due to $r_{k+1}\leq\tilde{G}(x^{k+1})-F^{*}$ and the fact that $\phi^{\prime}$ is nonincreasing, $c)$ from (78) and (77), $d)$ is a direct computation using (81) and $\beta^{\prime\prime}=\max\{\beta^{\prime},0\}$, and $e)$ is also a result of the definition of $b_{1}$. We have that $\phi^{\prime}(t)={e}t^{\theta-1}$. Thus, we obtain

[TABLE]

Part $1:$ Suppose that $\theta=1$, then for all $k$, by (96) we have $r_{k+1-T-K}-r_{k+1}\geq\frac{1}{b_{1}e^{2}}>0$, which cannot hold because $r_{k}\to 0$. Thus, $\{G(y^{k})\}$ must converge in finitely many steps, which is the stationary point of $\tilde{G}$.

In the following we assume that $r_{k}\neq 0$ for all $k\in S_{T}$ because otherwise the algorithm terminates.

Part $2:$ Suppose that $\theta\in[\frac{1}{2},1)$. We select $k_{1}$ large enough such that $r_{k+1}^{2-2\theta}\geq r_{k+1}$, for all $k\geq k_{1}$. Then for all $k\geq k_{1}$, and $k+1\in S_{T}$ we have

[TABLE]

Part $3:$ Suppose that $\theta\in(0,\frac{1}{2})$. Let $h(s):=s^{2\theta-2}$. Then from (96) we find that

[TABLE]

Let $R\in(1,\infty)$ be a fixed number. We consider two cases:

Case 1: Let $\frac{h(r_{k+1})}{h(r_{k+1-T-K})}\leq R$. Then we have

[TABLE]

Case 2: Let $\frac{h(r_{k+1})}{h(r_{k+1-T-K})}\geq R$. Then, since $h(r_{k+1})\geq h(r_{k+1-T-K})R$, we have $r_{k+1}^{2\theta-1}\geq r_{k+1-T-K}^{2\theta-1}R^{\frac{2\theta-1}{2\theta-2}}$. Therefore, we obtain

[TABLE]

Thus, we can deduce that

[TABLE]

Combining equations (99) and (100) yields

[TABLE]

From equation (101) we have

[TABLE]

where $b_{2}=\min(\frac{1}{b_{1}e^{2}R},\frac{r_{0}^{2\theta-1}(R^{\frac{2\theta-1}{2\theta-2}}-1)}{1-2\theta})$. Thus, we have the following bound for all $k\geq k_{0}$

[TABLE]

∎

Additional Numerical Results

We focus here on Sigmoid loss with $l_{2}$ regularization term:

[TABLE]

and

[TABLE]

Experiments are performed on $\it{Covetype}$ dataset. We conduct experiments for comparing AAPCD with other algorithms: ASCD, AASCD, and DSPG. For all the experiments we set the number of workers P=32. The block size for all experiments is $10$. The convergence results are presented in Figure 1. The results from AAPCD outperforms other algorithms. Moreover AAPCD obtains a much smaller objective value. Note that DSPG and ASCD have similar performance. AASCD is faster than ASCD, but the analyses for AASCD is only developed for convex functions. Overall, the results show that our algorithm is very efficient for nonconvex functions such as Sigmoid loss.