On the Sublinear Convergence of Randomly Perturbed Alternating Gradient Descent to Second Order Stationary Solutions

TL;DR

This paper introduces a perturbed alternating gradient descent algorithm that efficiently converges to second-order stationary solutions in nonconvex optimization, avoiding saddle points and local maxima with high probability.

Contribution

It proposes a novel perturbed AGD variant that guarantees convergence to second-order stationary solutions with a sublinear rate, a first in alternating algorithms.

Findings

Converges to second-order stationary solutions with high probability.

Achieves a sublinear convergence rate of (polylog(d)/^{7/3}) iterations.

Avoids saddle points and local maxima effectively.

Abstract

The alternating gradient descent (AGD) is a simple but popular algorithm which has been applied to problems in optimization, machine learning, data ming, and signal processing, etc. The algorithm updates two blocks of variables in an alternating manner, in which a gradient step is taken on one block, while keeping the remaining block fixed. When the objective function is nonconvex, it is well-known the AGD converges to the first-order stationary solution with a global sublinear rate. In this paper, we show that a variant of AGD-type algorithms will not be trapped by "bad" stationary solutions such as saddle points and local maximum points. In particular, we consider a smooth unconstrained optimization problem, and propose a perturbed AGD (PA-GD) which converges (with high probability) to the set of second-order stationary solutions (SS2) with a global sublinear rate. To the best of our knowledge, this is the first alternating type algorithm which takes $\mathcal{O}(\text{polylog}(d)/\epsilon^{7/3})$ iterations to achieve SS2 with high probability [where polylog$(d)$ is polynomial of the logarithm of dimension $d$ of the problem].

Tables

Table 1. Table 1: Convergence rates of algorithms to SS2 with the first order information, where p ≥ 4 𝑝 4 p\geq 4 , and 𝒪 ~ ~ 𝒪 \widetilde{\mathcal{O}} hides factor ploylog( d 𝑑 d ).

Algorithm	Iterations	$(ϵ,\gamma )$-SS2
SGD Ge et al. [2015]	$𝒪\left(d^p/{ϵ}^4\right)$	$(ϵ,{ϵ}^{1/4})$
SGLD Zhang et al. [2017]	$𝒪\left(d^p/{ϵ}^4\right)$	$(ϵ,{ϵ}^{1/2})$
Neon+SGD Xu and Yang [2017]	$\widetilde{𝒪}\left(1/{ϵ}^4\right)$	$(ϵ,{ϵ}^{1/2})$
Neon+Natasha Xu and Yang [2017]	$\widetilde{𝒪}\left(1/{ϵ}^{13/4}\right)$	$(ϵ,{ϵ}^{1/4})$
Neon2+SGD Allen-Zhu and Li [2017]	$\widetilde{𝒪}\left(1/{ϵ}^4\right)$	$(ϵ,{ϵ}^{1/2})$
${\text{Neon}}^{+}$ Xu et al. [2017]	$\widetilde{𝒪}\left(1/{ϵ}^{7/4}\right)$	$(ϵ,{ϵ}^{1/2})$
PGD Jin et al. [2017a]	$\widetilde{𝒪}\left(1/{ϵ}^2\right)$	$(ϵ,{ϵ}^{1/2})$
PAGD Jin et al. [2017b]	$\widetilde{𝒪}\left(1/{ϵ}^{7/4}\right)$	$(ϵ,{ϵ}^{1/2})$
PA-GD/PA-PP (This work)	$\widetilde{𝒪}\left(1/{ϵ}^{7/3}\right)$	$(ϵ,{ϵ}^{1/3})$

Equations

$$\underset{\mathbf{x}\in {\mathbb{R}}^{d\times 1}}{\min }f(\mathbf{x})$$

$$\Vert \nabla f(\mathbf{x})-\nabla f(\mathbf{y})\Vert \le L\Vert \mathbf{x}-\mathbf{y}\Vert ,\forall \mathbf{x},\mathbf{y}.$$

$$\Vert {\nabla }_{k}f({\mathbf{x}}_{-k},{\mathbf{x}}_k)-{\nabla }_{k}f({\mathbf{x}}_{-k},{\mathbf{x}}_k^{\prime })\Vert \le L_k\Vert {\mathbf{x}}_k-{\mathbf{x}}_k^{\prime }\Vert ,\forall \mathbf{x},{\mathbf{x}}^{\prime }$$

$$\Vert {\nabla }_{k}f({\mathbf{x}}_{-k},{\mathbf{x}}_k)-{\nabla }_{k}f({\mathbf{x}}_{-k}^{\prime },{\mathbf{x}}_k)\Vert \le {\tilde{L}}_k\Vert {\mathbf{x}}_{-k}-{\mathbf{x}}_{-k}^{\prime }\Vert ,\forall \mathbf{x},{\mathbf{x}}^{\prime }.$$

$$\Vert {\nabla }^{2}f(\mathbf{x})-{\nabla }^{2}f(\mathbf{y})\Vert \le \rho \Vert \mathbf{x}-\mathbf{y}\Vert ,\forall \mathbf{x},\mathbf{y}.$$

$$\Vert \nabla f(\mathbf{x})\Vert \le ϵ,\text{and}{\lambda }_{\min }({\nabla }^{2}f(\mathbf{x}))\ge -\gamma$$

$${\mathbf{x}}_k^{(t+1)}={\mathbf{x}}_k^{(t)}-\eta {\nabla }_{k}f({\mathbf{h}}_{-k}^{(t)},{\mathbf{x}}_k^{(t)}),k=1,2$$

$$\Vert \nabla f(\mathbf{x})\Vert \le ϵ,\text{and}{\lambda }_{\min }({\nabla }^{2}f(\mathbf{x}))\ge -(L_{\max }\rho ϵ{)}^{1/3}$$

$$\mathcal{O}\left(\frac{L_{\max }^{5/3}{\mathcal{P}}_1^7{\mathcal{P}}_2^2\Delta f}{{\rho }^{1/3}{ϵ}^{7/3}}{\log }^7\left(\frac{{\mathcal{P}}_1^6{\mathcal{P}}_2^{2}d{L}_{\max }^{5/3}{\Delta }_f}{c^5{\rho }^{1/3}{ϵ}^{7/3}\delta }\right)\right)$$

$$\mathcal{O}\left(\frac{L_{\max }^{5/3}{\log }^2(2d)\Delta f}{{\rho }^{1/3}{ϵ}^{7/3}}{\log }^7\left(\frac{{\mathcal{P}}_1^6{\mathcal{P}}_2^{2}d{L}_{\max }^{5/3}{\Delta }_f}{c^5{\rho }^{1/3}{ϵ}^{7/3}\delta }\right)\right).$$

$${\mathbf{x}}_k^{(t+1)}=\arg \underset{{\mathbf{x}}_k}{\min }f({\mathbf{h}}_{-k}^{(t)},{\mathbf{x}}_k)+\frac{\nu }{2}\Vert {\mathbf{x}}_k-{\mathbf{x}}_k^{(t)}{\Vert }^2,k=1,2$$

$${\mathbf{x}}_k^{(t+1)}={\mathbf{x}}_k^{(t)}-\frac{1}{\nu }{\nabla }_{k}f({\mathbf{h}}_{-k}^{(t)},{\mathbf{x}}_k^{(t+1)}),k=1,2,$$

$$\Vert \nabla f(\mathbf{x})\Vert \le ϵ,\text{and}{\lambda }_{\min }({\nabla }^{2}f(\mathbf{x}))\ge -(L_{\max }\rho ϵ{)}^{1/3}$$

$$\mathcal{O}\left(\frac{L_{\max }^{5/3}{\mathcal{P}}^2\Delta f}{{\rho }^{1/3}{ϵ}^{7/3}}{\log }^7\left(\frac{{\mathcal{P}}^{2}d{L}_{\max }^{5/3}{\Delta }_f}{c^5{\rho }^{1/3}{ϵ}^{7/3}\delta }\right)\right)$$

$${\mathbf{x}}^{(t+1)}={\mathbf{x}}^{(t)}-\eta \nabla f({\mathbf{x}}^{(t)})={\mathbf{x}}^{(t)}-\eta \nabla f(0)-\eta \left({\int }_0^1{\nabla }^{2}f(\theta {\mathbf{x}}^{(t)})d\theta \right){\mathbf{x}}^{(t)}.$$

$${\mathbf{x}}^{(t+1)}=(\mathbf{I}-\eta \mathcal{H}){\mathbf{x}}^{(t)}-\eta {\Delta }^{(t)}{\mathbf{x}}^{(t)}-\eta \nabla f(0)$$

$${\mathbf{x}}^{(t+1)}$$

$$={\mathbf{x}}^{(t)}-\eta \nabla f(0)-\eta {\int }_0^1{\mathcal{H}}_l^{(t)}d\theta {\mathbf{x}}^{(t+1)}-\eta {\int }_0^1{\mathcal{H}}_u^{(t)}d\theta {\mathbf{x}}^{(t)}$$

$$\boldsymbol{\mathcal{H}}^{(t)}_{l}\triangleq\left[\begin{array}[]{cc}\boldsymbol{0}&\boldsymbol{0}\\ \\ \nabla^{2}_{21}f(\theta\mathbf{x}^{(t+1)}_{1},\theta\mathbf{x}^{(t)}_{2})&\boldsymbol{0}\\ \end{array}\right]\quad\textrm{and}\quad\boldsymbol{\mathcal{H}}^{(t)}_{u}\triangleq\left[\begin{array}[]{cc}\nabla^{2}_{11}f(\theta\mathbf{x}^{(t)}_{1},\theta\mathbf{x}^{(t)}_{2})&\nabla^{2}_{12}f(\theta\mathbf{x}^{(t)}_{1},\theta\mathbf{x}^{(t)}_{2})\\ \\ \boldsymbol{0}&\nabla^{2}_{22}f(\theta\mathbf{x}^{(t+1)}_{1},\theta\mathbf{x}^{(t)}_{2})\\ \end{array}\right].$$

$$\displaystyle\boldsymbol{\mathcal{H}}_{u}=\left[\begin{array}[]{cc}\nabla^{2}_{11}f(\mathbf{x}^{*})&\nabla^{2}_{12}f(\mathbf{x}^{*})\\ \boldsymbol{0}&\nabla^{2}_{22}f(\mathbf{x}^{*})\end{array}\right],\quad\boldsymbol{\mathcal{H}}_{l}=\left[\begin{array}[]{cc}\boldsymbol{0}&\boldsymbol{0}\\ \nabla^{2}_{21}f(\mathbf{x}^{*})&\boldsymbol{0}\end{array}\right],$$

$${\mathbf{x}}^{(t+1)}+\eta {\mathcal{H}}_l{\mathbf{x}}^{(t+1)}={\mathbf{x}}^{(t)}-\eta {\mathcal{H}}_u{\mathbf{x}}^{(t)}-\eta {\Delta }_u^{(t)}{\mathbf{x}}^{(t)}-\eta {\Delta }_l^{(t)}{\mathbf{x}}^{(t+1)}$$

$$\mathbf{M}\triangleq \mathbf{I}+\eta {\mathcal{H}}_l,\mathbf{T}\triangleq \mathbf{I}-\eta {\mathcal{H}}_u.$$

$${\mathbf{x}}^{(t+1)}={\mathbf{M}}^{-1}\mathbf{T}{\mathbf{x}}^{(t)}-\eta {\mathbf{M}}^{-1}{\Delta }_u^{(t)}{\mathbf{x}}^{(t)}-\eta {\mathbf{M}}^{-1}{\Delta }_l^{(t)}{\mathbf{x}}^{(t+1)}.$$

$${\lambda }_{\max }({\mathbf{M}}^{-1}\mathbf{T})>1+\frac{\eta \gamma }{1+L/L_{\max }}$$

$$f({\mathbf{x}}^{(t+1)})\le f({\mathbf{x}}^{(t)})-\sum _{k=1}^2\frac{\eta }{2}\Vert {\nabla }_{k}f({\mathbf{h}}_{-k}^{(t)},{\mathbf{x}}_k^{(t)}){\Vert }^2.$$

$$\Vert \nabla f({\tilde{\mathbf{x}}}^{(t)}){\Vert }^2\le 4\sum _{k=1}^2\Vert {\nabla }_{k}f({\tilde{\mathbf{h}}}_{-k}^{(t)},{\tilde{\mathbf{x}}}_k^{(t)}){\Vert }^2\le 4g_{\text{th}}^2$$

$${\mathbf{x}}^{(t+1)}+\eta {\mathcal{H}}_l^{\prime }{\mathbf{x}}^{(t+1)}={\mathbf{x}}^{(t)}-\eta {\mathcal{H}}_u^{\prime }{\mathbf{v}}^{(t)}-\eta {\Delta }_u^{\prime (t)}{\mathbf{x}}^{(t)}-\eta {\Delta }_l^{\prime (t)}{\mathbf{x}}^{(t+1)}$$

$$\boldsymbol{\mathcal{H}}^{\prime}_{u}=\left[\begin{array}[]{cc}\boldsymbol{0}&\nabla^{2}_{12}f(\widetilde{\mathbf{x}}^{(t)})\\ \boldsymbol{0}&\boldsymbol{0}\end{array}\right],\quad\boldsymbol{\mathcal{H}}^{\prime}_{l}=\left[\begin{array}[]{cc}\nabla^{2}_{11}f(\widetilde{\mathbf{x}}^{(t)})&\boldsymbol{0}\\ \nabla^{2}_{21}f(\widetilde{\mathbf{x}}^{(t)})&\nabla^{2}_{22}f(\widetilde{\mathbf{x}}^{(t)})\end{array}\right],$$

$$\boldsymbol{\mathcal{H}}^{\prime(t)}_{l}\triangleq\left[\begin{array}[]{cc}\nabla^{2}_{11}f(\theta\mathbf{x}^{(t+1)}_{1},\theta\mathbf{x}^{(t)}_{2})&\boldsymbol{0}\\ \\ \nabla^{2}_{21}f(\theta\mathbf{x}^{(t+1)}_{1},\theta\mathbf{x}^{(t+1)}_{2})&\nabla^{2}_{22}f(\theta\mathbf{x}^{(t+1)}_{1},\theta\mathbf{x}^{(t+1)}_{2})\\ \end{array}\right],\quad\boldsymbol{\mathcal{H}}^{\prime(t)}_{u}\triangleq\left[\begin{array}[]{cc}\boldsymbol{0}&\nabla^{2}_{12}f(\theta\mathbf{x}^{(t+1)}_{1},\theta\mathbf{x}^{(t)}_{2})\\ \\ \boldsymbol{0}&\boldsymbol{0}\\ \end{array}\right].$$

$${\mathbf{M}}^{\prime }\triangleq \mathbf{I}+\eta {\mathcal{H}}_l^{\prime }{\mathbf{T}}^{\prime }\triangleq \mathbf{I}-\eta {\mathcal{H}}_u^{\prime }.$$

$${\mathbf{T}}^{\prime -1}{\mathbf{M}}^{\prime }{\mathbf{x}}^{(t+1)}={\mathbf{x}}^{(t)}-\eta {\mathbf{T}}^{\prime -1}{\Delta }_u^{\prime (t)}{\mathbf{x}}^{(t)}-\eta {\mathbf{T}}^{\prime -1}{\Delta }_l^{\prime (t)}{\mathbf{x}}^{(t+1)}.$$

Full text

In this paper, we show that a variant of AGD-type algorithms will not be trapped by “bad” stationary solutions such as saddle points and local maximum points. In particular, we consider a smooth unconstrained optimization problem, and propose a perturbed AGD (PA-GD) which converges (with high probability) to the set of second-order stationary solutions (SS2) with a global sublinear rate. To the best of our knowledge, this is the first alternating type algorithm which takes $\mathcal{O}(\text{polylog}(d)/\epsilon^{7/3})$ iterations to achieve SS2 with high probability [where polylog$(d)$ is polynomial of the logarithm of dimension $d$ of the problem].

1 Introduction

In this paper, we consider a smooth and unconstrained nonconvex optimization problem

[TABLE]

where $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is twice differentiable.

There are many ways of solving problem (1), such as gradient descent (GD), accelerated gradient descent (AGD), etc. When the problem dimension is large, it is natural to split the variables into multiple blocks and solve the subproblems with smaller size individually. The block coordinate descent (BCD) algorithm, and many of its variants such as block coordinate gradient descent (BCGD) and alternating gradient descent (AGD) Bertsekas [1999]; Li and Liang [2017], are among the most powerful tools for solving large scale convex/nonconvex optimization problems Nesterov [2012]; Beck and Tetruashvili [2013]; Razaviyayn et al. [2013]; Hong et al. [2017]. The BCD-type algorithms partition the optimization variables into multiple small blocks, and optimize each block one by one following certain block selection rule, such as cyclic rule Tseng [2001], Gauss-Southwell rule Tseng and Yun [2009], etc.

In recent years, there are many applications of BCD-type algorithms in the areas of machine learning and data mining, such as matrix factorization Zhao et al. [2015]; Lu et al. [2017a, b], tensor decomposition, matrix completion/decomposition Xu and Yin [2013]; Jain et al. [2013], and training deep neural networks (DNNs) Zhang and Brand [2017]. Under relatively mild conditions, the convergence of BCD-type algorithms to first-order stationary solutions (SS1) have been broadly investigated for nonconvex and non-differentiable optimization Tseng [2001]; Grippo and Sciandrone [2000]. In particular, it is known that under mild conditions, these algorithms also achieve global sublinear rates Razaviyayn et al. [2014]. However, despite its popularity and significant recent progress in understanding its behavior, it remains unclear whether BCD-type algorithms can converge to the set of second-order stationary solutions (SS2) with a provable global rate, even for the simplest problem with two blocks of variables.

1.1 Motivation

Algorithms that can escape from strict saddle points – those stationary points that have negative eigenvalues – have wide applications. Many recent works have analyzed the saddle points in machine learning problems Kawaguchi [2016]. Such as learning in shallow networks, the stationary points are either global minimum points or strict saddle points. In two-layer porcupine neural networks (PNNs), it has been shown that most local optima of PNN optimizations are also global optimizers Feizi et al. [2017]. Previous work in Ge et al. [2015] has shown that the saddle points in tensor decomposition are indeed strict saddle points. Also, it has been shown that any saddle points are strict in dictionary learning and phase retrieval problems theoretically and numerically in Sun et al. [2015, 2017]; Wang et al. [2017b, a]. More recently, Ge et al. [2017] proposed a unified analysis of saddle points for a board class of low rank matrix factorization problems, and they proved that these saddle points are strict.

1.2 Related Work

Many recent works have been focused on the performance analysis and/or design of algorithms with convergence guarantees to local minimum points/SS2 for nonconvex optimization problems. These include the trust region method Conn et al. [2000], cubic regularized Newton’s method Nesterov and Polyak [2006]; Carmon and Duchi [2016], and a mixed approach of the first-order and seconde-order methods Reddi et al. [2017], etc. However, these algorithms typically require second-order information, therefore they incur high computational complexity when problem dimension becomes large.

There has been a line of work on stochastic gradient descent algorithms, where properly scaled Gaussian noise is added to the iterates of the gradient at each time [also known as stochastic gradient Langevin dynamics, (SGLD)]. Some theoretical works have pointed out that SGLD not only converges to the local minimum points asymptotically but also may escape from local minima Zhang et al. [2017]; Raginsky et al. [2017]. Unfortunately, these algorithms require a large number of iterations with $\mathcal{O}(1/\epsilon^{4})$ steps to achieve the optimal point. There are fruitful results that show some carefully designed algorithms can escape from strict saddle point efficiently, such as negative-curvature-originated-from noise (Neon) Xu and Yang [2017], Neon2 Allen-Zhu and Li [2017], Neon+Xu et al. [2017] and gradient descent with one-step escaping (GOSE) Yu et al. [2017]. The Neon-type of algorithms utilizes the stochastic first-order updates to find the negative curvature direction, and GOSE just needs one negative curvature descent step with calculation of eigenvectors when the iterates of the algorithm are near the saddle point for saving the computational burden.

On the other hand, there is also a line of work analyzing the deterministic GD type method. With random initializations, it has been shown that GD only converges to SS2 for unconstrained smooth problems Lee et al. [2016]. More recently, block coordinate descent, block mirror descent and proximal block coordinate descent have been proven to almost always converge to SS2 with random initializations Lee et al. [2017], but there is no convergence rate reported. Unfortunately, a follow-up study indicated that GD requires exponential time to escape from saddle points for certain pathological problems Du et al. [2017]. Adding some noise occasionally to the iterates of the algorithm is another way of finding the negative curvature. A perturbed version of GD has been proposed with convergence guarantees to SS2 Jin et al. [2017a], which shows a faster provable convergence rate than the ordinary gradient descent algorithm with random initializations. Furthermore, the accelerated version of PGD (PAGD) is also proposed in Jin et al. [2017b], which shows the fastest convergence rate among all Hessian free algorithms.

1.3 Scope of This Paper

In this work, we consider a smooth unconstrained optimization problem, and develop a perturbed AGD algorithm (PA-GD) which converges (with high probability) to the set of SS2 with a global sublinear rate. Our work is inspired by the works Jin et al. [2017a]; Ge et al. [2015], which developed novel perturbed GDs that escapes from strict saddle points. Similarly as in Jin et al. [2017a], we also divide the entire iterates of GD into three types of points: those whose gradients are large, those that are local minimum, and those that are strict saddle points. At a given point, when the size of the gradient is large enough, we just implement the ordinary AGD. When the gradient norm is small, which may be either strict saddle or local minimum, a perturbation will be added on the iterates to help to escape from the saddle points.

From the above section, we know that many works have been developed to make use of negative curvature information around the saddle points. Unfortunately, these techniques cannot be directly applied to the BCD/AGD- type of algorithms. The key challenge here is that at each iteration only part of the variables are updated, therefore we have access only to partial second order information at the points of interest. For example, consider a quadratic objective function shown in Figure 1. While fixing one block, the problem is strongly convex with respect to the other block, but the entire problem is nonconvex. Even if the iterates converge for each block to the minimum points within the block, the stationary point could still be a saddle point for the overall objective function. Therefore, the analysis of how AGD type of algorithms exploit the negative curvature is one of the main tasks in this paper.

To the best of our knowledge, there is no work on modifying AGD algorithms to escape from strict saddle points with any convergence rate. The main contributions of this work are as follows.

1.4 Contributions of This Work

In this paper, we design and analyze a perturbed AGD algorithm for solving an unconstrained nonconvex problem, namely perturbed AGD. Through the perturbation of AGD, the algorithm is guaranteed to converge to a set of SS2 of a nonconvex problem with high probability. By utilizing the matrix perturbation theory, convergence rate of the proposed algorithm is also established, which shows that the algorithm takes $\mathcal{O}(\text{polylog}(d)/\epsilon^{7/3})$ iterations to achieve an ($\epsilon,\epsilon^{1/3}$)-SS2 with high probability. Also, considering the fact that there is a strong relation between GD and proximal point algorithm, we also study a perturbed alternating proximal point (PA-PP) algorithm with some random perturbation. By leveraging the techniques proposed in this paper, we show that PA-PP, which may not need to calculate the gradient at each step, converges as fast as PA-GD in the order of $\epsilon$ . The comparison of the algorithms which only use the first order information for escaping from strict saddle points is summarized as shown in Table 1.

The main contributions of the paper are highlighted below:

To the best of our knowledge, it is the first time that the convergence analysis shows that some variants of AGD (using first-order information) can converge to SS2 for nonconvex optimization problems. 2. 2.

The convergence rate of the perturbed AGD algorithm is analyzed, where the choice of the step size is only dependent on certain maximum Lipschitz constant over blocks rather than all variables. This is one of the major difference between GD and AGD. 3. 3.

By further extending the analysis in this paper, we also show that PA-PP can also escape from the strict points efficiently with the speed of $\mathcal{O}(\text{polylog}(d)/\epsilon^{7/3})$ .

2 Preliminaries

2.1 Notation

Notation. Bold upper case letters without subscripts (e.g., $\mathbf{X},\mathbf{Y}$) denote matrices and bold lower case letters without subscripts (e.g., $\mathbf{x},\mathbf{y}$) represent vectors. Notation $\mathbf{x}_{k}$ denotes the $k$th block of vector $\mathbf{x}\in\mathbb{R}^{d\times 1}$. We use $\nabla_{k}f(\mathbf{x}_{-k},\mathbf{x}_{k})$ to denote the partial gradient with respect to its $k$th block variable while the remaining one is fixed. Notation $\mathbb{B}_{\mathbf{x}}(r)$ denotes a $d$-dimensional ball centered at $\mathbf{x}$ with radius $r$, and $\lambda_{\min}(\mathbf{X})$, $\lambda_{\max}(\mathbf{X})$ denote the smallest and largest eigenvalues of matrix $\mathbf{X}$ respectively.

2.2 Definitions

The objective function has the following properties.

Definition 1**.**

A differentiable function $f(\cdot)$ is L-smooth with gradient Lipschitz constant $L$ (uniformly Lipschitz continuous), if

[TABLE]

The function is called block-wise smooth with gradient Lipschitz constants $\{L_{k}\}$, if

[TABLE]

or with gradient Lipschitz constants $\{\widetilde{L}_{k}\}$, if

[TABLE]

Further, let $L_{\max}\triangleq\max\{L_{k},\widetilde{L}_{k},\forall k\}\leq L$.

Definition 2**.**

For a differentiable function $f(\cdot)$, if $\|\nabla f(\mathbf{x})\|=0$, then $\mathbf{x}$ is a first-order stationary point. If $\|\nabla f(\mathbf{x})\|\leq\epsilon$, then $\mathbf{x}$ is an $\epsilon$-first-order stationary point.

Definition 3**.**

For a differentiable function $f(\cdot)$, if $\mathbf{x}$ is a SS1, and there exists $\epsilon>0$ so that for any $\mathbf{y}$ in the $\epsilon$-neighborhood of $\mathbf{x}$, we have $f(\mathbf{x})\leq f(\mathbf{y})$, then $\mathbf{x}$ is a local minimum. A saddle point $\mathbf{x}$ is a SS1 that is not a local minimum. If $\lambda_{\min}(\nabla^{2}f(\mathbf{x}))<0$, $\mathbf{x}$ is a strict (non-degenerate) saddle point.

Definition 4**.**

A twice-differentiable function $f(\cdot)$ is $\rho$-Hessian Lipschitz if

[TABLE]

Definition 5**.**

For a $\rho$-Hessian Lipschitz function $f(\cdot)$, $\mathbf{x}$ is a second-order stationary point if $\|\nabla f(\mathbf{x})\|=0$ and $\lambda_{\min}(\nabla^{2}f(\mathbf{x}))\geq 0$. If the following holds

[TABLE]

where $\epsilon,\gamma>0$, then $\mathbf{x}$ is a $(\epsilon,\gamma)$-SS2.

Assumption 1**.**

Function $f(\cdot)$ is $L$-smooth, block-wise smooth with gradient Lipschitz constants $\{L_{k},\widetilde{L}_{k}\},k=1,2$, and $\rho$-Hessian Lipschitz.

3 Perturbed Alternating Gradient Descent

3.1 Algorithm Description

AGD is a classical algorithm that optimizes the variables of an optimization problem in an alternating manner Bertsekas [1999], meaning that when one block of variables is updated, the remaining block is fixed to be the same as its previous solution. Mathematically, the iterates of AGD are updated by the following rule

[TABLE]

where superscript $(t)$ denotes the iteration counter; $\mathbf{h}^{(t)}_{-1}\triangleq\mathbf{x}^{(t)}_{2}$ and $\mathbf{h}^{(t)}_{-2}\triangleq\mathbf{x}^{(t+1)}_{1}$; $\eta>0$ is the step size. AGD can be considered as a special case of block coordinate gradient descent Nesterov [2012]; Beck and Tetruashvili [2013].

Our proposed algorithm is based on AGD, but modified in a way similar to the recent work [Jin et al., 2017a], which adds some noise in PGD. The details of the implementation of PA-GD are shown in Algorithm 1, where $c$ is a constant so that $\eta=c/L_{\max}$, $\Delta_{f}$ denotes the difference of the objective value at the initial point and global optimal solution, $\epsilon$ represents the predefined target error.

In each update of variables, we implement one step of the block gradient descent, and then proceed to the next block. Once the algorithm has sufficient decrease of the objective value, it implies that the algorithm converges to some good solution. Otherwise, some perturbation may be needed to help the iterates escape from the saddle points. If after the perturbation the objective value does not decrease sufficiently after a number of further iterations, the algorithm terminates and returns the iterate before the last perturbation.

To illustrate the practical behavior of the algorithm, we provide an example that shows the trajectory of AGD after a small perturbation at a stationary point. In Figure 1, it is clear that $\mathbf{x}=[0;0]$ is a SS1 and also a strict saddle point since the eigenvalues of $\mathbf{A}$ are $-1$ and $3$ respectively. When $\mathbf{x}_{1}$ is fixed, function $f(\mathbf{x})$ is convex with respect to $\mathbf{x}_{2}$ and vice versa, however, the objective function is nonconvex. It can be observed that PA-GD can escape from the strict saddle point efficiently.

3.2 Convergence Rate Analysis

Despite the fact that PA-GD exploits a different way of updating variables, we will show that it can still escape from strict saddle points with high probability with suitable perturbation. The main theorem is presented as follows.

Theorem 1**.**

Under Assumption 1, there exists a constant $c_{\max}$ such that: for any $\delta\in(0,1]$, $\epsilon\leq\frac{L^{2}_{\max}}{\rho}$, $\Delta_{f}\triangleq f(\mathbf{h}^{(0)}_{-1},\mathbf{x}^{(0)}_{1})-f^{*}$, and constant $c\leq c_{\max}$, with probability $1-\delta$, the iterates generated by PA-GD converge to an $\epsilon$-SS2 $\mathbf{x}$ satisfying

[TABLE]

in the following number of iterations:

[TABLE]

where $f^{*}$ denotes the global minimum value of the objective function, and $\mathcal{P}_{1}=(1+L/L_{\max})$ and $\mathcal{P}_{2}=(1+L\log(2d)/L_{\max})$.

Remark 1. When $\eta=c_{\max}/L$ is used, the convergence rate of PA-GD is

[TABLE]

It shows that if a smaller step size is used, the convergence rate of PA-GD is faster (with smaller constants) since the linear dependency of $\mathcal{P}^{7}_{1}$ and $\mathcal{P}^{2}_{2}$ in (5) both disappear. This property is consistent with the known result when BCD is used in convex optimization problems, i.e., when a smaller step size is used, the rate could become better; e.g., see [Sun and Hong, 2015, Theorem 2.1].

4 Perturbed Alternating Proximal Point

In many applications, AGD may not be efficient in the sense that the convergence rate of gradient in each block may be very slow. For example, consider matrix factorization problem $\min_{\mathbf{X},\mathbf{Y}}\|\mathbf{Z}-\mathbf{X}\mathbf{Y}\|^{2}_{F}$ where $\mathbf{Z}\in\mathbb{R}^{m\times d}$ is the given data, $d\gg m$, and $\mathbf{X}\in\mathbb{R}^{m\times r},\mathbf{Y}\in\mathbb{R}^{r\times d}$ are two block variables. For this problem, the alternating least squares algorithm (which exactly minimizes each block) would be a faster algorithm compared with the AGD which only uses gradient steps.

In this section, we consider the classical proximal point algorithm Parikh et al. [2014] in which each block of variables is exactly minimized with respect to certain quadratic surrogate. To be specific, we can replace (4) in Algorithm 1 by

[TABLE]

where $\nu>0$ is penalty parameter. The iteration can be explicitly written as

[TABLE]

which has the similar form as the PA-GD algorithm, but with the step size being $\eta\triangleq 1/\nu$, and with gradient evaluated at the new iterate. The resulting algorithm, detailed in the table above, is referred to as the perturbed alternating proximal point (PA-PP). It is worth noting that when the subproblem is convex, such as $\min_{\mathbf{X},\mathbf{Y}}\|\mathbf{Z}-\mathbf{X}\mathbf{Y}\|^{2}_{F}$, $\nu$ only needs to be a small number to make the corresponding subproblem strongly convex. This property is useful in practice.

Next, we can also give the convergence rate of PA-PP.

Corollary 1**.**

Under Assumption 1, there exists a constant $c_{\max}$ such that: for any $\delta\in(0,1]$, $\epsilon\leq\frac{L^{2}_{\max}}{\rho}$, $\Delta_{f}\triangleq f(\mathbf{h}^{(0)}_{-1},\mathbf{x}^{(0)}_{1})-f^{*}$, and constant $c\leq c_{\max}$, with probability $1-\delta$, the iterates generated by PA-PP converges to an $\epsilon$-SS2 $\mathbf{x}$ satisfying

[TABLE]

in the following number of iterations:

[TABLE]

where $f^{*}$ denotes the global minimum value of the objective function, and $\mathcal{P}=(1+L\log(2d)/L_{\max})$.

Comparing with Theorem 1, we can find that term $\mathcal{P}^{7}_{1},\mathcal{P}_{1}>2$ is removed so the convergence rate of PA-PP is slightly faster than PA-GD.

5 Convergence Analysis

In this section, we will present the main proof steps of convergence analysis of PA-GD.

5.1 The Main Difficulty of the Proof

Gradient Descent:

GD searches the descent direction of the objective function in the entire space $\mathbb{R}^{d}$. Without loss of generality, we assume $\mathbf{x}^{(0)}=0$. According to the mean value theorem, the GD update can be expressed as

[TABLE]

It can be observed that the update rule of GD contains the information of the Hessian matrix at point $\mathbf{x}^{(t)}$, i.e., $\nabla^{2}f(\theta\mathbf{x}^{(t)})$. To be more specific, letting $\boldsymbol{\mathcal{H}}\triangleq\nabla^{2}f(\mathbf{x}^{*})$ where $\mathbf{x}^{*}$ denotes an $\epsilon$-SS2 satisfying (3), we can rewrite (9) as

[TABLE]

where $\Delta^{(t)}\triangleq\int^{1}_{0}(\nabla^{2}f(\theta\mathbf{x}^{(t)})-\boldsymbol{\mathcal{H}})d\theta$.

Based on the $\rho$-Hessian Lipschitz property, we can quantify $\|\Delta^{(t)}\|$ that is upper bounded by the difference of iterates. By exploiting the negative curvature of the Hessian matrix at saddle point $\mathbf{x}^{*}$, we can project the iterate onto the direction $\vec{d}$ where the eigenvalue of $\mathbf{I}-\eta\boldsymbol{\mathcal{H}}$ is greater than 1. This leads to the fact that the norm of the iterates projected along direction $\vec{d}$ will be increasing exponentially as the algorithm proceeds around point $\mathbf{x}^{*}$, implying the sequence generated by GD is escaping from the saddle point. The details of characterizing the convergence rate have been analyzed previously in Jin et al. [2017a].

Alternating Gradient Descent:

However, the AGD algorithm only updates partial variables of vector $\mathbf{x}$, which belong to a subspace of the feasible set. Similarly, from the mean value theorem we can express the AGD rule of updating variables with assuming $\mathbf{x}^{(0)}=0$ as follows:

[TABLE]

where

[TABLE]

From the above expression, it can be seen clearly that the update rule of AGD does not include a full Hessian matrix at any point but only partial ones. Furthermore, the right hand side of (14) not only contains the second order information of the previous point, i.e., $[\mathbf{x}^{(t)}_{1},\mathbf{x}^{(t)}_{2}]$ but also the one of the most recently updated point, i.e., $[\mathbf{x}^{(t+1)}_{1},\mathbf{x}^{(t)}_{2}]$. These represent the main challenges in understanding the behavior of the sequence generated by the AGD algorithm.

5.2 The Main Idea of the Proof

Although the second order information is divided into two parts, we can still characterize the recursion of the iterates around strict saddle points. We can also split $\boldsymbol{\mathcal{H}}$ as two parts, which are

[TABLE]

and obviously we have $\boldsymbol{\mathcal{H}}=\boldsymbol{\mathcal{H}}_{l}+\boldsymbol{\mathcal{H}}_{u}$.

Then, recursion (14) can be written as

[TABLE]

where $\Delta^{(t)}_{u}\triangleq\int^{1}_{0}(\boldsymbol{\mathcal{H}}^{(t)}_{u}(\theta)-\boldsymbol{\mathcal{H}}_{u})d\theta$, $\Delta^{(t)}_{l}\triangleq\int^{1}_{0}(\boldsymbol{\mathcal{H}}^{(t)}_{l}(\theta)-\boldsymbol{\mathcal{H}}_{l})d\theta$. However, it is still unclear from (20) how the iteration evolves around the strict saddle point.

To highlight ideas, let us define

[TABLE]

It can be observed that $\mathbf{M}$ is a lower triangular matrix where the diagonal entries are all 1s; therefore it is invertible. After taking the inverse of matrix $\mathbf{M}$ on both sides of (20), we can obtain

[TABLE]

Our goal of analyzing the recursion of $\mathbf{x}^{(t)}$ becomes to find the maximum eigenvalue of $\mathbf{M}^{-1}\mathbf{T}$. With the help of the matrix perturbation theory, we can quantify the difference between the eigenvalues of matrix $\boldsymbol{\mathcal{H}}$ that contains the negative curvature and matrix $\mathbf{M}^{-1}\mathbf{T}$ that we are interested in analyzing. To be more precise, we give the following lemma.

Lemma 1**.**

Under Assumption 1, let $\boldsymbol{\mathcal{H}}\triangleq\nabla^{2}f(\mathbf{x})$ denote the Hessian matrix at an $\epsilon$-SS2 $\mathbf{x}$ where $\lambda_{\min}(\boldsymbol{\mathcal{H}})\leq-\gamma$ and $\gamma>0$. We have

[TABLE]

where $\mathbf{M},\mathbf{T}$ are defined in (19) and (21).

Lemma 1 illustrates that there exits a subspace spanned by the eigenvector of $\mathbf{M}^{-1}\mathbf{T}$ whose eigenvalue is greater than 1, indicating that the sequence generated by AGD can still potentially escape from the strict saddle point by leveraging such negative curvature information. Next, we can give a sketch of the proof of Theorem 1.

5.3 The Sketch of the Proof

The structure of the proof for quantifying the sufficient decrease of the objective function after the perturbation is borrowed from the proof of PGD Jin et al. [2017a], but PA-GD updates the variables block by block, so we have to provide the new proofs to show that PA-GD can still escape from saddle points with the perturbation technique.

First, if the size of the gradient is large enough, Algorithm 1 just implements the ordinary AGD. We give the descent lemma of AGD as follows.

Lemma 2**.**

Under Assumption 1, for the AGD algorithm with step size $\eta<1/L_{\max}$, we have

[TABLE]

Second, if the iterates are near a strict saddle point, we can show that the AGD algorithm after a perturbation can give a sufficient decrease with high probability in terms of the objective value. To be more precise, the statement is given as follows.

Lemma 3**.**

Under Assumption 1, there exists a absolute constant $c_{\max}$. Let $c\leq c_{\max}$, $\chi\geq 1$, and $\eta$, $r$, $g_{\textsf{th}}$, $t_{\textsf{th}}$ calculated as Algorithm 1 describes. Let $\widetilde{\mathbf{x}}^{(t)}$ be a strict saddle point, which satisfies

[TABLE]

and $\lambda_{\min}(\nabla^{2}f(\widetilde{\mathbf{x}}^{(t)}))\leq-\gamma$, where $\widetilde{\mathbf{h}}^{(t)}_{-1}\triangleq\widetilde{\mathbf{x}}^{(t)}_{2}$ and $\widetilde{\mathbf{h}}^{(t)}_{-2}\triangleq\mathbf{x}^{(t+1)}_{1}$.

Let $\mathbf{x}^{(t)}=\widetilde{\mathbf{x}}^{(t)}+\xi^{(t)}$ where $\xi^{(t)}$ is generated randomly which follows the uniform distribution over $\mathbb{B}_{0}(r)$, and let $\mathbf{x}^{(t+t_{\textsf{th}})}$ be the iterates of PA-GD. With at least probability $1-\frac{dL_{\max}}{(L_{\max}\rho\epsilon)^{1/3}}e^{-\chi}$, we have $f(\mathbf{x}^{(t+t_{\textsf{th}})})-f(\widetilde{\mathbf{x}}^{(t)})\leq-f_{\textsf{th}}$.

We remark that Lemma 2 is well-known and Lemma 3 is the core technique. In the following, we outline the main idea used in proving the latter. The formal statements of these steps are shown in the appendix; see Lemma 8–Lemma 10 therein.

We emphasize that the main contributions of this paper lies in the analysis of the first two steps, where the special update rule of PA-GD is analyzed so that the negative curvature of $\boldsymbol{\mathcal{H}}$ around the saddle points can be utilized.

Step 1

(Lemma 8) Consider a generic sequence $\mathbf{u}^{(t)}$ generated by PA-GD. As long as the initial point of $\mathbf{u}^{(t)}$ is close to saddle point $\widetilde{\mathbf{x}}^{(t)}$, the distance between $\mathbf{u}^{(t)}$ and $\widetilde{\mathbf{x}}^{(t)}$ can be upper bounded by using the $\rho$-Hessian Lipschitz continuity property.

Step 2

(Lemma 9) Leveraging the negative curvature around the strict saddle point, we know that there exits a direction, i.e., $\vec{\mathbf{e}}$, which is spanned by the eigenvector of $\mathbf{M}^{-1}\mathbf{T}$ whose corresponding eigenvalue is largest (greater than 1). Consider two sequences generated by PA-GD, $\mathbf{u}^{(t)},\mathbf{w}^{(t)}$ initialized around the saddle point. When the initial points of these two iterates are separated apart away from each other along direction $\vec{\mathbf{e}}$ with a small distance, meaning that $\mathbf{w}^{(0)}=\mathbf{u}^{(0)}+\upsilon r\vec{\mathbf{e}},\;\upsilon\in[\delta/(2\sqrt{d}),1]$ where $r$ denotes the radius of the perturbation ball defined in Algorithm 1, we can show that if iterate $\mathbf{u}^{(t)}$ is still near the saddle point after $T$ steps, the other sequence $\mathbf{w}^{(t)}$ will give a sufficient decrease of the objective value with less than $T$ steps, implying that iterates $\mathbf{w}^{(t)}$ can escape from the saddle point with less than $T$ steps.

Step 3

(Lemma 10) Consider $\mathbf{u}^{(0)},\mathbf{w}^{(0)}$ as the points after the perturbation from the saddle point. We can quantify the probability that the AGD sequence will give a sufficient decrease of the objective value within $T$ iterations after the perturbation [Jin et al., 2017a, Lemma 14,15].

5.4 Extension to PA-PP

By leveraging the convergence analysis of PA-GD and relation between PA-GD and PA-PP shown in (8), we can also write the recursion of the PA-PP iteration as

[TABLE]

where $\eta=1/\nu$, $\Delta^{\prime(t)}_{u}\triangleq\int^{1}_{0}(\boldsymbol{\mathcal{H}}^{\prime(t)}_{u}(\theta)-\boldsymbol{\mathcal{H}}^{\prime}_{u})d\theta$, $\Delta^{\prime(t)}_{l}\triangleq\int^{1}_{0}(\boldsymbol{\mathcal{H}}^{\prime(t)}_{l}(\theta)-\boldsymbol{\mathcal{H}}^{\prime}_{l})d\theta$,

[TABLE]

and

[TABLE]

Let

[TABLE]

We know that $\mathbf{T}^{\prime}$ is an upper triangular matrix where the diagonal entries are all 1s, so it is invertible. Different from the case of PA-GD, we take the inverse of matrix $\mathbf{T}^{\prime}$ on both sides of (24) and obtain

[TABLE]

Then, we can give the following result that characterizes the recursion of $\mathbf{x}^{(t)}$ generated by PA-PP.

Corollary 2**.**

Under Assumption 1, let $\boldsymbol{\mathcal{H}}\triangleq\nabla^{2}f(\mathbf{x})$ denote the Hessian matrix at an $\epsilon$-SS2 $\mathbf{x}$ where $\lambda_{\min}(\boldsymbol{\mathcal{H}})\leq-\gamma$ and $\gamma>0$. Let ${\lambda}^{+}_{\min}(\cdot)$ denote the minimum positive eigenvalue of a matrix. Then we have

[TABLE]

where $\mathbf{M}^{\prime},\mathbf{T}^{\prime}$ are defined in (25) and (26); $\eta\leq 1/L_{\max}$ and $\gamma\leq L_{\max}$.

We remark that Corollary 2 is useful since it can be leveraged to show that the norm of the iterates around saddle points can increase exponentially. Then, we can apply the similar analysis steps as the case of proving the convergence rate of PA-GD and obtain the results shown in Corollary 1.

6 Connection with Existing Works

Remark 2. In Theorem 1 we characterized the convergence rate to an $(\epsilon,{\epsilon}^{1/3})$-SS2. We can also translate this bound to the one for achieving an $(\epsilon,\sqrt{\epsilon})$-SS2, and in this case PA-GD needs $\widetilde{\mathcal{O}}(1/\epsilon^{3.5})$ iterations. Compared with the existing recent works Jin et al. [2017a], the convergence rate of PA-GD/PA-PP is slower than GD. The main reason is the fact that different from GD-type algorithms, PA-GD and PA-PP cannot fully utilize the Hessian information because they never see a full iteration. Similar situation happens for SGD-type of algorithms which also cannot get the exact negative curvature around strict saddle points.

From Table 1, it can be seen that the convergence rate of PA-GD/PA-PP is still faster than SGD Ge et al. [2015], SGLD Zhang et al. [2017], Neon+SGD Xu and Yang [2017], and Neon2+SGD Allen-Zhu and Li [2017] to achieve an $(\epsilon,\sqrt{\epsilon})$-SS2, but slower than the rest. We emphasize that PA-GD and PA-PP represent the first BCD-type algorithms with the convergence rate guarantee to escape from the strict saddle points efficiently. At this point, it is unclear whether our rate is the best that is achievable, and the question of whether the resulting rate can be improved will be left to future work.

7 Numerical Results

In this section, we present a simple example that shows the convergence behavior of PA-GD. Consider a nonconvex objective function, i.e.,

[TABLE]

First, we have the following properties of function $f(\mathbf{x})$ such that $f(\mathbf{x})$ satisfies the assumptions of the analysis.

Lemma 4**.**

For any $\tau\geq\lambda_{\max}(\mathbf{A})$ and $\mathbf{x}\in\{\mathbf{x}|\|\mathbf{x}\|^{2}\leq\tau\}$, $f(\mathbf{x})$ defined in (28) is $5\tau$-smooth and $6\sqrt{\tau}$-Hessian Lipschitz.

Here, we can easily show the shape of objective function (28) in the two dimensional (2D) case in Figure 2(a), where $\mathbf{A}=[1\>2;2\;1]\in\mathbb{R}^{2\times 2}$. It can be observed clearly that there exits a strict saddle point at $[0,0]$ and two other local optimal points. We randomly initialize the algorithms around strict saddle point $[0,0]$. The convergence comparison between AGD and PA-GD is shown in Figure 2(b). It can be observed that PA-GD converges faster than AGD to a local optimal point.

8 Conclusion

In this paper, the perturbed variants of AGD and alternating proximal point (APP) algorithms are proposed, with the objective of finding the second order stationary solutions of nonconvex smooth problems. Leveraging the recently developed idea of random perturbation for the first-order methods, the proposed algorithms add suitable perturbation to the AGD or APP iterates. The main contribution of this work is a new analysis that takes into consideration the block structure of the updates for the perturbed AGD and APP algorithms. By exploiting the negative curvature, it is established that with high probability the algorithms can converge to an $(\epsilon,\epsilon^{1/3})$-SS2 with $\mathcal{O}(\text{polylog}(d)/\epsilon^{7/3})$ iterations.

9 Acknowledgment

The authors would like to thank Chi Jin for discussion on the perturbed gradient descent algorithm.

Appendix A Preliminary

We provide the proofs of some preliminary lemmas (Lemma 5–Lemma 7) used in the proof of Section B.

First, Lemma 5 and Lemma 6 give the property that quantify the size of the difference of the second-order information of the objective values between two points.

Lemma 5**.**

If function $f(\cdot)$ is $\rho$-Hessian Lipschitz, we have

[TABLE]

Lemma 6**.**

Under Assumption 1, we have block-wise Lipschitz continuity as follows:

[TABLE]

and

[TABLE]

Then, we illustrate that the size of the partial gradient with one round update by the AGD algorithm has the following relation with the full size of the gradient.

Lemma 7**.**

If function $f(\cdot)$ is $L$-smooth with Lipschitz constant, then we have

[TABLE]

where sequence $\mathbf{x}^{(t)}_{k},k=1,2$ is generated by the AGD algorithm.

A.1 Proof of Lemma 5

Proof.

If function $f(\cdot)$ is $\rho$-Hessian Lipschitz, then we have

[TABLE]

where $(a)$ is true because of Hessian Lipschitz, in $(b)$ we used the triangle inequality. ∎

A.2 Proof of Lemma 6

There proof involves two parts:

Upper Triangular Matrix:

Consider three different vectors $\mathbf{x}$, $\mathbf{y}$ and $\mathbf{z}$. We can have

[TABLE]

where in $(a)$ we used

[TABLE]

and $\|\mathbf{I}_{1}\|=\|\mathbf{I}_{2}\|=1$.

Lower Triangular Matrix:

[TABLE]

where $(a)$ is true because we know $\|\mathbf{I}_{1}\|=\|\mathbf{I}_{2}\|=1$.

A.3 Proof of Lemma 7

Proof.

Recall the definition

[TABLE]

First, we have

[TABLE]

Using block-wise Lipschitz continuity, we have

[TABLE]

where $(a)$ is because we use the update rule of AGD, $(b)$ is true due to $\eta\leq 1/L_{\max}$.

Summing $\|\nabla_{1}f(\mathbf{x}^{(t)}_{1},\mathbf{x}^{(t)}_{2})\|^{2}$ on both sides of the above equation, we have

[TABLE]

∎

Appendix B Proofs of PA-GD

As stated in the main body of the paper, we can use Lemma 2 and Lemma 3 to prove Theorem 1. Lemma 2 is basically well-known. The main task focuses on proving Lemma 3, which consists of a sequence of lemmas (Lemma 8–Lemma 10) that lead to Lemma 3.

Before discussing the details of Lemma 3, we need to introduce some constants defined as follows,

[TABLE]

These quantities refer to different units of the algorithm. Specifically, $\mathcal{F}$ accounts for the objective value, $\mathcal{G}$ for the size of the gradient, $\mathcal{S}$ for the norm of the difference between iterates, and $\mathcal{T}$ for the number of iterations. Also, we define a condition number in terms of $\gamma$ as $\kappa\triangleq\frac{L_{\max}}{\gamma}\geq 1$.

These quantities, $\mathcal{F}$, $\mathcal{G}$, $\mathcal{S}$ and $\mathcal{T}$ have some certain relations as follows, which are useful of simplifying the expressions in the proofs.

[TABLE]

In the process of the proofs, we used conditions $\log(\frac{d\kappa}{\delta})\geq 1$, $\mathcal{P}_{1}\geq 2$ repeatedly to simply the expressions of the parameters. We also consider saddle point $\widetilde{\mathbf{x}}^{(t)}$ that satisfies the following condition.

Condition 1**.**

An $\epsilon$-second order stationary point $\widetilde{\mathbf{x}}^{(t)}$ satisfies the following conditions:

[TABLE]

where $g_{\textsf{th}}\triangleq\frac{\mathcal{G}}{2\kappa}$.

Condition 1 implies that point $\widetilde{\mathbf{x}}^{(t)}$ satisfies $\|\nabla f(\widetilde{\mathbf{x}}^{(t)})\|\leq\mathcal{G}/\kappa$ (see Lemma 7) and $\lambda_{\min}(\nabla^{2}f(\widetilde{\mathbf{x}}^{(t)}))\leq-\gamma$.

Sufficient Decrease after Perturbation

Consider $\widetilde{\mathbf{x}}^{(t)}$ satisfy Condition 1 and let $\boldsymbol{\mathcal{H}}\triangleq\nabla^{2}f(\widetilde{\mathbf{x}}^{(t)})$. We consider a second order approximation as the following

[TABLE]

With these definitions of parameters, we will study how PA-GD can escape from strict saddle points. The main part of the proof is to show that when two sequences are apart from each other with a certain distance along the $\vec{\mathbf{e}}$ direction at the starting points, where $\vec{\mathbf{e}}$ denotes the eigenvector of $\mathbf{M}^{-1}\mathbf{T}$ whose eigenvalue is maximum (greater than 1). Then, after a number of iterations at least one of them can give a sufficient decrease of the objective value. This property implies the iterates can easily escape from the saddle points as long as there is a large enough perturbation between the initial points of the two sequences along the $\vec{\mathbf{e}}$ direction. We will introduce the following two lemmas formally which are the main contributions of this work.

Lemma 8**.**

Under Assumption 1, consider $\widetilde{\mathbf{x}}^{(t)}$ that satisfies Condition 1 and a generic sequence $\mathbf{u}^{(t)}$ generated by AGD. For any constant $\widehat{c}\geq 2$, $\delta\in(0,\frac{d\kappa}{e}]$, when initial point $\mathbf{u}^{(0)}$ satisfies

[TABLE]

then, with the definition of

[TABLE]

there exits constants $c^{(1)}_{\max},\widehat{c}$ such that for any $\eta\leq c^{(1)}_{\max}/L_{\max}$, the iterates generated by PA-GD satisfy $\|\mathbf{u}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S},\forall t<T$.

Lemma 9**.**

Under Assumption 1, consider $\widetilde{\mathbf{x}}^{(t)}$ that satisfies Condition 1. There exist constants $c^{(2)}_{\max}$, $\widehat{c}$ such that: for any $\delta\in(0,\frac{d\kappa}{e}]$ and $\eta\leq c^{(2)}_{\max}/L_{\max}$, with the definition of

[TABLE]

where two iterates $\{\mathbf{u}^{(t)}\}$ and $\{\mathbf{w}^{(t)}\}$ that are generated by PA-GD with initial points $\{\mathbf{u}^{(0)},\mathbf{w}^{(0)}\}$ satisfying

[TABLE]

where $\vec{\mathbf{e}}$ denotes the eigenvector of $\mathbf{M}^{-1}\mathbf{T}$ whose eigenvalue is maximum, then, if $\|\mathbf{u}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S},\forall t<T$, we will have $T<\widehat{c}\mathcal{T}$.

Lemma 8 says that if the $\mathbf{u}^{(t)}$-iterate generated by PA-GD cannot provide a sufficient decrease of the objective value, then the iterates are constrained within the area which is very close to the saddle point. With this property, Lemma 9 shows if there exists another PA-GD iterate $\mathbf{w}^{(t)}$, which is initialized with a certain distance along the $\vec{\mathbf{e}}$ direction from the $\mathbf{u}$-iterate, then $\mathbf{w}^{(t)}$ will provide a sufficient decrease of the objective value. These two lemmas characterize the convergence behavior of the PA-GD iterates.

Escaping from Saddle Points

Then, we need to quantify the probability that after adding the perturbation the algorithm cannot escape from strict saddle points. In previous work about escaping from saddle points with GD, a characterization of the geometry around saddle points has been given [Jin et al., 2017a, Lemma 15]. Once we know that PA-GD also decreases the objective value sufficiently in Lemma 8 and Lemma 9, the following lemma can be claimed straightforwardly. To be more specific, we can obtain the probability that iterates will be stuck at the strict points after $T$ iterations as follows.

[TABLE]

where $\mathcal{X}_{\textrm{stuck}}$ denotes the set where the algorithm starts such that the sequence cannot escape from the strict saddle point after $T$ iterations, $(a)$ is true because probability $\mathbb{P}(\mathbf{w}^{(0)}\in\mathcal{X}_{\textrm{stuck}}|\mathbf{u}^{(0)}\in\mathcal{X}_{\textrm{stuck}})$ can be upper bounded by $\delta$, which is proven in the following lemma.

Lemma 10**.**

Under Assumption 1, there exists a universal constant $c_{\max}$, for any $\delta\in(0,d\kappa/e]$: consider a saddle point $\widetilde{\mathbf{x}}^{(t)}$ which satisfies Condition 1, let $\mathbf{x}^{(0)}=\widetilde{\mathbf{x}}^{(t)}+\xi$ where $\xi$ is generated randomly which follows the uniform distribution over a ball with radius $r$, and let $\mathbf{x}^{(t)}$ be the iterates of PA-GD starting from $\mathbf{x}^{(0)}$. Then, when step size $\eta\leq c_{\max}/L_{\max}$, with at least probability $1-\delta$, we have the following for any $T\geq\mathcal{T}/c_{\max}$

[TABLE]

Then, applying $\eta=\frac{c}{L_{\max}}$,$\gamma=(L_{\max}\rho\epsilon)^{1/3}$, and $\delta=\frac{dL_{\max}}{(L_{\max}\rho\epsilon)^{1/3}}e^{-\chi}$ into Lemma 10, we can get Lemma 3 immediately.

With these lemmas, we can give the proof of Theorem 1 as the following.

B.1 Proof of Theorem 1

Next, we prove the main theorem.

Proof.

Submitting $\eta=\frac{c}{L_{\max}}$,$\gamma=(L_{\max}\rho\epsilon)^{1/3}$, and $\delta=\frac{dL_{\max}}{(L_{\max}\rho\epsilon)^{1/3}}e^{-\chi}$ into the definitions of $\mathcal{F},\mathcal{G},\mathcal{T}$, we will have the following definitions.

[TABLE]

After applying Lemma 7, we know that

[TABLE]

where $c\leq 1,\chi,\mathcal{P}_{1},\mathcal{P}_{2}\geq 1$.

With a set of necessary lemmas and leveraging the proof of PGD [Jin et al., 2017a, Theorem 3], we have the following convergence analysis of PA-GD. Specifically, at any iteration, we need to consider two cases (we use the first iteration as an example):

In this case the gradient is large such that $\sum^{2}_{k=1}\|\nabla_{k}f(\mathbf{h}^{(0)}_{-k},\mathbf{x}^{(0)}_{k})\|^{2}>g^{2}_{\textsf{th}}$: According to Lemma 2, we have

[TABLE]

where in $(a)$ use the definition of $g^{2}_{\textsf{th}}$ and $\eta\leq c/L_{\max}$. 2. 2.

The gradient is small in all block directions, namely $\sum^{2}_{k=1}\|\nabla_{k}f(\mathbf{h}^{(0)}_{-k},\mathbf{x}^{(0)}_{k})\|^{2}\leq g^{2}_{\textsf{th}}$: in this case, we will add the perturbation to the iterates, and implement AGD for the next $t_{\textsf{th}}$ steps and then check the termination condition. If the termination condition is not satisfied, we must have

[TABLE]

which implies that the objective value in each step on average is decreased by

[TABLE]

Since $\kappa=L_{\max}/(L_{\max}\rho\epsilon)^{1/3}\geq 1$, we know that the right-hand side (RHS) of (80) is greater than RHS of (78).

With the results of these two cases, we can know that if there is a large size of the gradient, we can know the decrease of the objective function value by the result of case 1, and if not, we use the result of case 2. In summary, PA-GD can have a sufficient decrease of the objective function value by $\frac{c^{7}}{(\chi\mathcal{P}_{1})^{7}\mathcal{P}^{2}_{2}}\frac{\epsilon^{2}}{L_{\max}}\frac{(L_{\max}\rho\epsilon)^{1/3}}{L_{\max}}$ per iteration on average. This means that Algorithm 1 must stop within a finite number of iterations, which is

[TABLE]

where $\Delta f\triangleq f(\mathbf{h}^{(0)}_{-1},\mathbf{x}^{(0)}_{1})-f^{*}$.

According to Lemma 3, we know that with probability $1-\frac{dL_{\max}}{(L_{\max}\rho\epsilon)^{1/3}}e^{-\chi}$ the algorithm can give a sufficient descent with the perturbation when $\sum^{2}_{k=1}\|\nabla_{k}f(\mathbf{h}^{(t)}_{-k},\mathbf{x}^{(t)}_{k})\|^{2}\leq g^{2}_{\textsf{th}}$. Since the total number of perturbation we can add is at most

[TABLE]

Using the union bound, the probability of Lemma 3 being satisfied for all perturbations is

[TABLE]

With chosen $\chi=6\max\{\ln(\mathcal{C}/\delta),4\}$, we have $\chi^{6}e^{-\chi}\leq e^{-\chi/6}$, which implies $\chi^{6}e^{-\chi}\mathcal{C}\leq e^{-\chi/6}\mathcal{C}\leq\delta$.

The proof is complete. ∎

B.2 Proof of Lemma 1

Proof.

Recall the definitions:

[TABLE]

where $\widetilde{\mathbf{x}}^{(t)}$ is an $\epsilon$-second order stationary point, and

[TABLE]

Our goal of this lemma is to show that the maximum eigenvalue of $\mathbf{M}^{-1}\mathbf{T}$ is greater than 1 so that we can project iterates $\mathbf{v}^{(t)}$ onto the two subspaces, where the first subspace is spanned by the eigenvector of $\mathbf{M}^{-1}\mathbf{T}$ whose eigenvalue is the largest (greater than 1) and the other one is spanned by the remaining eigenvectors.

Note that $\det(\mathbf{M})=1$, which implies that $\det(\mathbf{M}^{-1}\mathbf{T}-\lambda\mathbf{I})=\det(\mathbf{T}-\lambda\mathbf{M})$, where $\lambda$ denotes the eigenvalue. We can analyze the determinant of $\mathbf{T}-\lambda\mathbf{M}$, i.e.,

[TABLE]

Then, we use two steps to show $\lambda_{\max}(\mathbf{M}^{-1}\mathbf{T})>1$: 1) we can show that all eigenvalues of $\mathbf{Q}(\lambda)$ are real; 2) there exists a $\lambda>1$ such that $\det(\mathbf{Q}(\lambda))=0$.

Consider a $\delta>0$. We have

[TABLE]

where

[TABLE]

meaning that $\mathbf{F}(\delta)$ is similar to $\mathbf{G}(\delta)$. Consequently, we can conclude that $\mathbf{F}(\delta)$ has the same eigenvalues of $\mathbf{G}(\delta)$. Since we know that $\boldsymbol{\mathcal{H}}$ and $\mathbf{G}(\delta)$ are diagonalizable (normal matrices), then we have the following result [Weyl, 1912] (or [Holbrook, 1992]) of quantifying the difference of the eigenvalues of the two normal matrices

[TABLE]

where $\lambda_{i}(\boldsymbol{\mathcal{H}})$ and $\lambda_{i}(\mathbf{G}(\delta))$ denote the $i$th eigenvalue of $\boldsymbol{\mathcal{H}}$ and $\mathbf{G}(\delta)$, which are listed in a decreasing order.

With the help of (95), we can check

[TABLE]

where $(a)$ is true since we used $\eta\leq c_{\max}/L_{\max}$ and the fact that $\|\boldsymbol{\mathcal{H}}\|\leq L$ and $\|\boldsymbol{\mathcal{H}}_{d}\|\leq L_{\max}$. Also, it can be observed that when $\delta=0$, matrix $\mathbf{G}(\delta)$ is reduced to $\eta\boldsymbol{\mathcal{H}}$. Note that if $\eta=1/L$ is used, then we have $\|\eta\boldsymbol{\mathcal{H}}-\mathbf{G}(\delta)\|\leq\delta+2(\sqrt{1+\delta}-1)$.

We know that the minimum eigenvalue of $\eta\boldsymbol{\mathcal{H}}$ which is $-\eta\gamma$ and the maximum difference of the eigenvalues between $\eta\boldsymbol{\mathcal{H}}$ and $\mathbf{G}(\delta)$ is upper bounded by (100). Then, we can choose a sufficient small $\delta$ such that $\mathbf{G}(\delta)$ also has a negative eigenvalue, meaning that we need to find a $\delta$ such that

[TABLE]

In other words, if we choose

[TABLE]

then we can conclude that $\mathbf{G}(\delta^{*})$ has a negative eigenvalue which is less than $-\eta\gamma+\delta^{*}=-\frac{\eta\gamma}{1+\frac{L_{\max}}{L}}$.

In the following, we will check that $\delta^{*}$ is a valid choice, meaning that equation (101) holds when $\delta^{*}=\frac{\eta\gamma}{1+\frac{L}{L_{\max}}}$.

First step

: since $L/L_{\max}\geq 1$, we have $\eta\gamma/(1+L/L_{\max})\leq\eta\gamma/2$.

Second step

: we only need to check

[TABLE]

meaning that it is sufficient to check

[TABLE]

It can be easily check that the left-hand side (LHS) of (102) with chosen $\delta^{*}$ is

[TABLE]

which is RHS of (102).

Therefore, we can conclude that $\mathbf{Q}(1+\delta^{*})$ has a negative eigenvalue.

When $\delta$ is large, it is easy to check $\mathbf{Q}(1+\delta)$ has a positive eigenvalue, since term $\delta^{2}\mathbf{I}$ dominates the spectrum of matrix $\mathbf{Q}(1+\delta)$ in (87). Since the eigenvalue is continuous with respect to $\delta$, we can conclude there exists a largest $\delta$, i.e., $\widehat{\delta}$, such that $\mathbf{Q}(1+\widehat{\delta})$ has a zero eigenvalue, i.e., $\det(\mathbf{Q}(1+\widehat{\delta}))=0$ where $1+\widehat{\delta}$ is at least

[TABLE]

Therefore, we can conclude that there exits a largest real eigenvalue of $\mathbf{M}^{-1}\mathbf{T}$ which is $1+\widehat{\delta}>1+\delta^{*}>1$. ∎

B.3 Proof of Lemma 2

Proof.

Under Assumption 1, we have (descent lemma)

[TABLE]

where (a) is true because of the update rule of gradient descent in each block and Assumption 1, in (b) we used $\eta\leq 1/L_{\max}$. ∎

B.4 Proof of Lemma 8

Proof.

Without loss of generality, let $\mathbf{u}^{(0)}$ be the origin, i.e., $\mathbf{u}^{(0)}=0$. According to the AGD update rules, we have

[TABLE]

Then, we use the mathematical induction to prove that

[TABLE]

When $t=0$, we have $\mathbf{u}^{(0)}=0$, so (108) is true.

Suppose (108) is true for the case where $\tau\leq t$. We will show that (108) is also true for the case where $\tau=t+1$.

First, we need to show the upper bound of $\|\mathbf{u}^{(t+1)}-\mathbf{u}^{(t)}\|$. According to the Taylor expansion and $\rho$-Hessian Lipschitz continuity, we have

[TABLE]

Comparing with the definition of $\widehat{f}_{\mathbf{u}^{(0)}}(\mathbf{u}^{(t)})$, we have

[TABLE]

where in $(a)$ we also used $\rho$-Hessian Lipschitz continuity.

According to the definition of $T$, we know that $f(\mathbf{u}^{(0)})-\widehat{f}_{\mathbf{u}^{(0)}}(\mathbf{u}^{(t)})\leq 3\mathcal{F}$ for all $t<T$, which implies that

[TABLE]

where in (111) we used $c_{\max}=\mathcal{P}_{2}/(5\widehat{c})^{3}$ and $\eta\leq c_{\max}/L_{\max}$.

From (104), we also know that

[TABLE]

For simplification of expression, we define

[TABLE]

Summing up (112) for $\tau=0,\ldots,t$, we have

[TABLE]

Combining (111) and (114), we know that

[TABLE]

which implies

[TABLE]

According to (107), we know

[TABLE]

where in $(a)$ we used Lipschitz continuity, i.e., $\sum^{2}_{k=1}\|\nabla_{k}f(\mathbf{z}^{(t)}_{-k},\mathbf{u}^{(t)}_{k})-\nabla_{k}f(\mathbf{z}^{(t-1)}_{-k},\mathbf{u}^{(t)}_{k})\|^{2}\leq L^{2}_{\max}\|\mathbf{u}^{(t+1)}_{1}-\mathbf{u}^{(t)}_{1}\|^{2}+L^{2}_{\max}\|\mathbf{u}^{(t)}_{2}-\mathbf{u}^{(t-1)}_{2}\|^{2}$, and $\sum^{2}_{k=1}\|\nabla_{k}f(\mathbf{z}^{(t-1)}_{-k},\mathbf{u}^{(t)}_{k})-\nabla_{k}f(\mathbf{z}^{(t-1)}_{-k},\mathbf{u}^{(t-1)}_{k})\|^{2}\leq L^{2}_{\max}\|\mathbf{u}^{(t+1)}_{1}-\mathbf{u}^{(t)}_{1}\|$.

Then, we have

[TABLE]

where $(a)$ is true because we have $\|\mathbf{u}^{(1)}-\mathbf{u}^{(0)}\|^{2}\leq 16\eta\mathcal{F}$ since $t<T$ and (116), and we used $\eta\leq c^{\prime}_{\max}/L_{\max}$ where $c^{\prime}_{\max}=1/10$ such that $\omega\approx 0.0435<1$.

Then, we can obtain

[TABLE]

Based on (117), we can get the upper bound of the sum of $\|\mathbf{u}^{(t+1)}-\mathbf{u}^{(t)}\|,\forall t<T$ as the following,

[TABLE]

which implies

[TABLE]

where in $(a)$ we used the triangle inequality and $\mathbf{u}^{(0)}=0$.

Due to the following fact

[TABLE]

we have $\|\mathbf{u}^{(t+1)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S}$ since $\widehat{c}\geq 2$. Therefore, we know that there exits $c^{(1)}_{\max}=\min\{c_{\max},c^{\prime}_{\max}\}$ such that $\|\mathbf{u}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S},\forall t<T$ when $\eta\leq c^{(1)}_{\max}/L_{\max}$, which completes the proof. ∎

B.5 Proof of Lemma 9

Proof.

Let $\mathbf{u}^{(0)}=0$ and define $\mathbf{v}^{(t)}\triangleq\mathbf{w}^{(t)}-\mathbf{u}^{(t)}$. According to the assumption of Lemma 9, we know that $\mathbf{v}^{(0)}=\upsilon[\eta L_{\max}\mathcal{S}/(\kappa\log(\frac{d\kappa}{\delta})\mathcal{P}_{1})]\vec{\mathbf{e}}$ when $\upsilon\in[\delta/(2\sqrt{d}),1]$. First, we define an auxiliary function

[TABLE]

then have

[TABLE]

Then, we consider sequence $\mathbf{w}^{(t)}$, i.e.,

[TABLE]

where in $(a)$ we used the following definitions:

[TABLE]

and

[TABLE]

Obviously, $\boldsymbol{\mathcal{H}}=\boldsymbol{\mathcal{H}}_{l}+\boldsymbol{\mathcal{H}}_{u}$.

Dynamics of $\mathbf{v}^{(t)}$:

Since the first two terms at RHS of (146) combined with $\mathbf{u}^{(t)}$ at LHS of (146) are exactly the same as (107). It can be observed that equation (146) gives the dynamic of $\mathbf{v}^{(t)}$, i.e.,

[TABLE]

Then, we can rewrite (150) in a matrix form as the following.

[TABLE]

It is worth noting that matrix $\mathbf{M}$ is a lower triangular matrix where the diagonal entries are all 1s, so it is invertible.

Taking the inverse of $\mathbf{M}$ on both sides of (151), we can obtain

[TABLE]

Let $\mathbb{P}_{\texttt{left}}$ denote the projection operator that projects the vector onto the space spanned by the eigenvector of $\mathbf{M}^{-1}\mathbf{T}$ whose eigenvalue is maximum. Taking the projection on both sides of (152), we have

[TABLE]

From Lemma 1, we know that the maximum eigenvalue of $\mathbf{M}^{-1}\mathbf{T}$ is greater than 1.

Relationship of the Norm of $\mathbf{v}^{(t)}$ Projected in the Two Subspaces:

Let $\phi^{(t)}$ denote the norm of $\mathbf{v}^{(t)}$ projected onto the space spanned by the eigenvector of $\mathbf{M}^{-1}\mathbf{T}$ whose maximum eigenvalue is $1+\widehat{\delta}$ where $\widehat{\delta}\geq\eta\gamma/(1+L/L_{\max})$ due to Lemma 1, and $\theta^{(t)}$ denote the norm of $\mathbf{v}^{(t)}$ projected onto the remaining space. From (153), we can have

[TABLE]

where $(a)$ is true because we applied the triangle inequality since $\eta$ is sufficiently small. Also, since $\mathbf{M}^{-1}=\mathbf{I}-\eta\boldsymbol{\mathcal{H}}_{l}$, we have

[TABLE]

where in $(a)$ $\odot$ denotes the Hadamard product and

[TABLE]

and inequality $(b)$ comes from the result on the spectral norm of the triangular truncation operator (please see [Theorem 1]Angelos et al. [1992]). In particular, by defining

[TABLE]

we have

[TABLE]

$(c)$ is true for $d\geq 3$, in $(d)$ we used the fact that $\|\boldsymbol{\mathcal{H}}\|\leq L$ and $\|\boldsymbol{\mathcal{H}}_{d}\|\leq L_{\max}$.

Since $\|\mathbf{w}^{(0)}-\widetilde{\mathbf{x}}^{(t)}\|\leq\|\mathbf{u}^{(0)}-\widetilde{\mathbf{x}}^{(t)}\|+\|\mathbf{v}^{(0)}\|\leq 2r$, we can apply Lemma 8. Then, we know $\|\mathbf{w}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S},\forall t<T$. According to the assumptions of Lemma 9, we have $\|\mathbf{u}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S}$, and

[TABLE]

From (117), we know that

[TABLE]

since $\mathcal{P}_{1}\geq 2$ and we choose $\eta\leq c_{\max}/L_{\max}$ and $c_{\max}=1/10$. Similarly, we also have $\|\mathbf{u}^{(t+1)}-\mathbf{u}^{(t)}\|\leq\mathcal{S}$.

According to Lipsichiz continuity, we have the following bounds of $\|\mathbf{v}^{(t+1)}\|$, $\|\widetilde{\Delta}^{(t)}_{u}\|$ and $\|\widetilde{\Delta}^{(t)}_{l}\|$.

Relation between $\|\mathbf{v}^{(t)}\|$ and $\|\mathbf{v}^{(t+1)}\|$: We also know that

[TABLE]

where $(a)$ is true due to Lipschitz continuity.

We can express (174) as

[TABLE]

which implies

[TABLE]

where we choose $\eta\leq c_{\max}/L_{\max}$ and $c_{\max}=1/10$. 2. 2.

Bounds of $\|\widetilde{\Delta}^{(t)}_{u}\|$ and $\|\widetilde{\Delta}^{(t)}_{l}\|$:

According to $\rho$-Hessian Lipschitz continuity and Lemma 6, we have the size of $\widetilde{\Delta}^{(t)}_{u}$ as the following.

[TABLE]

where $(a)$ is true because

[TABLE]

Applying Lemma 6, we can also get the upper bound of $\|\widetilde{\Delta}^{(t)}_{l}\|$, i.e.,

[TABLE]

With the upper bounds of $\|\mathbf{v}^{(t+1)}\|$, $\|\widetilde{\Delta}^{(t)}_{u}\|$, $\|\widetilde{\Delta}^{(t)}_{l}\|$ and relation between $\|\mathbf{v}^{(t+1)}\|$ and $\|\mathbf{v}^{(t)}\|$, we can further simply (154) and (155) as follows,

[TABLE]

and further we have

[TABLE]

since $\|\mathbf{v}^{(t)}\|=\sqrt{(\phi^{(t)})^{2}+(\theta^{(t)})^{2}}$.

Consequently, we can arrive at

[TABLE]

where $\mu$ is the upper bound of $\eta(1.5\|\widetilde{\Delta}^{(t)}_{l}\|+\|\widetilde{\Delta}^{(t)}_{u}\|)\|\mathbf{M}^{-1}\|$ and can be obtained by

[TABLE]

Quantifying the Norm of $\mathbf{v}^{(t)}$ Projected at Different Subspaces:

Then, we will use mathematical induction to prove

[TABLE]

It is true when $t=0$ since $\|\theta^{(0)}\|\mathop{=}\limits^{\eqref{eq.inicond}}0$.

Assuming that equation $\eqref{eq.phinorm}$ is true at the $t$th iteration, we need to prove

[TABLE]

Applying (186) into RHS of (190), we have

[TABLE]

and substituting (187) into LHS of (190), we have

[TABLE]

Then, our goal is to prove RHS of (191) is greater than RHS of (192). After some manipulations, it is sufficient to show

[TABLE]

In the following, we will show that the above relation is true.

First step

: We know that

[TABLE]

where $(a)$ is true because $\mathcal{P}_{1}\geq 2$ and we choose $c^{\prime}_{\max}=1/(2\widehat{c}(2.5+62\widehat{c}))$ and $\eta\leq c^{\prime}_{\max}/L_{\max}$.

Second step

: Also, we know that

[TABLE]

With the above two steps, we have $\theta^{(t+1)}\leq 4\mu(t+1)\phi^{(t+1)}$, which completes the induction.

Recursion of $\phi^{(t)}$

:Using (189), we have $\theta^{(t)}\mathop{\leq}\limits^{\eqref{eq.phinorm}}4\mu t\phi^{(t)}\mathop{\leq}\limits^{\eqref{eq.bdofmu}}\phi^{(t)}$, which implies

[TABLE]

where in $(a)$ we used Lemma 1, and $(b)$ is true because

[TABLE]

where in $(a)$ we choose $c^{\prime\prime}_{\max}=1/(2\sqrt{2}(2.5+62\widehat{c}))$ and $\eta\leq c^{\prime\prime}_{\max}/L_{\max}$.

Quantifying Escaping Time:

From (158), we have

[TABLE]

where in $(a)$ we use condition $\upsilon\in[\delta/(2\sqrt{d}),1]$, in $(b)$ we used $\eta=c/L_{\max}$.

Since (197) is true for all $t<T$, we can have

[TABLE]

where $(a)$ comes from inequality $\log(1+x)>x/2$ when $x<1$, in $(b)$ we used relation $\log(x)<x,x>0$, and $(c)$ is true because $\delta\in(0,\frac{d\kappa}{e}]$ and $\log(d\kappa/\delta)>1$ and $\mathcal{P}_{1}>1$ we have

[TABLE]

From (198), we know that

[TABLE]

where $(a)$ is true due to the fact that $\eta L_{\max}\geq 1$, $\log(d\kappa/\delta)>1$ and $\mathcal{P}_{1}>1$ so we know $\mathcal{T}\geq 1$.

When

[TABLE]

we will have $T<\widehat{c}\mathcal{T}$ where $c^{(2)}_{\max}\triangleq\min\{c_{\max},c^{\prime}_{\max},c^{\prime\prime}_{\max}\}$.

Since $\widehat{c}\geq 2$, we have $c_{\max}=\min\{c^{(1)}_{\max},c^{(2)}_{\max}\}\leq 1/(5\widehat{c})^{3}$. Also, we know that $c\leq c_{\max}$. Combining with (200), we need

[TABLE]

meaning that

[TABLE]

It can be observed that LHS of (202) is a polynomial with respect to $\widehat{c}$ and RHS of (202) is a exponential function in terms of $\widehat{c}$, implying there exists a universal $\widehat{c}$ such that (202) holds. The proof is complete. ∎

B.6 Proof of Lemma 10

Proof.

The proof of Lemma 10 is similar as the one of proving convergence of PGD shown in [Jin et al., 2017a, Lemma 14,15]. Considering the completeness of the whole proof in this paper, here we give the following proof of this lemma in details.

First, after the random perturbation, the objective function value in the worst case is increased at most by

[TABLE]

where $\mathbf{u}^{(0)}$ is a vector that follows uniform distribution within the ball $\mathbb{B}^{(d)}_{\widetilde{\mathbf{x}}^{(t)}}(r)$, $\mathbb{B}^{(d)}_{\widetilde{\mathbf{x}}^{(t)}}$ denotes the $d$-dimensional ball centered at $\widetilde{\mathbf{x}}^{(t)}$ with radius $r$, $\xi_{k}$ represents the $k$th block of the vector which is the difference between random generated vector $\mathbf{u}^{(0)}$ and $\widetilde{\mathbf{x}}^{(t)}$, and $(a)$ is true because $\xi\triangleq[\xi_{1},\ldots,\xi_{K}]$, $\|\xi_{k}\|\leq\|\xi\|,\forall k$, and in $(b)$ we used $\kappa>1$, $\log(d\kappa/\delta)>1$ and Condition 1.

Second, under Assumption 1, let $\widetilde{\mathbf{x}}^{(t)}$ satisfy conditions Condition 1, and two PA-GD iterates $\{\mathbf{u}^{(t)}\}$ $\{\mathbf{w}^{(t)}\}$ satisfy the conditions as in Lemma 9. Selecting $c_{\max}=\min\{c^{(1)}_{\max},c^{(2)}_{\max}\}$, so we have that $\eta\leq c_{\max}/L_{\max}$ is small enough such that Lemma 8 and Lemma 9 can both hold.

Let $T^{*}\triangleq\widehat{c}\mathcal{T}$ and $T^{\prime}\triangleq\inf_{t}\{t|\widehat{f}_{\mathbf{u}^{(0)}}(\mathbf{u}^{(t)})-f(\mathbf{u}^{(0)})\leq-3\mathcal{F}\}$. Then, we have the following two cases to analyze the decrease of the objective value after $T$ iterations with the random perturbation.

Case $T^{\prime}\leq T^{*}$:

[TABLE]

Based on Lemma 2, we know that AGD is always decreasing the objective function. For any $T\geq\mathcal{T}/c_{\max}\geq\widehat{c}\mathcal{T}=T^{*}\geq T^{\prime}$, we have

[TABLE]

where $c_{\max}=\min\{1,1/\widehat{c}\}$. 2. 2.

Case $T^{\prime}>T^{*}$: Applying Lemma 8, we know that $\|\mathbf{u}^{(t)}-\mathbf{u}^{(0)}\|\leq 5\widehat{c}\mathcal{S}$ for $t\leq T^{*}$. Define $T^{\prime\prime}=\inf_{t}\{t|\widehat{f}_{\mathbf{w}^{(0)}}(\mathbf{w}^{(t)})-f(\mathbf{w}^{(0)})\leq-3\mathcal{F}\}$. Then, after applying Lemma 9, we know $T^{\prime\prime}\leq T^{*}$. Similar as (204), for $T\geq 1/c_{\max}\mathcal{T}$, we also have $f(\mathbf{w}^{(T)})-f(\mathbf{w}^{(0)})\leq f(\mathbf{w}^{T^{*}})-f(\mathbf{w}^{(0)})\leq f(\mathbf{w}^{T^{\prime\prime}})-f(\mathbf{w}^{(0)})\leq-2.5\mathcal{F}$.

Combining the above two cases, we have

[TABLE]

meaning that at least one of the sequences can give a sufficient decrease of the objective function if the initial points of the two sequences are separated apart with each other far enough along direction $\vec{\mathbf{e}}$.

Therefore, we can conclude that if $\mathbf{u}^{(0)}\in\mathcal{X}_{\textsf{stuck}}$, then $(\mathbf{u}^{(0)}\pm\upsilon r\vec{\mathbf{e}})\notin\mathcal{X}_{\textsf{stuck}}$ where $\upsilon\in[\frac{\delta}{2\sqrt{d}},1]$.

Finally, we give the upper bound of the volume of $\mathcal{X}_{\textsf{stuck}}$,

[TABLE]

where $I_{\textsf{stuck}}(\mathbf{u})$ is an indicator function showing that $\mathbf{u}$ belongs to set $\mathcal{X}_{\textsf{stuck}}$, and $u_{1}$ represents the component of vector $\mathbf{u}$ along $\vec{\mathbf{e}}$ direction, and $\mathbf{u}_{-1}$ is the remaining $d-1$ dimensional vector.

Then, the ratio of $\textsf{Vol}(\mathcal{X}_{\textsf{stuck}})$ over the whole volume of the perturbation ball can be upper bounded by

[TABLE]

where $\Gamma(\cdot)$ denotes the Gamma function, and inequality is true due to the fact that $\Gamma(x+1)/\Gamma(x+1/2)<\sqrt{x+1/2}$ when $x\geq 0$.

Combining (203) and (205), we can show that

[TABLE]

with at least probability $1-\delta$. ∎

Appendix C Proof of PA-PP

First, we need to introduce some constants defined as follows,

[TABLE]

where $\eta=1/\nu$. In order to keep the completeness of the proof, the certain relations of these quantities are listed as follows, which are useful of simplifying the expressions in the proofs.

[TABLE]

We also consider saddle point $\widetilde{\mathbf{x}}^{(t)}$ that satisfies the following condition.

Condition 2**.**

An $\epsilon$-second order stationary point $\widetilde{\mathbf{x}}^{(t)}$ satisfies the following conditions:

[TABLE]

where $g_{\textsf{th}}=\frac{\mathcal{G}}{2\kappa}$.

Then, we have the following preliminary lemmas.

Lemma 11**.**

If function $f(\cdot)$ is $L$-smooth with Lipschitz constant, the we have

[TABLE]

where sequence $\mathbf{x}^{(t)}_{k},k=1,2$ is generated by the APP algorithm.

Lemma 12**.**

Under Assumption 1, we have block-wise Lipschitz continuity as the follows:

[TABLE]

and

[TABLE]

Second, we can have the descent lemma as the following

Lemma 13**.**

Under Assumption 1, for the APP algorithm with penalizer $\nu\geq 3L_{\max}$, we have

[TABLE]

Third, we need to characterize the convergence behaviour of PA-PP when $\|\mathbf{x}^{(t+1)}-\mathbf{x}^{(t)}\|$ is small. In this case, we need three steps to arrive the final results.

Step 1

: Quantify upper bound of the distance between generic iterate $\mathbf{u}^{(t)}$ and saddle point $\widetilde{\mathbf{x}}^{(t)}$.

Lemma 14**.**

Under Assumption 1, consider saddle point $\widetilde{\mathbf{x}}^{(t)}$ that satisfies Condition 2. For any constant $\widehat{c}\geq 2$, $\delta\in(0,\frac{d\kappa}{e}]$, when initial point $\mathbf{u}^{(0)}$ satisfies

[TABLE]

then, with the definition of

[TABLE]

there exits constants $c^{(1)}_{\max},\widehat{c}$ such that for any $\nu\geq L_{\max}/c^{(1)}_{\max}$, the iterates generated by PA-PP satisfy $\|\mathbf{u}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S},\forall t<T$.

Step 2

: Quantify the escaping time of iterates near a strict saddle point.

Lemma 15**.**

Under Assumption 1, consider saddle point $\widetilde{\mathbf{x}}^{(t)}$ that satisfies satisfies Condition 2. There exist constants $c^{(2)}_{\max}$, $\widehat{c}$ such that: for any $\delta\in(0,\frac{d\kappa}{e}]$ and $\nu\geq L_{\max}/c^{(2)}_{\max}$, with the definition of

[TABLE]

where two iterates $\{\mathbf{u}^{(t)}\}$ and $\{\mathbf{w}^{(t)}\}$ that are generated by PA-PP with initial points $\{\mathbf{u}^{(0)},\mathbf{w}^{(0)}\}$ satisfying

[TABLE]

where $\vec{\mathbf{e}}^{\prime}$ denotes the eigenvector of $\mathbf{T}^{\prime-1}\mathbf{M}^{\prime}$ whose corresponding positive eigenvalue is minimum, if $\|\mathbf{u}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S},\forall t<T$, we will have $T<\widehat{c}\mathcal{T}$.

Step 3

: Quantify sufficient decrease with random perturbation. With Lemma 14 and Lemma 15, we can apply Lemma 10 directly and obtain the following lemma.

Lemma 16**.**

Under Assumption 1, there exists a universal constant $c_{\max}$, for any $\delta\in(0,d\kappa/e]$: consider a saddle point $\widetilde{\mathbf{x}}^{(t)}$ which satisfies (3), let $\mathbf{x}^{(0)}=\widetilde{\mathbf{x}}^{(t)}+\xi$ where $\xi$ is generated randomly which follows the uniform distribution over a ball with radius $r$, and let $\mathbf{x}^{(t)}$ be the iterates of PA-PP starting from $\mathbf{x}^{(0)}$. Then, when step size $\nu\geq L_{\max}/c_{\max}$, with at least probability $1-\delta$, we have the following for any $T\geq\mathcal{T}/c_{\max}$

[TABLE]

Substituting $\nu=\frac{L_{\max}}{c}$,$\gamma=(L_{\max}\rho\epsilon)^{1/3}$, and $\delta=\frac{dL_{\max}}{(L_{\max}\rho\epsilon)^{1/3}}e^{-\chi}$ in to Lemma 16, we can obtain the following lemma immediately.

Lemma 17**.**

Under Assumption 1, there exists a absolute constant $c_{\max}$. Let $c\leq c_{\max}$, $\chi\geq 1$, and $\eta$, $r$, $g_{\textsf{th}}$, $t_{\textsf{th}}$ calculated as Algorithm 2 describes. Let $\widetilde{\mathbf{x}}^{(t)}$ be a strict saddle point, which satisfies

[TABLE]

and

[TABLE]

Let $\mathbf{x}^{(t)}=\widetilde{\mathbf{x}}^{(t)}+\xi^{(t)}$ where $\xi^{(t)}$ is generated randomly which follows the uniform distribution over $\mathbb{B}_{\widetilde{\mathbf{x}}^{(t)}}(r)$, and let $\mathbf{x}^{(t+t_{\textsf{th}})}$ be the iterates of PA-PP. With at least probability $1-\frac{dL_{\max}}{(L_{\max}\rho\epsilon)^{1/3}}e^{-\chi}$, we have

[TABLE]

Finally, we can get the convergence rate of PA-PP as the following.

C.1 Proof of Corollary 1

Next, we prove the main theorem.

Proof.

Submitting $\nu=\frac{L_{\max}}{c}$,$\gamma=(L_{\max}\rho\epsilon)^{1/3}$, and $\delta=\frac{dL_{\max}}{(L_{\max}\rho\epsilon)^{1/3}}e^{-\chi}$ into the definition of $\mathcal{F},\mathcal{G},\mathcal{T}$, we will have the following definitions.

[TABLE]

After applying Lemma 7, we know that

[TABLE]

where $c\leq 1,\chi,\mathcal{P}\geq 1$.

Similarly, at any iteration, we need to consider two cases (we use the first iteration as an example):

In this case the gradient is large such that $\|\mathbf{x}^{(1)}-\mathbf{x}^{(0)}\|>g_{\textsf{th}}/\nu$: According to Lemma 13, we have

[TABLE]

where in $(a)$ use the definition of $g^{2}_{\textsf{th}}$ and $\nu\geq L_{\max}/c$. 2. 2.

The gradient is small in all block directions, namely $\|\mathbf{x}^{(t+1)}-\mathbf{x}^{(t)}\|^{2}\leq g_{\textsf{th}}/\nu$: in this case, we will add the perturbation to the iterates, and implement APP for the next $t_{\textsf{th}}$ steps and then check the termination condition. If the termination condition is not satisfied, we must have

[TABLE]

which implies that the objective value in each step on average is decreased by

[TABLE]

Since $\kappa=L_{\max}/(L_{\max}\rho\epsilon)^{1/3}\geq 1$ and $c\leq 1/3$, we know that RHS of (223) is greater than RHS of (221).

With the results of these two cases, we can know that if there is a large size of the gradient, we can know the decrease of the objective function value by the result of case 1, and if not, we use the result of case 2. In summary, PA-PP can have a sufficient decrease of the objective function value by $\frac{c^{7}}{\chi^{7}\mathcal{P}^{2}}\frac{\epsilon^{2}}{L_{\max}}\frac{(L_{\max}\rho\epsilon)^{1/3}}{L_{\max}}$ per iteration on average. This means that Algorithm 1 must stop within a finite number of iterations, which is

[TABLE]

where $\Delta f\triangleq f(\mathbf{h}^{(0)}_{-1},\mathbf{x}^{(0)}_{1})-f^{*}$.

According to Lemma 3, we know that with probability $1-\frac{dL_{\max}}{(L_{\max}\rho\epsilon)^{1/3}}e^{-\chi}$ the algorithm can give a sufficient descent with the perturbation when $\|\mathbf{x}^{(t+1)}-\mathbf{x}^{(t)}\|^{2}\leq g_{\textsf{th}}/\nu$. Since the total number of perturbation we can add is at most

[TABLE]

Using the union bound, the probability of Lemma 3 being satisfied for all perturbations is

[TABLE]

With chosen $\chi=6\max\{\ln(\mathcal{C}^{\prime}/\delta),4\}$, we have $\chi^{6}e^{-\chi}\leq e^{-\chi/6}$, which implies $\chi^{6}e^{-\chi}\mathcal{C}^{\prime}\leq e^{-\chi/6}\mathcal{C}^{\prime}\leq\delta$.

The proof is complete. ∎

C.2 Proof of Corollary 2

Proof.

Recall the definitions:

[TABLE]

where $\widetilde{\mathbf{x}}^{(t)}$ is an $\epsilon$-second order stationary point, and

[TABLE]

Obviously, we also have $\boldsymbol{\mathcal{H}}=\boldsymbol{\mathcal{H}}^{\prime}_{l}+\boldsymbol{\mathcal{H}}^{\prime}_{u}$.

Note that $\det(\mathbf{T}^{\prime})=1$, which implies that $\det(\mathbf{T}^{\prime-1}\mathbf{M}^{\prime}-\lambda\mathbf{I})=\det(\mathbf{M}^{\prime}-\lambda\mathbf{T}^{\prime})$, where $\lambda$ denotes the eigenvalue. We can analyze the determinant of $\mathbf{M}^{\prime}-\lambda\mathbf{T}^{\prime}$. We have

[TABLE]

It can be observed that

[TABLE]

meaning that $\mathbf{Q}^{\prime}(\lambda)$ is similar to $\mathbf{G}^{\prime}(\lambda)$. Consequently, we can conclude that $\mathbf{Q}^{\prime}(\delta)$ has the same eigenvalues of $\mathbf{G}^{\prime}(\delta)$. Furthermore, since matrix $\mathbf{G}^{\prime}(\lambda)$ is symmetric, we know that all eigenvalues of $\mathbf{Q}^{\prime}(\lambda)$ and $\mathbf{G}^{\prime}(\lambda)$ are real. Then, we can need to show there exists $\lambda$ such that $\det(\mathbf{Q}^{\prime}(\lambda))=0$.

Consider $0\leq\delta\leq 1$. We have

[TABLE]

Since we know that $\boldsymbol{\mathcal{H}}$ and $\mathbf{G}(1-\delta)$ are diagonalizable (normal matrices), then we have the following result Weyl [1912] (or Holbrook [1992]) of quantifying the difference of the eigenvalues of the two matrices

[TABLE]

where $\lambda_{i}(\boldsymbol{\mathcal{H}})$ and $\lambda_{i}(\mathbf{G}^{\prime}(1-\delta))$ denote the $i$th eigenvalue of $\boldsymbol{\mathcal{H}}$ and $\mathbf{G}^{\prime}(1-\delta)$, which are listed in a decreasing order.

With the help of (230), we can check

[TABLE]

where $(a)$ is true since we used $\eta\leq c_{\max}/L_{\max}$. Also, it can be observed that when $\delta=0$, matrix $\mathbf{G}^{\prime}(\delta)$ is reduced to $\eta\boldsymbol{\mathcal{H}}$.

We know that the minimum eigenvalue of $\eta\boldsymbol{\mathcal{H}}$ which is $-\eta\gamma$ and the maximum difference of the eigenvalues between $\eta\boldsymbol{\mathcal{H}}$ and $\mathbf{G}^{\prime}(\delta)$ is upper bounded by (235). Then, we can choose a sufficient small $\delta$ such that $\mathbf{G}^{\prime}(\delta)$ also has a negative eigenvalue, meaning that we need to find a $\delta\in[0,1]$ such that

[TABLE]

In other words, if we choose

[TABLE]

then we can conclude that $\mathbf{G}^{\prime}(\delta^{*})$ has a negative eigenvalue which is less than $-\eta\gamma+\delta^{*}=-\frac{\eta\gamma}{2}$. In the following, we will check that $\delta^{*}$ is a valid choice, meaning that equation (236) holds when $\delta^{*}=\frac{\eta\gamma}{2}$.

Actually, equation (236) can be rewritten as

[TABLE]

Since $\kappa=L_{\max}/\gamma\geq 1$ and $\eta\leq c_{\max}/L_{\max}$ where $c_{\max}\leq 1/2$, we have

[TABLE]

which implies that equation (236) is true with chosen $\delta^{*}$ Therefore, we can conclude that $\mathbf{Q}^{\prime}(1+\delta^{*})$ has a negative eigenvalue.

When $\delta$ is large, i.e., $\delta>1$, we have

[TABLE]

where $j$ denotes the imaginary number, so $\mathbf{Q}^{\prime}(1-\delta)$ is similar to $\mathbf{G}^{\prime}(1-\delta)$ when $\delta>1$. Also, we know that $\mathbf{G}^{\prime}(1-\delta)$ is a Hermitian matrix. It is easy to check $\mathbf{Q}^{\prime}(1-\delta)$ has a positive eigenvalue, since term $\delta\mathbf{I}$ dominates the spectrum of matrix $\mathbf{Q}^{\prime}(1-\delta)$ in (239). Considering the eigenvalue is continuous with respect to $\delta$, we can conclude there exists a $\delta$, i.e., $\widehat{\delta}^{\prime}$, such that $\mathbf{Q}^{\prime}(1-\widehat{\delta}^{\prime})$ has a zero eigenvalue, i.e., $\det(\mathbf{Q}^{\prime}(1-\widehat{\delta}^{\prime}))=0$ where $1-\widehat{\delta}^{\prime}$ is at least as small as

[TABLE]

meaning that $1-\widehat{\delta}^{\prime}\leq 1-\frac{\eta\gamma}{2}$. ∎

In the following, we will give the proofs of Lemma 12–Lemma 16 in details.

Appendix D Proofs of Lemma 11–Lemma 16

D.1 Proof of Lemma 11

Proof.

First, we have

[TABLE]

where in $(a)$ we used block-wise Lipschitz continuity, in $(b)$ we choose $\eta\leq 1/(2L_{\max})$.

[TABLE]

where $(a)$ we also choose $\eta\leq 1/(2L_{\max})$.

Summing (241) and (242), we have

[TABLE]

where $\mathbf{h}^{(t)}_{-1}=\mathbf{x}^{(t)}_{2}$ and $\mathbf{h}^{(t)}_{-2}=\mathbf{x}^{(t+1)}_{1}$. ∎

D.2 Proof of Lemma 12

There proof involves two parts:

Upper Triangular Matrix:

Consider three different vectors $\mathbf{x}$, $\mathbf{y}$ and $\mathbf{z}$. We can have

[TABLE]

where in $(a)$ we use

[TABLE]

and $\|\mathbf{I}_{1}\|=\|\mathbf{I}_{2}\|=1$.

Lower Triangular Matrix:

[TABLE]

where $(a)$ is true because we know $\|\mathbf{I}_{1}\|=\|\mathbf{I}_{2}\|=1$.

D.3 Proof of Lemma 13

Proof.

Under Assumption 1, we have (descent lemma)

[TABLE]

where (a) is true because of the update rule of APP in each block and Assumption 1 and block-wise Lipschitz continuity, in (b) we choose $\eta\leq 1/(3L_{\max})$ and $\nu=1/\eta$. ∎

D.4 Proof of Lemma 14

Proof.

Without loss of generality, let $\mathbf{u}^{(0)}$ be the origin, i.e., $\mathbf{u}^{(0)}=0$. According to the APP update rule of variables, we have

[TABLE]

It can be observed that the update rule of PA-PP is very similar as the one of PA-GD. The proof of this lemma is also similar as Lemma 8. We only need to replace $\nabla_{1}f(\mathbf{u}^{(t)}_{1},\mathbf{u}^{(t)}_{2})$ as $\nabla_{1}f(\mathbf{u}^{(t+1)}_{1},\mathbf{u}^{(t)}_{2})$ and $\nabla_{2}f(\mathbf{u}^{(t+1)}_{1},\mathbf{u}^{(t)}_{2})$ as $\nabla_{2}f(\mathbf{u}^{(t+1)}_{1},\mathbf{u}^{(t+1)}_{2})$, which can give us the claimed result after following the proof of Lemma 8. Hence, we ignore the repeated part with the proof of Lemma 8 for simplicity of expressions. ∎

D.5 Proof of Lemma 15

Proof.

Let $\mathbf{u}^{(0)}=0$ and define $\mathbf{v}^{(t)}\triangleq\mathbf{w}^{(t)}-\mathbf{u}^{(t)}$. According to the assumption of Lemma 9, we know that $\mathbf{v}^{(0)}=\upsilon[\eta L_{\max}\mathcal{S}/(\kappa\log(\frac{d\kappa}{\delta})\mathcal{P}_{1})]\vec{\mathbf{e}}^{\prime}$ when $\upsilon\in[\delta/(2\sqrt{d}),1]$. First, we define the following auxiliary function

[TABLE]

then have

[TABLE]

Then, we consider sequence $\mathbf{w}^{(t)}$, i.e.,

[TABLE]

where in $(a)$ we used the following definitions

[TABLE]

and

[TABLE]

Obviously, $\boldsymbol{\mathcal{H}}=\boldsymbol{\mathcal{H}}^{\prime}_{l}+\boldsymbol{\mathcal{H}}^{\prime}_{u}$.

Dynamics of $\mathbf{v}^{(t)}$:

Since the first two terms at RHS of (305) combined with $\mathbf{u}^{(t)}$ at LHS of (305) are exactly the same as (279). It can be observed that equation (305) gives the dynamic of $\mathbf{v}^{(t)}$, i.e.,

[TABLE]

which can be equivalently expressed by

[TABLE]

It is worth noting that matrix $\mathbf{T}^{\prime}$ is an upper triangular matrix where the diagonal entries are all 1s, so it is invertible. Taking the inverse of $\mathbf{T}^{\prime}$ on both sides of (308), we can obtain

[TABLE]

Let $\mathbb{P}^{\prime}_{\texttt{left}}$ denote the projection operator that projects the vector onto the space spanned by the eigenvector of $\mathbf{T}^{\prime-1}\mathbf{M}$ whose corresponding positive eigenvalue is minimum. Taking the projection on both sides of (309), we have

[TABLE]

Relationship of the Norm of $\mathbf{v}^{(t)}$ Projected onto the Two Subspaces:

Let $\phi^{(t)}$ denote the norm of $\mathbf{v}^{(t)}$ projected onto the space spanned by the eigenvector of $\mathbf{T}^{\prime-1}\mathbf{M}^{\prime}$ whose positive minimum eigenvalue of $\mathbf{M}^{\prime-1}\mathbf{T}^{\prime}$ is $1-\widehat{\delta}^{\prime}>0$ and $\theta^{(t)}$ denote the norm of $\mathbf{v}^{(t)}$ projected onto the remaining space. From (310), we can have

[TABLE]

where $(a)$ is true because we applied the triangle inequality since $\eta$ is sufficiently small.

Since $\|\mathbf{w}^{(0)}-\widetilde{\mathbf{x}}^{(t)}\|\leq\|\mathbf{u}^{(0)}-\widetilde{\mathbf{x}}^{(t)}\|+\|\mathbf{v}^{(0)}\|\leq 2r$, we can apply Lemma 14. Then, we know $\|\mathbf{w}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S},\forall t<T$. According to the assumptions of Lemma 15, we have $\|\mathbf{u}^{(t)}-\widetilde{\mathbf{x}}^{(t)}\|\leq 5\widehat{c}\mathcal{S}$, and

[TABLE]

From (117), we know that

[TABLE]

where we choose $\eta\leq c_{\max}/L_{\max}$ and $c_{\max}=1/10$. Similarly, we also have $\|\mathbf{u}^{(t+1)}-\mathbf{u}^{(t)}\|\leq\mathcal{S}$.

Then, we need to quantify the upper bounds of $\|\mathbf{M}^{\prime-1}\|$, $\|\mathbf{v}^{(t+1)}\|$, $\|\widetilde{\Delta}^{\prime(t)}_{u}\|$ and $\|\widetilde{\Delta}^{\prime(t)}_{l}\|$.

Upper bound of $\|\mathbf{M}^{\prime-1}\|$: applying the steps of deriving (156), we can quantify the inverse of matrix $\mathbf{T}^{\prime}$ as follows

[TABLE] 2. 2.

Relation between $\|\mathbf{v}^{(t)}\|$ and $\|\mathbf{v}^{(t+1)}\|$: We also know that

[TABLE]

where $(a)$ is true due to Lipschitz continuity.

We can express (329) as

[TABLE]

which implies

[TABLE]

where we choose $\eta\leq c_{\max}/L_{\max}$ and $c_{\max}=1/10$. 3. 3.

Upper bound of $\|\widetilde{\Delta}^{\prime(t)}_{l}\|$: applying Lemma 12, we can also get the upper bound of $\|\widetilde{\Delta}^{\prime(t)}_{l}\|$, i.e.,

[TABLE] 4. 4.

Upper bound of $\|\widetilde{\Delta}^{\prime(t)}_{u}\|$: according to $\rho$-Hessian Lipschitz continuity and Lemma 12, we have the size of $\widetilde{\Delta}^{\prime(t)}_{u}$ as the following.

[TABLE]

With the bounds of $\|\mathbf{v}^{(t+1)}\|$, $\|\widetilde{\Delta}^{\prime(t)}_{u}\|$, $\|\widetilde{\Delta}^{\prime(t)}_{l}\|$ and relation between $\|\mathbf{v}^{(t+1)}\|$ and $\|\mathbf{v}^{(t)}\|$, we can further simply (311) and (312) as follows,

[TABLE]

since $\|\mathbf{v}^{(t)}\|=\sqrt{(\phi^{(t)})^{2}+(\theta^{(t)})^{2}}$.

Consequently, we can arrive at

[TABLE]

where $\mu$ is the upper bound of term $\eta(1.5\|\widetilde{\Delta}^{\prime(t)}_{l}\|+\|\widetilde{\Delta}^{\prime(t)}_{u}\|)\|\mathbf{T}^{\prime-1}\|$ and can be obtained by

[TABLE]

Quantifying the Norm of $\mathbf{v}^{(t)}$ Projected at Different Subspaces:

Then, we will use mathematical induction to prove

[TABLE]

It is true when $t=0$ since $\|\theta^{(0)}\|\mathop{=}\limits^{\eqref{eq.inicond2}}0$.

Assuming that equation $\eqref{eq.bphinorm}$ is true at the $t$th iteration, we need to prove

[TABLE]

Applying (337) into RHS of (341), we have

[TABLE]

and substituting (338) into LHS of (341), we have

[TABLE]

Then, our goal is to prove RHS of (342) is greater than RHS of (343). After some manipulations, it is sufficient to show

[TABLE]

In the following, we will show that the above relation is true.

First step

: We know that

[TABLE]

where $(a)$ is true because we choose $c^{\prime}_{\max}=1/(2\widehat{c}(4+62\widehat{c}))$ and $\eta\leq c^{\prime}_{\max}/L_{\max}$.

Second step

: Also, we know that

[TABLE]

With the above two steps, we have $\theta^{(t+1)}\leq 4\mu(t+1)\phi^{(t+1)}$, which completes the induction.

Recursion of $\phi^{(t)}$

:Using (340), we have $\theta^{(t)}\mathop{\leq}\limits^{\eqref{eq.bphinorm}}4\mu t\phi^{(t)}\mathop{\leq}\limits^{\eqref{eq.bbdofmu}}\phi^{(t)}$, and have

[TABLE]

which implies

[TABLE]

where $(a)$ is true because $1-\widehat{\delta}^{\prime}>0$, in $(b)$ we used Corollary 2, i.e., $0<1-\widehat{\delta}^{\prime}\leq 1-\frac{\eta\gamma}{2}$, and $(c)$ is true because $\theta^{(t)}\leq\phi^{(t)}$ and

[TABLE]

where in $(a)$ we choose $c^{\prime\prime}_{\max}=1/(4\sqrt{2}(4+62\widehat{c}))$ and $\eta\leq c^{\prime\prime}_{\max}/L_{\max}$.

Quantifying Escaping Time:

From (313), we have

[TABLE]

where in $(a)$ we use condition $\upsilon\in[\delta/(2\sqrt{d}),1]$, in $(b)$ we used $\eta=c/L_{\max}$.

Since (347) is true for all $t<T$, we can have

[TABLE]

where $(a)$ comes from inequality $\log(1+x)>x/2$ when $x<1$, in $(b)$ we used relation $\log(x)<x,x>0$, and $(c)$ is true because $\delta\in(0,\frac{d\kappa}{e}]$ and $\log(d\kappa/\delta)>1$.

From (348), we know that

[TABLE]

where $(a)$ is true due to the fact that $\eta L_{\max}\geq 1$ and $\log(d\kappa/\delta)>1$ so we know $\mathcal{T}\geq 1$.

Applying the proof from (200) to (202), we can also conclude that there exists a universal $\widehat{c}$ such that (349) holds. The proof is complete.

∎

D.6 Proof of Lemma 16

First, after the random perturbation, the objective function value in the worst case is increased at most by

[TABLE]

where $\mathbf{u}^{(0)}$ is a vector that follows uniform distribution within the ball $\mathbb{B}^{(d)}_{\widetilde{\mathbf{x}}^{(t)}}(r)$, $\mathbb{B}^{(d)}_{\widetilde{\mathbf{x}}^{(t)}}$ denotes the $d$-dimensional ball centered at $\widetilde{\mathbf{x}}^{(t)}$ with radius $r$, $\xi_{k}$ represents the $k$th block of the vector which is the difference between random generated vector $\mathbf{u}^{(0)}$ and saddle point $\widetilde{\mathbf{x}}^{(t)}$, and in $(a)$ we choose $\eta\leq 1/(4L_{\max})$ and $(b)$ is true because $\xi\triangleq[\xi_{1},\ldots,\xi_{K}]$, $\|\xi_{k}\|\leq\|\xi\|,\forall k$, and in $(c)$ we used $\kappa>1$, $\log(d\kappa/\delta)>1$, $\mathcal{P}\geq 2$ and Condition 2 where $g_{\textsf{th}}$ is defined in (219).

Then, the rest of proof of Lemma 16 is the same as the rest of Lemma 10, therefore ignored for simplicity.

Appendix E Numerical Results

E.1 Proof of Lemma 4

Proof.

Consider function

[TABLE]

where $\mathbf{x}\in\mathcal{S}$, $\mathcal{S}=\{\mathbf{x}|\|\mathbf{x}\|^{2}\leq\tau\}$ and $\tau\geq\lambda_{\max}(\mathbf{A})$.

To prove L-smooth Lipschitz continuity

:

[TABLE]

where $x_{i}$ denotes the $i$th entry of vector $\mathbf{x}$, and $(a)$ is true because

[TABLE]

To prove block-wise Lipschitz continuity

: Without loss of generality, consider first block $\mathbf{x}_{1}\in\mathcal{S}^{\prime}$ where $\mathcal{S}^{\prime}=\{\mathbf{x}_{1}|\|\mathbf{x}_{1}\|^{2}\leq\tau^{\prime},\mathbf{x}_{1}\in\mathbb{R}^{d^{\prime}\times 1}\}$ and $d^{\prime}$ denotes the dimension of $\mathbf{x}_{1}$. Consider $\tau^{\prime}\geq\lambda_{\max}(\mathbf{A}^{\prime})$ where $\mathbf{A}^{\prime}\in\mathbb{R}^{d^{\prime}\times d^{\prime}}$ is the leading principal minor of matrix $\mathbf{A}$ of order $d^{\prime}$. Obviously, we have $\tau^{\prime}\leq\tau$.

[TABLE]

where $(a)$ is true because we used \mathbf{I}^{\prime}_{1}\triangleq\left[\begin{array}[]{cc}\mathbf{I}_{d^{\prime}}&0\\ 0&0\end{array}\right] which selects the first $d^{\prime}$ rows of \mathbf{A}\left(\left[\begin{array}[]{c}\mathbf{x}_{1}\\ \mathbf{x}_{-1}\end{array}\right]-\left[\begin{array}[]{c}\mathbf{x}^{\prime}_{1}\\ \mathbf{x}_{-1}\end{array}\right]\right).

To prove Hessian Lipschitz continuity

:

[TABLE]

where $(a)$ is true because $x_{i}+y_{i}\leq\sqrt{(x_{i}+y_{i})^{2}}=\sqrt{x^{2}_{1}+2x_{i}y_{i}+y^{2}_{i}}\mathop{\leq}\limits^{\eqref{eq.scain}}2\sqrt{\tau},\forall i$.

∎

E.2 Additional Simulation

Random matrix $\mathbf{A}$

: we also test the algorithms with a randomly generated symmetric matrix $\mathbf{A}$ by the following steps: 1) randomly generate a diagonal matrix $\mathbf{D}$ whose entries follow i.i.d. Gaussian distribution with zero mean and variance two; 2) generate an orthogonal matrix $\mathbf{U}\in\mathbb{R}^{d\times d}$; 3) obtain matrix $\mathbf{A}=\mathbf{U}\mathbf{D}\mathbf{U}^{\mathsf{T}}$. We initialize the PA-GD/AGD algorithms around the saddle point which is at the origin. The results are shown in Figure 3 where $d=100$. It can be observed that PA-GD can still escape from the strict saddle point faster than ordinary AGD, illustrating the benefit of adding the random perturbation when the gradient size is small.