On the Complexity of an Augmented Lagrangian Method for Nonconvex Optimization

TL;DR

This paper analyzes the worst-case complexity of an inexact Augmented Lagrangian method for nonconvex constrained optimization, providing bounds on iterations and evaluations depending on penalty parameter behavior and problem constraints.

Contribution

It establishes new complexity bounds for the Augmented Lagrangian method in nonconvex settings, including cases with bounded and unbounded penalty parameters, and different types of constraints.

Findings

Bounded penalty parameters yield $ ext{O}(| ext{log}( ext{epsilon})|)$ iteration complexity.

Unbounded penalty parameters lead to $ ext{O}( ext{epsilon}^{-2/( ext{alpha}-1)})$ iteration complexity.

Specific evaluation complexity bounds are derived for linearly constrained problems with different inner solver orders.

Abstract

In this paper we study the worst-case complexity of an inexact Augmented Lagrangian method for nonconvex constrained problems. Assuming that the penalty parameters are bounded, we prove a complexity bound of $\mathcal{O}(|\log(\epsilon)|)$ outer iterations for the referred algorithm to generate an $\epsilon$-approximate KKT point, for $\epsilon\in (0,1)$. When the penalty parameters are unbounded, we prove an outer iteration complexity bound of $\mathcal{O}\left(\epsilon^{-2/(\alpha-1)}\right)$, where $\alpha>1$ controls the rate of increase of the penalty parameters. For linearly constrained problems, these bounds yield to evaluation complexity bounds of $\mathcal{O}(|\log(\epsilon)|^{2}\epsilon^{-2})$ and $\mathcal{O}\left(\epsilon^{-\left(\frac{2(2+\alpha)}{\alpha-1}+2\right)}\right)$, respectively, when appropriate first-order methods ($p=1$) are used to approximately solve the unconstrained subproblems at each iteration. In the case of problems having only linear equality constraints, the latter bounds are improved to $\mathcal{O}(|\log(\epsilon)|^{2}\epsilon^{-(p+1)/p})$ and $\mathcal{O}\left(\epsilon^{-\left(\frac{4}{\alpha-1}+\frac{p+1}{p}\right)}\right)$, respectively, when appropriate $p$-order methods ($p\geq 2$) are used as inner solvers.

Equations

$$\underset{x\in {\mathbb{R}}^n}{\min }$$

$$c_i(x)=0,i=1,\dots ,m_e,$$

$$c_i(x)\ge 0,i=m_e+1,\dots ,m,$$

$$\Vert \nabla f(x_{k_0})-\sum _{i=1}^m{\lambda }_i^{(k_0)}\nabla c_i(x_{k_0})\Vert \le ϵ,$$

$$\Vert c_E(x_{k_0})\Vert \le ϵ,\Vert c_I^{(-)}(x_{k_0})\Vert \le ϵ,$$

$${\lambda }_i^{(k_0)}\ge 0,\forall i\in \left\{m_e+1,\dots ,m\right\},$$

$${\lambda }_i^{(k_0)}=0\text{whenever}{c}_i(x_{k_0})>ϵ,\forall i\in \left\{m_e+1,\dots ,m\right\},$$

$$\underset{x\in {\mathbb{R}}^n}{\min }$$

$$a_i^{T}x-b_i=0,i=1,\dots ,m.$$

$${\mathcal{L}}_f(u)=\left\{x\in {\mathbb{R}}^n|f(x)\le u\right\}$$

$$L(x,\lambda )=f(x)-\sum _{i=1}^m{\lambda }_i{c}_i(x),$$

$$P(x,\lambda ,\sigma )$$

$$\underset{x\in {\mathbb{R}}^n}{\min }P(x,{\lambda }^{(k)},{\sigma }^{(k)}),$$

$$P(x_{k+1},{\lambda }^{(k)},{\sigma }^{(k)})\le \min \left\{P(x_k,{\lambda }^{(k)},{\sigma }^{(k)}),P(x_0,{\lambda }^{(k)},{\sigma }^{(k)})\right\},$$

$$\Vert {\nabla }_{x}P(x_{k+1},{\lambda }^{(k)},{\sigma }^{(k)}){\Vert }_{\infty }\le ϵ.$$

$${\theta }^{(k+1)}=\max \left\{{\Vert \frac{{\lambda }^{(k)}}{{\sigma }^{(k)}}\Vert }_{\infty },{\Vert c_E(x_{k+1})-\frac{{\lambda }_E^{(k)}}{{\sigma }^{(k)}}\Vert }_{\infty },\Vert c_I^{(-)}(x_{k+1}){\Vert }_{\infty }\right\},$$

$$\sigma^{(k+1)}=\left\{\begin{array}[]{ll}\sigma^{(k),}&\text{if}\quad\theta^{(k+1)}\leq\gamma\theta^{(k)},\\ \max\left\{(k+1)^{\alpha},\sigma^{(k)}\right\},&\text{otherwise.}\end{array}\right.$$

$${\lambda }_i^{(k+1)}={\lambda }_i^{(k)}-{\sigma }^{(k)}{c}_i(x_{k+1}),\text{for}i=1,\dots ,m_e,$$

$${\lambda }_i^{(k+1)}=\max \left\{{\lambda }_i^{(k)}-{\sigma }^{(k)}{c}_i(x_{k+1}),0\right\}\text{for}i=m_e+1,\dots ,m.$$

$$\Vert {\nabla }_{x}L(x_k,{\lambda }^{(k)}){\Vert }_{\infty }\le ϵ,$$

$$\Vert c_E(x_k){\Vert }_{\infty }\le ϵ,\Vert c_I^{(-)}(x_k){\Vert }_{\infty }\le ϵ,$$

$${\lambda }_i^{(k)}\ge 0,\forall i\in \left\{m_e+1,\dots ,m\right\},$$

$${\lambda }_i^{(k)}=0\text{whenever}{c}_i(x_k)>ϵ,\forall i\in \left\{m_e+1,\dots ,m\right\},$$

$$P(\bar{x},\lambda ,\sigma )\le f(\bar{x}).$$

$$c_i(\bar{x})-\frac{{\lambda }_i}{\sigma }\ge -\frac{{\lambda }_i}{\sigma }.$$

$$0\ge {\left(c_i(\bar{x})-\frac{{\lambda }_i}{\sigma }\right)}_{-}\ge -\frac{{\lambda }_i}{\sigma },$$

$${\left(c_i(\bar{x})-\frac{{\lambda }_i}{\sigma }\right)}_{-}^2\le {\left(\frac{{\lambda }_i}{\sigma }\right)}^2,i=m_e+1,\dots ,m.$$

$$P(x,\lambda ,\sigma )\ge f(x)-\frac{1}{2}\sum _{i=1}^m\frac{{\left[{\lambda }_i\right]}^2}{\sigma },\forall x\in {\mathbb{R}}^n.$$

$$P(x,\lambda ,\sigma )$$

$${\nabla }_{x}P(x^{+},\lambda ,\sigma )=0.$$

$${\mathcal{L}}_P(\bar{x})=\left\{z\in {\mathbb{R}}^n|P(z,\lambda ,\sigma )\le P(\bar{x},\lambda ,\sigma )\right\}$$

$$f(x)-\frac{1}{2}\sum _{i=1}^m\frac{{\left[{\lambda }_i\right]}^2}{\sigma }\le P(x,\lambda ,\sigma )\le P(\bar{x},\lambda ,\sigma )\le f(\bar{x})$$

Full text

1 Introduction

In this paper we consider the constrained optimization problem

[TABLE]

where $f,c_{i}:\mathbb{R}^{n}\to\mathbb{R}$ ($i=1,\ldots,m$) are continuously differentiable functions, possibly nonconvex. Augmented Lagrangian Methods are among the most efficient schemes for nonconvex constrained optimization problems (see [24, 29, 3, 4]). The theoretical analysis of these methods usually focus on global convergence properties. More specifically, in the case of (1.1)-(1.3), for any starting pair $(x_{0},\lambda^{(0)})\in\mathbb{R}^{n}\times\left(\mathbb{R}^{m_{e}}\times\mathbb{R}^{m-m_{e}}_{+}\right)$, one tries to show that the corresponding sequence $\left\{(x_{k},\lambda^{(k)})\right\}_{k\geq 0}$ generated by the Augmented Lagrangian method possess the following asymptotic property (see, e.g., [1] and [2]):

• GC.

Given $\epsilon>0$, there exists $k_{0}\in\mathbb{N}$ such that

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

where $c_{E}(x)=(c_{1}(x),\ldots,c_{m_{e}}(x))$, $c_{I}(x)=(c_{m_{e}+1}(x),\ldots,c_{m}(x))$, and for any vector $v$, the corresponding vector $v^{(-)}$ is defined by $v_{i}^{(-)}=\min\left\{v_{i},0\right\}$.

In this paper we obtain worst-case complexity bounds for the first $k_{0}$ such that (1.4)-(1.7) hold, when $\left\{(x_{k},\lambda^{(k)})\right\}_{k\geq 0}$ is generated by a certain Augmented Lagrangian method that allows inexact solutions of its subproblems. Assuming that the penalty parameters are bounded, we prove a complexity bound of $\mathcal{O}(|\log(\epsilon)|)$ outer iterations for the referred algorithm to generate an $\epsilon$-approximate KKT point, for $\epsilon\in(0,1)$. When the penalty parameters are unbounded, we prove an outer iteration complexity bound of $\mathcal{O}\left(\epsilon^{-2/(\alpha-1)}\right)$, where $\alpha>1$ controls the rate of increase of the penalty parameters. In light of these results, for linearly constrained problems we are able to obtain evaluation complexity bounds of $\mathcal{O}(|\log(\epsilon)|^{2}\epsilon^{-2})$ and $\mathcal{O}\left(\epsilon^{-\left(\frac{2(2+\alpha)}{\alpha-1}+2\right)}\right)$, respectively, when appropriate first-order methods ($p=1$) are used to approximately solve the unconstrained subproblems at each iteration of our Augmented Lagrangian scheme. For problems having only linear equality constraints, these bounds are improved to $\mathcal{O}(|\log(\epsilon)|^{2}\epsilon^{-(p+1)/p})$ and $\mathcal{O}\left(\epsilon^{-\left(\frac{4}{\alpha-1}+\frac{p+1}{p}\right)}\right)$, respectively, when suitable $p$-order methods ($p\geq 2$) are used as inner solvers.

1.1 Related Literature

[25] proposed a Primal-Dual Algorithm (Prox-PDA) for the linearly constrained problem

[TABLE]

It was shown that Prox-PDA enjoys an outer iteration complexity of $\mathcal{O}(\epsilon^{-1})$. Therefore, even when the penalty parameters are unbounded, our outer iteration complexity bound of $\mathcal{O}(\epsilon^{-\frac{2}{\alpha-1}})$ (given in Theorem 8) is bether than the one proved in [25], if we take $\alpha>3$. Regarding the evaluation complexity, when the penalty parameters are bounded, our bound of $\mathcal{O}\left(|\log(\epsilon)|^{2}\epsilon^{-2}\right)$ for $p=1$ (given in Theorem 12) is slightly worse (in terms of $\epsilon$) than the bounds of $\mathcal{O}(\epsilon^{-2})$ proved for the schemes proposed by [12, 13, 15] and by [10].

In the context of second-order methods, [14] proposed a Two-Phase algorithm for the general constrained problem (1.1)-(1.3). When the subproblems in their method are solved by a suitable second-order scheme, they proved a complexity bound of $\mathcal{O}\left(\epsilon^{-\frac{3}{2}}\right)$ problem evaluations. Bounds of the same order for different second-order methods were also proved by [18] for equality constrained problems and by [7] and [23] for linearly constrained problems. Improve complexity bounds of $\mathcal{O}\left(\epsilon^{-\frac{p+1}{p}}\right)$ for $p$-order methods in the constrained case have been obtained by [5], [26] and [16, 17]. Thus, assuming that the penalty parameters are bounded, our evaluation complexity bound of $\mathcal{O}\left(|\log(\epsilon)|^{2}\epsilon^{-\frac{p+1}{p}}\right)$ (given in Theorem 14) for problem (1.8)-(1.9) is slightly worse than the bounds mentioned above111In fact, we can obtain a bound of $\mathcal{O}\left(|\log(\epsilon)|^{2}\epsilon^{-\frac{p+1}{p}}\right)$ even when the penalty parameters are unbounded. This is discussed on Remark 3.2..

Finally, very recently, the worst-case complexity of Augmented Lagrangian methods for nonconvex problems have also been investigated by [9] and [33]. Specifically, [9] prove a bound of $\mathcal{O}(|\log(\epsilon)|)$ outer iterations for the Augmented Lagrangian method corresponding to Algencan (see [1]) to find an $\epsilon$-approximate KKT point (or an approximate stationary point of a infeasibility measure, when the penalty parameter goes to infinity). On the other hand, [33] analyze the worst-case complexity of a Proximal Augmented Lagrangian method for problems having only equality constraints. Using a constant penalty parameter $\sigma=\mathcal{O}(\epsilon^{-\eta})$, they show that the referred algorithm takes at most $\mathcal{O}(\epsilon^{-(2-\eta)})$ outer iterations to find an $\epsilon$-approximate first-order (or second-order) KKT point, where $\eta$ is an algorithmic parameter chosen in the interval $[0,2]$. Complexity bounds for the total number of inner iterations are also obtained when the Newton-CG method ($p=2$) proposed by [30] is used to solve the subproblems. For problems with linear equality constraints and $\eta=2$, they prove evaluations complexity bounds of $\mathcal{O}(\epsilon^{-3/2})$ and $\mathcal{O}(\epsilon^{-3})$ for approximate first-order and second-order optimality, respectively.

1.2 Contents

The paper is organized as follows. In section 2, we describe the algorithm and obtain outer iteration complexity bounds. In section 3, we obtain worst-case evaluation complexity bounds for linearly constrained problems. Finally, in section 4, we summarize our results and mention some directions for future research.

2 Algorithm and Outer Iteration Complexity Analysis

In what follows we will consider the following assumptions:

• A1

Function $f(\,.\,)$ is continuously differentiable, bounded below by $f_{low}$ and, for every $u\in\mathbb{R}$, the set

[TABLE]

is bounded.

• A2

There exists $\bar{x}\in\mathbb{R}^{n}$ satisfying (1.2)-(1.3) and, for each $i\in\left\{1,\ldots,m\right\}$, function $c_{i}(\,.\,)$ is continuously differentiable.

The Lagrangian function $L(x,\lambda)$ and the Augmented Lagrangian function $P(x,\lambda,\sigma)$ associated to (1.1)-(1.3) are given respectively by:

[TABLE]

and

[TABLE]

where $\lambda_{i}$ ($i=1,\ldots,m$) are Lagrange multipliers, $\sigma$ is the penalty parameter, and $(\tau)_{-}\equiv\min\left\{0,\tau\right\}$.

Let us consider the following Augmented Lagrangian Method. It is a slight variation of Algorithm 10.4.1 in [32], with a different rule for updating the penalty parameter. Moreover, we allow inexact solutions of the subproblems.

Algorithm 1. Augmented Lagrangian Method

Step 0. Given a feasible point $x_{0}\in\mathbb{R}^{n}$ of (1.2)-(1.3), $\lambda^{(0)}\in\mathbb{R}^{m}$ with $\lambda_{i}^{(0)}\geq 0$ for $i=m_{e}+1,\ldots,m$, $\alpha>1$, $\sigma^{(0)}>0$ and $\gamma,\epsilon\in(0,1)$, set $k:=0$.

Step 1. Find an approximate solution $x_{k+1}$ to

(2.16)

$\min_{x\in\mathbb{R}^{n}}\,P(x,\lambda^{(k)},\sigma^{(k)}),$

such that

(2.17)

$P(x_{k+1},\lambda^{(k)},\sigma^{(k)})\leq\min\left\{P(x_{k},\lambda^{(k)},\sigma^{(k)}),P(x_{0},\lambda^{(k)},\sigma^{(k)})\right\},$

and

(2.18)

$\|\nabla_{x}P(x_{k+1},\lambda^{(k)},\sigma^{(k)})\|_{\infty}\leq\epsilon.$

Step 2. Compute

(2.19)

$\theta^{(k+1)}=\max\left\{\left\|\frac{\lambda^{(k)}}{\sigma^{(k)}}\right\|_{\infty},\left\|c_{E}(x_{k+1})-\frac{\lambda_{E}^{(k)}}{\sigma^{(k)}}\right\|_{\infty},\|c_{I}^{(-)}(x_{k+1})\|_{\infty}\right\},$

where $\lambda_{E}^{(k)}=(\lambda_{1}^{(k)},\ldots,\lambda_{m_{e}}^{(k)})$. If $k=0$, set $\sigma^{(k+1)}=\sigma^{(k)}$. Otherwise, set

(2.20)

\sigma^{(k+1)}=\left\{\begin{array}[]{ll}\sigma^{(k),}&\text{if}\quad\theta^{(k+1)}\leq\gamma\theta^{(k)},\\ \max\left\{(k+1)^{\alpha},\sigma^{(k)}\right\},&\text{otherwise.}\end{array}\right.

Step 3. Set

(2.21)

$\lambda_{i}^{(k+1)}=\lambda_{i}^{(k)}-\sigma^{(k)}c_{i}(x_{k+1}),\,\,\text{for}\,\,i=1,\ldots,m_{e},$

and

(2.22)

$\lambda_{i}^{(k+1)}=\max\left\{\lambda_{i}^{(k)}-\sigma^{(k)}c_{i}(x_{k+1}),0\right\}\,\,\text{for}\,\,i=m_{e}+1,\ldots,m.$

Step 4. Set $k:=k+1$ and go back to Step 1.

In this section, our goal is to establish iteration complexity bounds for Algorithm 1, that is, upper bounds on the number of outer iterations necessary to generate a pair $(x_{k},\lambda^{(k)})$ such that

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

for a given $\epsilon\in(0,1)$. For that, we need some auxiliary results. The first one states that the Augmented Lagrangian function is bounded from above by the objective function on points of the feasible set.

Lemma 1**.**

Suppose that A2 holds. Let $\sigma>0$, $\lambda\in\mathbb{R}^{m}$ with $\lambda_{i}\geq 0$ for $i=m_{e}+1,\ldots,m$, and $\bar{x}\in\mathbb{R}^{n}$ be a feasible point of (1.2)-(1.3). Then,

[TABLE]

Proof.

By assumptions, for $i=m_{e}+1,\ldots,m$, we have $c_{i}(\bar{x})\geq 0$ and $\dfrac{\lambda_{i}}{\sigma}\geq 0$. Thus,

[TABLE]

Consequently,

[TABLE]

which shows that

[TABLE]

Then, (2.27) follows from the above inequality, the equality $c_{i}(\bar{x})=0$ for $i=1,\ldots,m_{e}$ and (2.15). ∎

The lemma below gives a lower bound for the Augmented Lagrangian function.

Lemma 2**.**

Let $\sigma>0$ and $\lambda\in\mathbb{R}^{m}$ with $\lambda_{i}\geq 0$ for $i=m_{e}+1,\ldots,m$. Then,

[TABLE]

Proof.

By (2.15), we have

[TABLE]

for all $x\in\mathbb{R}^{n}$. ∎

In view of Lemmas 1 and 2, we can show that the iterates of Algorithm 1 are well defined.

Lemma 3**.**

Suppose that A1 and A2 hold. Given $\sigma>0$ and $\lambda\in\mathbb{R}^{m}$, with $\lambda_{i}\geq 0$ for $i=m_{e}+1,\ldots,m$, let $P(\,.\,,\lambda,\sigma)$ be defined by (2.15). Then, there exists at least one point $x^{+}$ such that

[TABLE]

Proof.

By A1 and A2, it follows that $P(\,.\,,\lambda,\sigma)$ is differentiable. Thus, it is enough to show that $P(\,.\,,\lambda,\sigma)$ has a global minimizer. For that, given $\bar{x}\in\mathbb{R}^{n}$ feasible, we will show that the set

[TABLE]

is compact (then the existence of a global minimizer will follows from Weierstrass Theorem). Indeed, consider $x\in\mathcal{L}_{p}(\bar{x})$. Then, by Lemmas 1 and 2 we have

[TABLE]

[TABLE]

and so, $x\in\mathcal{L}_{f}(u)$ for $u=f(\bar{x})+\frac{1}{2}\sum_{i=1}^{m}\left[\lambda_{i}\right]^{2}/\sigma$, where $\mathcal{L}_{f}(u)$ is defined in (2.10). This means that $\mathcal{L}_{P}(\bar{x})\subset\mathcal{L}_{f}(u)$. Since $\mathcal{L}_{f}(u)$ is bounded (due to A1), it follows that $\mathcal{L}_{P}(\bar{x})$ is also bounded. Then, by the continuity of $P(\,.\,,\lambda,\sigma)$ we conclude that $\mathcal{L}_{P}(\bar{x})$ is compact. ∎

The next lemma provides an upper bound for the constraint violation on infeasible points.

Lemma 4**.**

Let $\sigma>0$, $\lambda\in\mathbb{R}^{m}$ with $\lambda_{i}\geq 0$ for $i=m_{e}+1,\ldots,m$, and $\bar{x}\in\mathbb{R}^{n}$ be a feasible point of (1.2)-(1.3). If $x^{+}$ is an infeasible point of (1.2)-(1.3) such that

[TABLE]

then

[TABLE]

where $\lambda_{E}=(\lambda_{1},\ldots,\lambda_{m_{e}})$.

Proof.

From (2.28), Lemma 1 and (2.15), it follows that

[TABLE]

Therefore,

[TABLE]

Let $J=\left\{j\in\left\{m_{e}+1,\ldots,m\right\}\,|\,c_{j}(x^{+})<0\right\}$. Since $x^{+}$ is an infeasible point to (1.2)-(1.3), we may have $J\neq\emptyset$. In this case,

[TABLE]

For $i\in J$, we have

[TABLE]

which gives

[TABLE]

Thus, combining (2.30), (2.31), (2.32) and the norm inequality $\|\,.\,\|_{2}\geq\|\,.\,\|_{\infty}$, we get (2.29).

∎

The following lemma specializes the upper bound from Lemma 4 to points generated by Algorithm 1.

Lemma 5**.**

Suppose that A1 and A2 hold, and let the sequence $\left\{(x_{k},\lambda^{(k)},\sigma^{(k)})\right\}_{k\geq 0}$ be generated by Algorithm 1. Then, for all $k\geq 1$, we have

[TABLE]

where

[TABLE]

Proof.

Since $x_{0}$ is a feasible point of (1.2)-(1.3), it follows from (2.17), Lemma 4, (2.34) and the bound $f(x_{k+1})\geq f_{low}$ that

[TABLE]

Now, let us compute an upper bound for $\|\mu^{(k)}\|_{2}^{2}$. From (2.34), (2.20), (2.21), (2.22), (2.15), Lemma 1 and (2.17), we have

[TABLE]

Therefore, by induction,

[TABLE]

Finally, combining (2.35) and (2.37) we get (2.33). ∎

Our last auxiliary result gives an upper bound for $\theta^{(k)}$.

Lemma 6**.**

Suppose that A1 and A2 hold, and let the sequence $\left\{(x_{k},\lambda^{(k)},\sigma^{(k)})\right\}_{k\geq 0}$ be generated by Algorithm 1. Then

[TABLE]

where $\theta^{(k)}$ is defined in (2.19).

Proof.

By (2.34), we have

[TABLE]

Thus, combining (2.39) and (2.37), it follows that

[TABLE]

Finally, combining (2.40), (2.33) and (2.19), we obtain (2.38). ∎

Now, we will analyse the outer iteration complexity of Algorithm 1 considering separately the following cases:

• (i)

$\left\{\sigma^{(k)}\right\}_{k\geq 0}$ is bounded from above by $\sigma_{max}$.

• (ii)

$\lim_{k\to+\infty}\sigma^{(k)}=+\infty$.

By (2.11), (2.21), (2.22), (2.15) and (2.18), we have

[TABLE]

for all $k\geq 1$. Thus, to bound the number of outer iterations necessary to ensure (2.23)-(2.26), we only need to bound the number of iterations in which (2.24) or (2.26) does not hold. Note that, if

[TABLE]

and $c_{i}(x_{k})>\epsilon$, then

[TABLE]

and so

[TABLE]

Therefore, if (2.41) holds, by (2.22) we have

[TABLE]

On the other hand, it is easy to see that

[TABLE]

In view of these remarks, all we have to do is to bound the number of iterations in which

[TABLE]

The theorem below gives an upper bound of $\mathcal{O}\left(|\log(\epsilon)|\right)$ iterations in case (i).

Theorem 7**.**

Suppose that A1 and A2 hold, and let the sequence $\left\{(x_{k},\lambda^{(k)},\sigma^{(k)})\right\}_{k\geq 0}$ be generated by Algorithm 1. Moreover, assume that $\left\{\sigma^{(k)}\right\}_{k\geq 0}$ is bounded from above by $\sigma_{max}$. If

[TABLE]

then

[TABLE]

where $\mu^{(0)}$ is defined by (2.34) and $\alpha$ and $\gamma$ are the parameters in (2.20).

Proof.

Consider the set

[TABLE]

Given $k\in\mathcal{U}$, we must have $k\leq\sigma_{max}^{\frac{1}{\alpha}}$, since otherwise, by (2.20) we would have $\sigma^{(k+1)}\geq(k+1)^{\alpha}>\sigma_{max}$, contradicting our assumption on the penalty parameters.

Let us denote $\bar{k}=\max\left\{j\,|\,j\in\mathcal{U}\right\}$. By (2.42), we have $T=\bar{k}+2+s$ with

[TABLE]

Note that

[TABLE]

Consequently,

[TABLE]

Since $\bar{k}\in\mathcal{U}$, it follows that $\sigma^{(\bar{k}+1)}\geq(\bar{k}+1)^{\alpha}$. Thus, by Lemma 2.4 we have

[TABLE]

[TABLE]

[TABLE]

Combining (2.46) and (2.47), we obtain

[TABLE]

By (2.45),

[TABLE]

Therefore,

[TABLE]

and the proof is complete. ∎

The next theorem establishes an upper bound of $\mathcal{O}\left(\epsilon^{-\frac{2}{\alpha-1}}\right)$ iterations in case (ii).

Theorem 8**.**

Suppose that A1 and A2 hold, and let the sequence $\left\{(x_{k},\lambda^{(k)},\sigma^{(k)})\right\}_{k\geq 0}$ be generated by Algorithm 1 such that

[TABLE]

If $\lim_{k\to+\infty}\sigma^{(k)}=+\infty$, then

[TABLE]

where $\mu^{(0)}$ is defined by (2.34) and $\alpha$ and $\gamma$ are the parameters in (2.20).

Proof.

Again, consider the set $\mathcal{U}$ defined in (2.44). Given $k\in\mathcal{U}$, it follows from (2.20) and Lemma 6 that

[TABLE]

[TABLE]

Thus,

[TABLE]

By (2.48), this means that

[TABLE]

Suppose that $T-2>\tilde{k}+2$ and define $s=(T-2)-(\tilde{k}+2)$. Then,

[TABLE]

and so

[TABLE]

Since $\tilde{k}\in\mathcal{U}$, it follows that $\sigma^{(\tilde{k}+1)}\geq(\tilde{k}+1)^{\alpha}$. Thus, using Lemma 6 as in the proof of Theorem 7, we obtain

[TABLE]

Combining (2.50) and (2.51), we obtain

[TABLE]

Note that, if

[TABLE]

we would have

[TABLE]

contradicting (2.48). Hence, we must have

[TABLE]

Therefore,

[TABLE]

which gives (2.49). ∎

In summary, Theorem 7 means that if the sequence of penalty parameters is bounded from above, then Algorithm 1 takes at most $\mathcal{O}(|\log(\epsilon)|)$ outer iterations to generate $(x_{k},\lambda^{(k)})$ satisfying (2.23)-(2.26), that is, an $\epsilon$-approximate KKT point of (1.1)-(1.3). On the other hand, if the sequence of penalty parameters is unbounded, Theorem 8 gives an upper bound of $\mathcal{O}(\epsilon^{-\frac{2}{\alpha-1}})$ outer iterations, where $\alpha>1$ defines the increase of the penalty parameter. As expected (see [3], p. 104), the bigger $\alpha$ is, the more aggressive the update of the penalty parameter is, and consequently, the maximum number of outer iterations needed for Algorithm 1 to find and $\epsilon$-approximate KKT point is smaller.

3 Evaluation Complexity for Linearly Constrained Problems

At each outer iteration of Algorithm 1, subproblem (2.16) must be approximately solved. In general, this is done by applying an iterative optimization method for unconstrained problems. When its invoked, this unconstrained method requires a certain number of calls of the oracle222By calls of the oracle we mean the joint computation of $f$, $c_{i}$ ($i=1,\ldots,m$) and their derivatives., which is proportional to the number of inner iterations. Therefore, a full estimation of the worst-case complexity of Algorithm 1 must also take into account the iteration complexiy of the auxiliary unconstrained method used to solve the subproblems. With this goal in mind, now we shall consider the following special case of (1.1)-(1.3):

[TABLE]

where $f:\mathbb{R}^{n}\to\mathbb{R}$ is continuously differentiable, possibly nonconvex, $a_{i}\in\mathbb{R}^{n}$ and $b_{i}\in\mathbb{R}$ for $i=1,\ldots,m$. Given $\lambda\in\mathbb{R}^{m}$, with $\lambda_{i}\geq 0$ for $i=m_{e}+1,\ldots,m$, and $\sigma>0$, the corresponding augmented Lagrangian function to (3.52)-(3.54) is

[TABLE]

For each $i\in\left\{1,\ldots,n\right\}$, let us denote by $a_{ij}$ the $j$-th component of vector $a_{i}$. Specifically, let us consider the following additional assumption on $f$:

• A3

The gradient of $f(\,.\,)$, $\nabla f:\mathbb{R}^{n}\to\mathbb{R}^{n}$, is $L_{1}$-Lipschitz continuous, i.e.,

[TABLE]

Lemma 9**.**

Given $\lambda\in\mathbb{R}^{n}$, with $\lambda_{i}\geq 0$ for $i=m_{e}+1,\ldots,m$, and $\sigma>0$, let function $P(\,.\,,\lambda,\sigma)$ be defined by (3.57). Suppose that A3 holds and that

[TABLE]

Then,

[TABLE]

where the $i$-th row of $A\in\mathbb{R}^{m\times n}$ is vector $a_{i}^{T}$ and $\|\,.\,\|_{F}$ denotes the Frobenius norm.

Proof.

Indeed, from (3.57), a direct calculation shows that

[TABLE]

Thus, given $x,y\in\mathbb{R}^{n}$, we have

[TABLE]

where

[TABLE]

for $i=m_{e}+1,\ldots,m$. If $\max\left\{a_{i}^{T}x-b_{i},a_{i}^{T}y-b_{i}\right\}<\frac{\lambda_{i}}{\sigma}$, then

[TABLE]

If $a_{i}^{T}x-b_{i}<\frac{\lambda_{i}}{\sigma}\leq a_{i}^{T}y-b_{i}$, then (3.58) implies that

[TABLE]

and so

[TABLE]

Finally, if $a_{i}^{T}y-b_{i}<\frac{\lambda_{i}}{\sigma}\leq a_{i}^{T}x-b_{i}$, then (3.58) implies that

[TABLE]

and so

[TABLE]

In summary,

[TABLE]

Combining (3.61) and (3.62), we obtain

[TABLE]

Similarly, we also have

[TABLE]

Therefore,

[TABLE]

From (3.63) and A3, we obtain

[TABLE]

that is, (3.59) holds. ∎

Remark 10**.**

Recall that any linearly constrained problem can be formulated in the standard form

[TABLE]

for which assumption (3.58) holds.

From Lemma 9 it follows that $\nabla P(\,.\,,\lambda,\sigma)$ is Lipschitz continuous when $\nabla f(\,.\,)$ is Lipschitz continuous. In this case, one can minimize $P(\,.\,,\lambda,\sigma)$ by using a first-order method $\mathcal{M}_{1}$. More specifically, let us consider the following assumption on $\mathcal{M}_{1}$:

• A4

Given any differentiable function $g:\mathbb{R}^{n}\to\mathbb{R}$, bounded below by $g_{low}$, with $L$-Lipschitz gradient, method $\mathcal{M}_{1}$ with starting point $\tilde{x}$, can find an $\epsilon$-approximate stationary point of $g(\,.\,)$ in at most

[TABLE]

iterations, where $C_{\mathcal{M}_{1}}$ is a positive constant that depends only on the method $\mathcal{M}_{1}$.

Examples of methods that satisfy A4 can be found in [27], [22], [11] and [21]. The next lemma establishes an upper bound on the number of iterations for these methods to compute approximate solutions of (2.16) satisfying (2.17)-(2.18).

Lemma 11**.**

Suppose that A1 and A3 hold, and let $\left\{(x_{j},\lambda^{(j)},\sigma^{(j)})\right\}_{j=0}^{k}$ be generated by Algorithm 1 applied to (3.52)-(3.54), with $\left\{a_{i}\right\}_{i=m_{e}+1}^{m}$ satisfying (3.58). Moreover, assume that a monotone method $\mathcal{M}_{1}$ is applied to minimize $P(\,.\,,\lambda^{(k)},\sigma^{(k)})$ with starting point

[TABLE]

If method $\mathcal{M}_{1}$ satisfies A4, then $\mathcal{M}_{1}$ takes at most $\mathcal{O}\left(\max\left\{1,\sigma^{(k)}\right\}(k+1)\sqrt{n}\epsilon^{-2}\right)$ iterations to generate $\tilde{x}_{k,\ell}$ such that

[TABLE]

Proof.

By Lemma 2, A1 and (2.37), for all $x\in\mathbb{R}^{n}$ we have

[TABLE]

On the other hand, it follows from (3.67) and Lemma 1 that

[TABLE]

Finally, from A3 and Lemma 9, it follows that $\nabla P(\,.\,,\lambda^{(k)},\sigma^{(k)})$ is Lipschitz continuous with constant

[TABLE]

Thus, by A4, (3.68) and (3.69), we conclude that method $\mathcal{M}_{1}$ takes at most

[TABLE]

iterations to generate an $\epsilon$-approximate stationary point of $P(\,.\,,\lambda^{(k)},\sigma^{(k)})$. ∎

Now, combining Theorems 7 and 8 with Lemma 11 we can obtain worst-case complexity bounds for the total number of inner iterations performed in Algorithm 1 to find an $\epsilon$-approximate KKT point of (3.52)-(3.54).

Theorem 12**.**

Suppose that Algorithm 1 is applied to solve (3.52)-(3.54), with $f$ satisfying A1 and A3, and $\left\{a_{i}\right\}_{m_{e}+1}^{m}$ satisfying (3.58). Moreover, assume that at each outer iteration of Algorithm 1, a monotone method $\mathcal{M}_{1}$ satisfying A4 is used to approximately solve (2.16) with starting point $\tilde{x}_{k,0}$ given in (3.67). Then, the following statements are true:

• (a)

If $\left\{\sigma^{(k)}\right\}_{k\geq 0}$ is bounded from above by $\sigma_{max}$, then Algorithm 1 takes at most $\mathcal{O}\left(\sigma_{max}\sqrt{n}|\log(\epsilon)|^{2}\epsilon^{-2}\right)$ inner iterations of $\mathcal{M}_{1}$ to generate and $\epsilon$-approximate KKT point of (3.52)-(3.54).

• (b)

If $\lim_{k\to+\infty}\sigma^{(k)}=+\infty$ and $\alpha\geq 2$ is integer, then Algorithm 1 takes at most $\mathcal{O}\left(\sqrt{n}\epsilon^{-\left(\frac{2(2+\alpha)}{\alpha-1}+2\right)}\right)$ iterations of $\mathcal{M}_{1}$ to generate an $\epsilon$-approximate KKT point of (3.52)-(3.54).

Proof.

By A1 and Theorems 7 and 8, there exists an iteration number $T$ such that (2.23)-(2.26) hold for $k=T+1$. Without loss of generality, assume that $\sigma^{(0)}=1$. Then, $\sigma^{(k)}\geq 1$ and, by Lemma 11, the total number of inner iterations of $\mathcal{M}_{1}$ performed until iteration $T+1$ of Algorithm 1 is proportional to

[TABLE]

If $\left\{\sigma^{(k)}\right\}_{k\geq 0}$ is bounded from above by $\sigma_{max}$, we have

[TABLE]

Thus, by Theorem 7, we get a bound of $\mathcal{O}\left(|\log(\epsilon)|^{2}\sigma_{max}\sqrt{n}\epsilon^{-2}\right)$ for the total number of inner iterations of $\mathcal{M}_{1}$, proving statement (a).

Suppose that $\lim_{k\to+\infty}\sigma^{(k)}=+\infty$. By (2.20) and $\sigma^{(0)}=1$ we have $\sigma^{(k)}\leq k^{\alpha}$ for all $k$. Since $\alpha$ is a positive integer, we have

[TABLE]

where the last equality is due to the fact that $\sum_{k=1}^{T+2}k^{1+\alpha}=\mathcal{O}((T+2)^{2+\alpha})$ (see, e.g., [31]). Thus, by Theorem 8, we get a bound of $\mathcal{O}\left(\sqrt{n}\epsilon^{-\left(\frac{2(2+\alpha)}{\alpha-1}+2\right)}\right)$ for the total number of inner iterations of $\mathcal{M}_{1}$, proving statement (b). ∎

Now, let us consider linearly constrained problems of the form

[TABLE]

where $f:\mathbb{R}^{n}\to\mathbb{R}$ is $p$-times differentiable ($p\geq 2$), possibly nonconvex. This is a particular case of (1.1)-(1.3) with $m=m_{e}$. The corresponding augmented Lagrangian function to (3.70)-(3.71) is

[TABLE]

We denote by $D^{p}f(x)$ the $p$-th order derivative of $f$ at point $x$, which is a tensor defined by

[TABLE]

In particular $D^{2}f(x)=\nabla^{2}f(x)$. The next lemma relates the $p$-th order derivatives of $P(\,.\,,\lambda,\sigma)$ and $f(\,.\,)$.

Lemma 13**.**

Given $f:\mathbb{R}^{n}\to\mathbb{R}$, $\lambda\in\mathbb{R}^{m}$ and $\sigma>0$, let $P(x,\lambda,\sigma)$ be defined by (3.72). If $f$ is $p$-times differentiable with $p\geq 2$, then

[TABLE]

where the $i$-th row of $A\in\mathbb{R}^{m\times n}$ is vector $a_{i}^{T}$.

Proof.

Indeed, from (3.72), a direct calculation shows that

[TABLE]

Then,

[TABLE]

which gives

[TABLE]

Therefore, (3.73) holds. ∎

For $p\geq 2$, it follows from Lemma 13 that $D^{p}P(\,.\,,\lambda,\sigma)$ is $L_{p}$-Lipschitz continuous when $D^{p}f(\,.\,)$ is $L_{p}$-Lipschitz continuous. In this case, one can minimize $P(\,.\,,\lambda,\sigma)$ by using a $p$-order method $\mathcal{M}_{p}$. More specifically, let us consider the following assumptions on $f$ and $\mathcal{M}_{p}$:

• A5.

$D^{p}f(\,.\,)$ is $L_{p}$-Lipschitz continuous, i.e.,

[TABLE]

• A6.

Given any $p$-times differentiable function $g:\mathbb{R}^{n}\to\mathbb{R}$, bounded below by $g_{low}$, with $L$-Lipschitz continous $p$-th order derivative ($p\geq 2$), method $\mathcal{M}_{p}$ with starting point $\tilde{x}$, can find an $\epsilon$-approximate stationary point of $g(\,.\,)$ in at most

[TABLE]

iterations, where $C_{\mathcal{M}_{p}}$ is a positive constant that depends only on the method $\mathcal{M}_{p}$.

An important class of unconstrained methods that satisfy A6 for $p=2$ is the class of methods based on the cubic regularization of Newton’s method (see, e.g., [28, 11, 6, 19]). Moreover, for $p\geq 3$, several tensor methods satisfying A6 have been proposed recently (see, e.g., [6, 16, 20]).The use of these tensor methods to approximately solve (2.16) give us complexity bounds better than the ones obtained in Theorem 12, as we can see below.

Theorem 14**.**

Suppose that Algorithm 1 is applied to solve (3.70)-(3.71) with $f$ satisfying A1 and A5. Moreover, assume that at each outer iteration of Algorithm 1, a monotone method $\mathcal{M}_{p}$ satisfying A6 is used to approximately solve (2.16) with starting point $\tilde{x}_{k,0}$ given in (3.67). Then, the following statements are true:

• (a)

If $\left\{\sigma^{(k)}\right\}_{k\geq 0}$ is bounded from above by $\sigma_{max}$, then Algorithm 1 takes at most $\mathcal{O}\left(|\log(\epsilon)|^{2}\epsilon^{-\frac{p+1}{p}}\right)$ inner iterations of $\mathcal{M}_{p}$ to generate an $\epsilon$-approximate KKT point of (3.70)-(3.71).

• (b)

If $\lim_{k\to+\infty}\sigma^{(k)}=+\infty$, then Algorithm 1 takes at most $\mathcal{O}\left(\epsilon^{-\left(\frac{4}{\alpha-1}+\frac{p+1}{p}\right)}\right)$ inner iterations of $\mathcal{M}_{p}$ to generate an $\epsilon$-approximate KKT point of (3.70)-(3.71).

Proof.

As in Lemma 11, by A1, A5 and A6 we conclude that, at the $k$-th iteration, method $\mathcal{M}_{p}$ takes at most $\mathcal{O}\left(k\epsilon^{-\frac{p+1}{p}}\right)$ iterations to generate $\tilde{x}_{k,\ell}$ such that

[TABLE]

Thus, the total number of inner iterations of $\mathcal{M}_{p}$ performed until iteration $T+1$ is proportional to

[TABLE]

Consequently, (a) and (b) follow directly from the upper bounds on $T$ given in (2.42) and (2.49). ∎

Remark 15**.**

It is worth mentioning that a complexity bound of $\mathcal{O}\left(|\log(\epsilon)|^{2}\epsilon^{-\frac{p+1}{p}}\right)$ can be obtained even when the penalty parameters are unbounded. For that, it is enough to replace (2.20) by the following update rule:

[TABLE]

Indeed, using (3.74) in the proof of Theorem 8, we can obtain $\tilde{k}\leq\mathcal{O}\left(|\log(\epsilon)|\right)$ which gives an outer iteration complexity bound of $\mathcal{O}\left(|\log(\epsilon)|\right)$. This corresponds to an evaluation complexity bound of $\mathcal{O}\left(|\log(\epsilon)|^{2}\epsilon^{-\frac{p+1}{p}}\right)$ for problem (3.70)-(3.71) when the penalty parameters are unbounded. However, (3.74) increases the penalty parameters much more aggressively than (2.20), and can lead to a premature ill-conditioning of (2.16). Therefore, despite the improved worst-case complexity, it is unlikely that (3.74) will give an efficient algorithm in practice.

Remark 16**.**

Consider now the following problem

[TABLE]

where $f:\mathbb{R}^{n}\to\mathbb{R}$ is four times differentiable, possibly nonconvex, $R_{i}\in\mathbb{R}^{n\times n}$, $s_{i}\in\mathbb{R}^{n}$ and $t_{i}\in\mathbb{R}$, for $i=1,\ldots,m$. Let $P(x,\lambda,\sigma)$ denote the augmented Lagrangian function corresponding to (3.75)-(3.76). Then, as in Lemma 13, one can see that $D^{5}P(x,\lambda,\sigma)=D^{5}f(x)$ for all $x\in\mathbb{R}^{n}$. Thus, if $D^{5}f(\,.\,)$ is bounded, it follows that $D^{4}P(\,.\,,\lambda,\sigma)$ is Lipschitz continuous, and we can apply Algorithm 1 to (3.75)-(3.76), solving (2.16) with a fourth-order tensor method $\mathcal{M}_{4}$. Specifically, if $\mathcal{M}_{4}$ satisfies A6 for $p=4$, then (as in Theorem 14) we conclude that Algorithm 1 applied to (3.75)-(3.76) takes at most $\mathcal{O}\left(|\log(\epsilon)|^{2}\epsilon^{-\frac{5}{4}}\right)$ inner iterations of $\mathcal{M}_{4}$ if $\left\{\sigma^{(k)}\right\}$ is bounded, and $\mathcal{O}\left(\epsilon^{-\left(\frac{4}{\alpha-1}+\frac{5}{4}\right)}\right)$ inner iterations of $\mathcal{M}_{4}$ if $\left\{\sigma^{(k)}\right\}$ is unbounded.

4 Conclusion

In this paper, we have studied the worst-case complexity of an inexact Augmented Lagrangian method for constrained nonconvex optimization problems. For the case in which the penalty parameters are bounded, we established a complexity bound of $\mathcal{O}(|\log(\epsilon)|)$ outer iterations for the referred algorithm to generate an $\epsilon$-approximate KKT point, for $\epsilon\in(0,1)$. For the case in which the penalty parameters are unbounded, we proved an iteration complexity bound of $\mathcal{O}\left(\epsilon^{-2/(\alpha-1)}\right)$, where $\alpha>1$ controls the rate of increase of the penalty parameters. In the particular class of linearly constrained problems, these bounds yield to evaluation complexity bounds of $\mathcal{O}(\sigma_{max}\sqrt{n}|\log(\epsilon)|^{2}\epsilon^{-2})$ and $\mathcal{O}\left(\sqrt{n}\epsilon^{-\left(\frac{2(2+\alpha)}{\alpha-1}+2\right)}\right)$, respectively, when appropriate first-order methods ($p=1$) are used to approximately solve the unconstrained subproblems at each iteration. For problems having only linear equality constraints, these bounds are improved to $\mathcal{O}(|\log(\epsilon)|^{2}\epsilon^{-(p+1)/p})$ and $\mathcal{O}\left(\epsilon^{-\left(\frac{4}{\alpha-1}+\frac{p+1}{p}\right)}\right)$, respectively, when appropriate $p$-order methods ($p\geq 2$) are used as inner solvers.

A key point in the Augmented Lagrangian method considered in this work is that it requires the feasibility of the starting point $x_{0}$, which may be difficult to compute for general nonconvex constraints. Up to now, it is not clear how this restriction can be avoided. Another natural question is whether our analysis can be adapted in order to cover possibly less aggressive update rules for the penalty parameters, such as

[TABLE]

The authors are planning to address these and other interesting questions in their future research.

Acknowledgments

We are very grateful to Coralia Cartis and the two anonymous referees, whose comments helped to improve significantly the paper. We are also grateful to Mario Martínez and Stephen Wright for their insightful comments on the first version of this work.

Funding

G.N. Grapiglia was partially supported by the National Council for Scientific and Technological Development - Brazil (grant 406269/2016-5). Y. Yuan was partially supported by NSFC (grant 11688101).