Two efficient gradient methods with approximately optimal stepsizes based on regularization models for unconstrained optimization

TL;DR

This paper introduces two efficient gradient methods with approximately optimal stepsizes based on regularization models, improving convergence and performance in large-scale unconstrained optimization tasks.

Contribution

The paper proposes novel gradient methods utilizing regularization models for stepsize selection, enhancing efficiency and convergence in unconstrained optimization.

Findings

Proposed methods outperform the BBQ method in numerical experiments.

Methods are competitive with CGDESCENT and CGOPT software packages.

Convergence is established under weak conditions.

Abstract

It is widely accepted that the stepsize is of great significance to gradient method. Two efficient gradient methods with approximately optimal stepsizes mainly based on regularization models are proposed for unconstrained optimization. More exactly, if the objective function is not close to a quadratic function on the line segment between the current and latest iterates, regularization models are exploited carefully to generate approximately optimal stepsizes. Otherwise, quadratic approximation models are used. In particular, when the curvature is non-positive, special regularization models are developed. The convergence of the proposed methods is established under the weak conditions. Extensive numerical experiments indicated the proposed method is superior to the BBQ method (SIAM J. Optim. 2021,31(4), 3068-3096) and other efficient gradient methods, and is competitive to two famous and efficient conjugate gradient software packages CG$ \_ $DESCENT (5.0) (SIAM J. Optim. 16(1), 170-192, 2005) and CGOPT (1.0) (SIAM J. Optim. 23(1), 296-320, 2013) for the CUTEr library. Due to the surprising efficiency, we believe that gradient methods with approximately optimal stepsizes can become strong candidates for large-scale unconstrained optimization.

Tables

Method	$N_{\text{linsear}}=0$	$N_{\text{linsear}}\le 1$	$N_{\text{linsear}}\le 2$	$N_{\text{linsear}}\le 3$	total problems
BB	41	46	48	50	145(CUTEr145)
GM_AOS (Reg p=3)	68	81	85	90	145(CUTEr145)
GM_AOS (Reg p=4)	51	57	60	62	80(Andr80)

Equations

$$\underset{x\in R^n}{\min }f\left(x\right),$$

$$x_{k+1}=x_k-{\alpha }_k{g}_k,$$

$${\alpha }_k^{B{B}_1}=\frac{{\Vert s_{k-1}\Vert }^2}{s_{k-1}^T{y}_{k-1}}\text{or}{\alpha }_k^{B{B}_2}=\frac{s_{k-1}^T{y}_{k-1}}{{\Vert y_{k-1}\Vert }^2}.$$

$${\alpha }_k^{AOS}=\arg \underset{\alpha >0}{\min }{\varphi }_k(\alpha ).$$

$${\mu }_k=\left|\frac{2(f_{k-1}-f_k+g_k^T{s}_{k-1})}{s_{k-1}^T{y}_{k-1}}-1\right|.$$

$${\mu }_k\le c_1\text{or}\max \left\{{\mu }_k,{\mu }_{k-1}\right\}\le c_2,$$

$$m_k(d)=f(x_k)+g_k^{T}d+\frac{1}{2}{d}^T{B}_{k}d+\frac{1}{3}{\sigma }_k{\Vert d\Vert }^3,$$

$$m_k(d)=f(x_k)+g_k^{T}d+\frac{1}{2}{d}^T{B}_{k}d+\frac{1}{p}{\sigma }_k\left(p\right){\Vert d\Vert }^p,$$

$${\varphi }_{11}(\alpha )=f(x_k)-\alpha g_k^T{g}_k+\frac{1}{2}{\alpha }^2{g}_k^T{B}_k{g}_k+\frac{1}{3}{\alpha }^3{\sigma }_k\left(3\right){\Vert g_k\Vert }^3.$$

$$B_k=D_k-\frac{D_k{s}_{k-1}{s}_{k-1}^T{D}_k}{s_{k-1}^T{D}_k{s}_{k-1}}+\frac{{\overline{y}}_{k-1}{\overline{y}}_{k-1}^T}{s_{k-1}^T{\overline{y}}_{k-1}},$$

$${\overline{r}}_k=3(s_{k-1}^T{y}_{k-1}-s_{k-1}^T{\nabla }^{2}f(x_{k-1}+{\mu }_1{s}_{k-1})s_{k-1}),$$

$${\overline{r}}_k=\min \left\{\max \left\{{\overline{r}}_k,-{\xi }_1{s}_{k-1}^T{y}_{k-1}\right\},{\xi }_1{s}_{k-1}^T{y}_{k-1}\right\},$$

$${\Delta }_{11}=(g_k^T{B}_k{g}_k{)}^2+4{\sigma }_k\left(3\right){\Vert g_k\Vert }^5>0,$$

$${\bar{\alpha }}_k^{AOS(11)}=\frac{2{\Vert g_k\Vert }^2}{\sqrt{{\Delta }_{11}}+g_k^T{B}_k{g}_k}.$$

$${\alpha }_k^{AOS(11)}=\max \left\{\min \left\{{\bar{\alpha }}_k^{AOS(11)},{\alpha }_k^{B{B}_1}\right\}{\alpha }_k^{B{B}_2}\right\}$$

$${\varphi }_{12}(\alpha )=f(x_k)-\alpha g_k^T{g}_k+\frac{1}{2}{\alpha }^2{g}_k^T{B}_k{g}_k+\frac{1}{4}{\alpha }^4{\sigma }_k(4){\Vert g_k\Vert }^4,$$

$${\Delta }_{12}=\frac{1}{4{\sigma }_k(4){\Vert g_k\Vert }^4}+\frac{{\left(g_k^T{B}_k{g}_k\right)}^3}{27{\sigma }_k(4){\Vert g_k\Vert }^2}>0,$$

$${\bar{\alpha }}_k^{AOS(12)}=\sqrt[3]{\frac{1}{2{\sigma }_k(4){\Vert g_k\Vert }^2}+\sqrt{{\Delta }_{12}}}+\sqrt[3]{\frac{1}{2{\sigma }_k(4){\Vert g_k\Vert }^2}-\sqrt{{\Delta }_{12}}}.$$

$${\alpha }_k^{AOS(12)}=\max \left\{\min \left\{{\bar{\alpha }}_k^{AOS(12)},{\alpha }_k^{B{B}_1}\right\},{\alpha }_k^{B{B}_2}\right\}$$

$$f_{k-1}=f_k-g_k^T{s}_{k-1}+\frac{1}{2}{s}_{k-1}^T{B}_k{s}_{k-1}+\frac{{\sigma }_k\left(p\right)}{p}{\Vert s_{k-1}\Vert }^p,$$

$${\sigma }_k\left(p\right)=\frac{p\left(f_{k-1}-f_k+g_k^T{s}_{k-1}-\frac{1}{2}\left(s_{k-1}^T{y}_{k-1}+{\overline{r}}_k\right)\right)}{{\Vert s_{k-1}\Vert }^p},$$

$${\sigma }_k\left(p\right)=\max \left\{\min \left\{\left|{\sigma }_k\left(p\right)\right|,{\sigma }_{\max }\right\},{\sigma }_{\min }\right\},$$

$${\varphi }_2(\alpha )=f(x_k)-\alpha g_k^T{g}_k+\frac{1}{2}{\alpha }^2{g}_k^T{B}_k{g}_k,$$

$${\alpha }_k^{AOS(2)}=\frac{g_k^T{g}_k}{g_k^T{B}_k{g}_k}.$$

$${\alpha }_k^{AOS(2)}=\max \left\{\min \left\{{\alpha }_k^{AOS(2)},{\alpha }_k^{B{B}_1}\right\},{\alpha }_k^{B{B}_2}\right\}$$

$$\frac{{\Vert g_{k-1}\Vert }^2}{{\Vert g_k\Vert }^2}\ge {\xi }_2,$$

$$\varphi (\alpha )=f_k-\alpha g_k^T{g}_k+\frac{1}{2}{\alpha }^2{g}_k^T{\nabla }^{2}f(x_k)g_k+\frac{{\sigma }_k(3)}{3}{\alpha }^3{\Vert g_k\Vert }^3.$$

$$g_k^T{\nabla }^{2}f(x_k)g_k\approx g_{k-1}^T{\nabla }^{2}f(x_k)g_{k-1}\approx \frac{\left|{\left(g(x_k+{\alpha }_{k-1}{g}_{k-1})-g(x_k)\right)}^T{g}_{k-1}\right|}{{\alpha }_{k-1}}=\frac{\left|s_{k-1}^T{y}_{k-1}\right|}{{\alpha }_{k-1}^2},$$

$${\varphi }_{31}(\alpha )=f_k-\alpha g_k^T{g}_k+\frac{1}{2}{\alpha }^2\frac{\left|s_{k-1}^T{y}_{k-1}\right|}{{\alpha }_{k-1}^2}+\frac{{\sigma }_k(3)}{3}{\alpha }^3{\Vert g_k\Vert }^3.$$

$${\Delta }_{31}=\frac{{\left|s_{k-1}^T{y}_{k-1}\right|}^2}{{\alpha }_{k-1}^4}+4{\sigma }_k(3){\Vert g_k\Vert }^5>0,$$

Full text

††journal:

1 Introduction

We consider the unconstrained optimization problem:

[TABLE]

where $f:{R^{n}}\to R$ is continuously differentiable and its gradient is denoted by $g(x)$. The gradient method for solving (1) has the form

[TABLE]

where ${\alpha_{k}}$ is the stepsize and ${g_{k}}=\nabla f({x_{k}})$. Throughout this paper, ${f_{k}}=f({x_{k}})$, ${s_{k-1}}={x_{k}}-{x_{k-1}}$, ${y_{k-1}}={g_{k}}-{g_{k-1}}$ and $\left\|.\right\|$ denotes the Euclidean norm.

It is widely accepted that the stepsize is of great significance to the theory and numerical performance of gradient method, and the stepsize for gradient method has attracted extensive attentions. The classical steepest descent method 1 , in which the stepsize is given by $\alpha_{k}^{SD}=\arg\mathop{\min}\limits_{\alpha>0}f({x_{k}}-\alpha{g_{k}}),$ is badly affected by ill conditioning and thus converges slowly Akaike1959On . In 1988, Barzilai and Borwein 2 proposed a new gradient method (BB method), where the famous stepsize (BB stepsize) is given by

[TABLE]

Due to the simplicity and nice numerical efficiency, the BB method has received extensive attentions. The BB method has been shown to be globally 3 and R-linearly 4 convergent for any dimensional strictly convex quadratic functions. In 2021, Li and Sun SunRlinearBB2021 presented an interesting improved R-linear convergence result of the BB method. Dai et al.5 presented an efficient gradient method by adaptively choosing the BB stepsizes. Raydan6 proposed a global BB method by incorporating the nonmonotone line search (GLL line search)7 . Dai et al.8 viewed the BB stepsize from a new angle and constructed a quadratic model and a conic model to derive two step sizes for BB-like methods. Based on a fourth order conic model and some modified secant equations, Biglari and Solimanpur9 presented some BB-like methods. More BB-like methods can be found in Dai2006CBB ; Xiao2010Notes ; Nosratipour2017An ; Miladinovic2011Scalar .

In 2018, Liu et al.11 viewed the stepsize $\alpha_{k}^{BB_{1}}$ from an approximation model and introduced a new type of stepsize called approximately optimal stepsize for gradient method.

Definition 1.111 Suppose $f$ is continuously differentiable, and let $\phi_{k}(\alpha)$ be an approximation model of $f({x_{k}}-\alpha{g_{k}}).$ A positive constant ${\alpha_{k}^{AOS}}$ is called approximately optimal stepsize associated to $\phi_{k}(\alpha)$ for gradient method if $\alpha_{k}^{AOS}$ satisfies

[TABLE]

Based on (4), it is easy to obtain the following simple facts:

(i)If ${\phi_{k}}\left(\alpha\right)=f\left(x_{k}-\alpha g_{k}\right)$, then the resulted approximately optimal stepsize corresponds to Cauchy stepsize or optimal stepsize. This is the reason that we call the stepsize (4) approximately optimal stepsize.

(ii)If ${\phi_{k}}\left(\alpha\right)={f_{k}}-\alpha{\left\|{{g_{k}}}\right\|^{2}}+\frac{1}{2}{\alpha^{2}}g_{k}^{T}\left({\frac{{s_{k-1}^{T}{y_{k-1}}}}{{{{\left\|{{s_{k-1}}}\right\|}^{2}}}}I}\right){g_{k}}$, then the resulted approximately optimal stepsize corresponds to the BB stepsize $\alpha_{k}^{BB_{1}}$.

(iii)If ${\phi_{k}}\left(\alpha\right)={f_{k}}-\alpha{\left\|{{g_{k}}}\right\|^{2}}+\frac{1}{2}{\alpha^{2}}g_{k}^{T}\left(\dfrac{1}{t}I\right){g_{k}}$, where $t>0$, then the resulted approximately optimal stepsize corresponds to the fixed stepsize $t$. In fact, for any existing stepsize $\alpha_{k}>0$, let ${\phi_{k}}\left(\alpha\right)={f_{k}}-\alpha{\left\|{{g_{k}}}\right\|^{2}}+\frac{1}{2}{\alpha^{2}}g_{k}^{T}\left(\dfrac{1}{\alpha_{k}}I\right){g_{k}}$, we can easily see that the resulted approximately optimal stepsize is exactly $\alpha_{k}$.

Therefore, all existing stepsizes for gradient methods can be regarded as approximately optimal stepsizes in this sense. Some gradient methods with approximately optimal stepsizes 12 ; Liu2018Several were proposed, and the numerical experiments in 12 ; Liu2018Several indicated that these gradient methods are very efficient. Gradient methods with approximately optimal stepsizes have illustrated powerful potentiality for unconstrained optimization.

Besides, an new and important advance for gradient method is the BBQ method BBQ-HDL . Motivated by Yuan’s stepsize Yuan-stepsize , Huang, Dai and Liu BBQ-HDL equiped the Barzilai and Borwein (BB) method with two dimensional quadratic termination property and proposed a novel stepsize for gradient method (BBQ, corresponding to Algorithm 3.1 in BBQ-HDL ) for general unconstrained optimization.

Contributions. According to Definition 1.1, it is not difficult to see that the effectiveness of approximately optimal stepsize relies heavily on the approximation model $\phi_{k}(\alpha)$. To obtain more efficient gradient methods with approximately optimal stepsizes, one should take full advantage of the properties of $f$ at $x_{k}$ to exploit suitable approximation models including quadratic models and non-quadratic models for deriving approximately optimal stepsize. Two efficient gradient methods with approximately optimal stepsizes are proposed for unconstrained optimization in this paper. In the proposed methods, if the objective function $f$ is not close to a quadratic on the line segment between $x_{k-1}$ and $x_{k}$, some regularization models are exploited to generate approximately optimal stepsizes. Otherwise, a quadratic approximation model is used to derive approximately optimal stepsize. In particular, when $s_{k-1}^{T}{y_{k-1}}\leq 0,$ some special regularization models are developed carefully. The global convergence of the proposed methods is analyzed. Some numerical results indicate that the proposed method is superior to the BBQ methodBBQ-HDL and other efficient gradient methods, and is competitive to two famous conjugate gradient software packages CGOPT (1.0) Dai2014A and CG$\_$DESCENT (5.0) Hager2005A for the 145 test problems in the CUTEr library Gould2003CUTEr , and has significant improvement over CGOPT (1.0) Dai2014A and CG$\_$DESCENT (5.0) Hager2005A for the 80 test problems mainly from Andrei2008collection . It is noted that CGOPT and CG$\_$DESCENT are widely treated as two most efficient conjugate gradient software packages.

The rest of the paper is organized as follows. In Section 2, some approximation models including regularization models and quadratic models are exploited to generate approximately optimal stepsizes for gradient methods. In Section 3, two efficient gradient methods with the approximately optimal stepsizes are described. The global convergence of the proposed methods is analyzed in Section 4. In Section 5, the numerical results are presented. Conclusion and discussion are given in the last section.

2 Derivation of Approximately Optimal Stepsizes

Based on the properties of $f$ at the current iterate $x_{k}$, some approximation models including regularization models and quadratic models are exploited carefully to derive approximately optimal stepsizes for gradient methods in the section.

As mentioned above, the effectiveness of approximately optimal stepsize relies heavily on approximation model. So we take full advantage of the properties of $f$ at $x_{k}$ to construct suitable approximation models for generating approximately optimal stepsizes. The choices of approximation models are based on the following observations.

Define

[TABLE]

According to 11 , ${\mu_{k}}$ is an important criterion for judging the degree of $f$ to approximate quadratic model. If the condition 8 ; 12

[TABLE]

where $0<{c_{1}}<{c_{2}},$ holds, then $f$ might be close to a quadratic function on the line segment between $x_{k-1}$ and $x_{k}$.

When $f$ is close to a quadratic on the line segment between ${x_{k-1}}$ and ${x_{k}},$ quadratic approximation model is preferable. However, if the objective function $f$ possesses high non-linearity, then quadratic models might not work very well14 ; 15 , so some non-quadratic approximation models should be considered. In recent years, regularization algorithms for unconstrained optimization, which are defined as the standard quadratic model plus a regularization term, have become an alternative to trust region and line search schemes16 . An adaptive regularization algorithm using cubics (ARC) was proposed by Cartis et al.16 . The trial step in ARC algorithm is probably computed by minimizing the following regularization model:

[TABLE]

where ${B_{k}}$ is a symmetric approximation to the Hessian matrix, ${\sigma_{k}}>0$ is an adaptive positive parameter which can be viewed as the reciprocal of the trust region radius. And the numerical results in 17 indicated that ARC algorithm is quite efficient. An alternative approach to compute an approximate minimizer of the cubic model has been recently proposed in 18 . In 19 , a nonmonotone cubic overestimation algorithm has been put forward, which follows the one presented in 20 . In 21 , a new algorithm has been designed by combining the regularization method with line search and nonmonotone techniques. All of this indicates that when $f$ is not close to a quadratic on the line segment between ${x_{k-1}}$ and ${x_{k}}$, regularization models might serve better than quadratic models. Based on the above observations, if $f$ is not close to a quadratic on the line segment between ${x_{k-1}}$ and ${x_{k}}$, then we consider the following regularization models

[TABLE]

where $p=3$ or 4, and ${\sigma_{k}}\left(p\right)>0$ is a regularization parameter relative to $p$, otherwise we construct quadratic models to generate approximately optimal stepsizes.

We derive the approximately optimal stepsizes for gradient methods in the following four cases.

Case I. $s_{k-1}^{T}{y_{k-1}}>0$ holds and the condition (6) does not hold.

**(i)p=3 **

In the case, the objective function $f$ might be not close to a quadratic on the line segment between ${x_{k-1}}$ and ${x_{k}}$, we thus consider the regularization model (8) with $d=-\alpha g_{k}$ and $p=3$:

[TABLE]

Given that the computational cost and storage, $B_{k}$ is generated by imposing the modified Broyden-Fletcher-Goldfarb-Shanno (BFGS) update formula22 on a scalar matrix ${D_{k}}$:

[TABLE]

where ${\overline{y}_{k-1}}={y_{k-1}}+\frac{{{{\overline{r}}_{k}}}}{{{{\left\|{{s_{k-1}}}\right\|}^{2}}}}{s_{k-1}}$ and ${\overline{r}_{k}}=3{({g_{k}}+{g_{k-1}})^{T}}{s_{k-1}}+6({f_{k-1}}-{f_{k}})$. Here we take $D_{k}$ as ${D_{k}}={\xi_{0}}\frac{{y_{k-1}^{T}{y_{k-1}}}}{{s_{k-1}^{T}{y_{k-1}}}}I$, where ${\xi_{0}}\geq 1$. Since there exists ${\mu_{1}}\in[0,1]$ such that

[TABLE]

to improve the numerical performance we restrict ${\overline{r}_{k}}$ as

[TABLE]

where $0<{\xi_{1}}<0.1.$

It is not difficult to obtain the following lemma.

Lemma 2.1. * Suppose that $s_{k-1}^{T}{y_{k-1}}>0.$ Then $s_{k-1}^{T}{\overline{y}_{k-1}}>0$ and $B_{k}$ is symmetric and positive definite.*

By imposing $\frac{{d{\phi_{11}}}}{{d\alpha}}=0$, we obtain the equation: $-g_{k}^{T}{g_{k}}+\alpha g_{k}^{T}{B_{k}}{g_{k}}+{\alpha^{2}}{\sigma_{k}}\left(3\right){\left\|{{g_{k}}}\right\|^{3}}=0.$ Since

[TABLE]

by solving the above equation we can obtain the approximately optimal stepsize

[TABLE]

where $B_{k}$ is given by (10) with (12).

It is observed by numerical experiments that the bound $\left[{\alpha_{k}^{B{B_{2}}},\alpha_{k}^{B{B_{1}}}}\right]$ for $\bar{\alpha}_{k}^{AOS(11)}$ is very preferable. Therefore, if $s_{k-1}^{T}{y_{k-1}}>0$ and the condition (6) does not hold, then we take the following truncated approximately optimal stepsize

[TABLE]

for gradient method.

(ii)p=4

We consider the regularization model (8) with $d=-\alpha g_{k}$ and $p=4$:

[TABLE]

where $B_{k}$ is given (10) with (12) for the sake of simplicity.

By imposing $\frac{{d{\phi_{12}}}}{{d\alpha}}=0$, we get the equation $-g_{k}^{T}{g_{k}}+\alpha g_{k}^{T}{B_{k}}{g_{k}}+{\alpha^{3}}{\sigma_{k}}(4){\left\|{{g_{k}}}\right\|^{4}}=0.$ Since

[TABLE]

the above equation only has a real root and two imaginary roots, and thus the approximately optimal stepsize is the real root:

[TABLE]

Similar to the case of $p=3$, we also impose the bound $\left[{\alpha_{k}^{B{B_{2}}},\alpha_{k}^{B{B_{1}}}}\right]$ for $\bar{\alpha}_{k}^{AOS(12)}$. Therefore, if $s_{k-1}^{T}{y_{k-1}}>0$ holds and the condition (6) does not hold, then we take the following truncated approximately optimal stepsize

[TABLE]

for the gradient method.

The choice of regularization parameter in the regularization model

When the regularization models are applied, the regularization parameter $\sigma_{k}(p)$ should be determined properly. The regularization parameter is significant to the effectiveness of regularization models. However, it is universally acknowledged that it is challenging to determine a proper regularization parameter $\sigma_{k}\left(p\right)$. Some ways Cartis2011Adaptive ; Gould2012Updating were developed to determine the regularization parameter $\sigma_{k}\left(p\right)$, including the interpolation condition and the trust-region strategy. Here we use the interpolation condition to determine the regularization parameter $\sigma_{k}\left(p\right)$:

[TABLE]

which implies that

[TABLE]

where ${{\overline{r}}_{k}}$ is given by (12) and $p=3\;\text{or}\;4$. To improve the numerical performance and make it to be positive, we take the following truncated form:

[TABLE]

where $0<\sigma_{\min}<\sigma_{\max}$ and $p=3\;\text{or}\;4$.

Case II. $s_{k-1}^{T}{y_{k-1}}>0$ holds and the condition (6) holds.

In the case, the objective function $f$ might be close a quadratic on the line segment between ${x_{k-1}}$ and ${x_{k}}$, we thus consider the following quadratic approximation model:

[TABLE]

where ${B_{k}}$ is given by (10) with (12) for simplicity. By imposing $\dfrac{d\phi_{2}}{d\alpha}=0$, we can easily obtain the approximately optimal stepsize:

[TABLE]

It is also observed by numerical experiments that the bound $[\alpha_{k}^{B{B_{2}}},\alpha_{k}^{B{B_{1}}}]$ for $\bar{\alpha}_{k}^{AOS(2)}$ is very preferable. Therefore, if $s_{k-1}^{T}{y_{k-1}}>0$ holds and the condition (6) holds, then we take the truncated approximately optimal stepsize

[TABLE]

for gradient method.

Case III. $s_{k-1}^{T}{y_{k-1}}\leq 0$ holds and the condition (24) holds

When $s_{k-1}^{T}{y_{k-1}}\leq 0$, $f$ may enjoy poor properties at some neighbors of $x_{k}$, it is thus difficult to determine suitable stepsize for gradient method. In some modified BB methods8 ; 9 , the initial stepsize is usually set simply to ${\alpha_{k}}=10^{30}$ when $s_{k-1}^{T}{y_{k-1}}\leq 0$. As a result, it will cause large computational cost for seeking a suitable stepsize in a line search for gradient method.

It follows from $s_{k-1}^{T}{y_{k-1}}\leq 0$ that $0<\frac{{\left\|{{g_{k-1}}}\right\|}}{{\left\|{{g_{k}}}\right\|}}\leq 1.$ Consequently, if the following condition:

[TABLE]

where $0<{\xi_{2}}<1$ is close to 1, holds, then ${g_{k}}$ and ${g_{k-1}}$ incline to be collinear and are approximately equal. Based on the above observation, we will give a new way to estimate $g_{k}^{T}{\nabla^{2}}f({x_{k}}){g_{k}}$ in approximation model. Therefore, when $s_{k-1}^{T}{y_{k-1}}\leq 0$, we construct a regularization model to derive approximately optimal stepsizes based on the condition (24).

(i)p=3

Suppose for the moment that $f$ is twice continuously differentiable, we consider the following regularization model:

[TABLE]

When the condition (24) holds, we use $g_{k-1}^{T}{\nabla^{2}}f({x_{k}}){g_{k-1}}$ to approximate $g_{k}^{T}{\nabla^{2}}f({x_{k}}){g_{k}}$ and thus get that

[TABLE]

which gives the following approximation model:

[TABLE]

By imposing $\frac{{d{{\phi}_{31}}}}{{d\alpha}}=0$, we get the equation $-{\left\|{{g_{k}}}\right\|^{2}}+\alpha\frac{{\left|{s_{k-1}^{T}{y_{k-1}}}\right|}}{{\alpha_{k-1}^{2}}}+{\alpha^{2}}{\sigma_{k}(3)}{\left\|{{g_{k}}}\right\|^{3}}=0$. Since

[TABLE]

the above equation only has a real root and two imaginary roots, and thus the approximately optimal stepsize is the real root:

[TABLE]

(ii)p=4

Suppose for the moment that $f$ is twice continuously differentiable, we consider the following regularization model:

[TABLE]

Using (26), we get the following model:

[TABLE]

By imposing $\frac{{d{\phi_{32}}}}{{d\alpha}}=0$, we obtain the equation: ${\sigma_{k}(4)}{\left\|{{g_{k}}}\right\|^{4}}{\alpha^{3}}+\frac{{\left|{s_{k-1}^{T}{y_{k-1}}}\right|}}{{\alpha_{k-1}^{2}}}{\alpha^{2}}-{\left\|{{g_{k}}}\right\|^{2}}=0.$ Since

[TABLE]

the above equation only has a real root and two imaginary roots, and thus the approximately optimal stepsize is the real root:

[TABLE]

The choice of regularization parameter in the regularization model

Similar to Case I, we also use the interpolation condition to determine the regularization parameter $\sigma_{k}\left(p\right)$:

[TABLE]

which implies that

[TABLE]

Here $p=3\;\text{or}\;4$. To improve the numerical performance and make it to be positive, we take the following truncation form:

[TABLE]

where $0<\sigma_{\min}<\sigma_{\max}$ are the same as that in (20) and $p=3\;\text{or}\;4$.

Case IV. $s_{k-1}^{T}{y_{k-1}}\leq 0$ holds and the condition (24) does not hold

It also has been shown that if $\alpha_{k}^{\text{BB}}$ is reused in a cyclic fashion, then the convergence rate is accelerated Miladinovi2011 . It appears that the stepsize $\alpha_{k-1}$ may provide some important information for the current stepsize. As a result, we take $\alpha_{k}=\xi_{3}\alpha_{k-1}$ as the stepsize, where $\xi_{3}>0$. In actual, the stepsize can also be regarded as the approximately optimal stepsize. By taking $B_{k}=\frac{1}{\xi_{3}\alpha_{k-1}}I$, we can get the following quadratic approximation model

[TABLE]

By imposing $\frac{{d{{\phi}_{4}}}}{{d\alpha}}=0$, we obtain the approximately optimal stepsize:

[TABLE]

3 Two Efficient Gradient Methods with Approximately Optimal Stepsizes

We describe two efficient gradient methods with approximately optimal stepsizes in the section.

The famous nonmonotone line search (GLL line search) 7 was firstly incorporated into the BB method 6 . Though GLL line search works well in many cases, there are some drawbacks, for example, some good function values may be discarded, or the numerical performance depends very much on the choice of a pre-fixed memory constant. To overcome the above drawbacks, another well-known nonmonotone Armijo line search (Zhang-Hager line search) 10 was proposed by Zhang and Hager and is defined as

[TABLE]

where $0<\delta<1$,

[TABLE]

It is observed that Zhang-Hager line search 10 is usually preferable for the BB-like methods. To improve the numerical performance and obtain nice convergence, we take $\eta_{k}$ as :

[TABLE]

where $0<c<1$ and $\bmod(k,n)$ represents the residue for $k$ modulo $n$. As a result, Zhang-Hager line search 10 with (37) and the following strategy 23 :

[TABLE]

where $\alpha^{(0)}$ is approximately optimal stepsize described in Section 2 and ${\overline{\alpha}}$ is obtained by a quadratic interpolation at $x_{k}$ and ${x_{k}}-{\alpha}{g_{k}},$ is used in the proposed methods.

We describe the gradient method with approximately optimal stepsize (GM_AOS (Reg p=3)) in detail.

Remark. If “2.2 If $s_{k-1}^{T}{y_{k-1}}>0$ holds and the condition (6) does not hold, then compute ${\alpha_{k}}$ by (19) and update $\sigma_{k}$ by $\eqref{eq:sigma1}$ with $p=4$” and “2.4 If $s_{k-1}^{T}{y_{k-1}}\leq 0$ holds and the condition (24) holds, then compute ${\alpha_{k}}$ by (29) and update $\sigma_{k}$ by $\eqref{eq:sigma2}$ with $p=4$” are used to replace of 2.2 and 2.4 of Algorithm 1, relatively, then the resulting method corresponds to another gradient method with approximately optimal stepsize called GM$\_$AOS (Reg p=4). We use GM$\_$AOS (Reg) to denote either GM$\_$AOS (Reg p=3) or GM$\_$AOS (Reg p=4).

4 Convergence Analysis

In the section the global convergence of GM$\_$AOS (Reg) is analyzed under weak conditions. In the convergence analysis the following assumptions are done.

D1. $f(x)$ is continuously differentiable on $\mathbb{R}^{n}$.

D2. $f(x)$ is bounded below on $\mathbb{R}^{n}$.

D3. The gradient $g(x)$ is ** uniformly continuous** on $\mathbb{R}^{n}$.

Lemma 4.1

For $Q_{k}$ in (34) , we have ${Q_{k+1}}\leq 1+\dfrac{n}{{1-c}}$.

Proof It follows from (34) that

[TABLE]

which together with (37) suggests that

[TABLE]

where $\left\lfloor\cdot\right\rfloor$ is the floor function.

By (41) and the fact that $0<c<1$, we obtain that

[TABLE]

which completes the proof. ∎

Lemma 4.2

Suppose that D1, D2 and D3 hold. Then,

[TABLE]

Proof According to (33) and (34), we have

[TABLE]

and

[TABLE]

As a result, the inequality (42) holds. The proof is completed. ∎

The above lemma implies that the sequence $\left\{C_{k}\right\}$ is convergent.

Theorem 4.1

Suppose that D1, D2 and D3 hold, and let $\left\{x_{k}\right\}$ be the sequence generated by GM$\_$AOS (Reg). Then,

[TABLE]

Proof By (33) and (34), we obtain that

[TABLE]

which together with Lemma 4.1 implies that

[TABLE]

It then follows from Lemma 4.2 and D2 that

[TABLE]

We suppose, by way of contradiction, that there exists a subsequence $\left\{x_{k_{j}}\right\}$ such that

[TABLE]

We denote

[TABLE]

It follows from (46) that there exists a positive integer ${j_{0}}$ such that

[TABLE]

Therefore, we obtain from (45) that $\mathop{\lim}\limits_{j\to\infty}{\alpha_{{k_{j}}}}=0$ and

[TABLE]

By (40), we know that there exists $\bar{\delta}_{k_{j}}\in[0.1,0.9]$ such that

[TABLE]

Combining (49) and ${f\left({{x_{{k_{j}}}}-{\alpha_{{k_{j}}}}{g_{{k_{j}}}}}\right)}\leq{C_{{k_{j}}}}-\sigma{\alpha_{{k_{j}}}}{\left\|{{g_{{k_{j}}}}}\right\|^{2}}$, we obtain that

[TABLE]

It follows from the mean-value theorem that there exists ${w_{{k_{j}}}}\in\left[{0,1}\right]$ such that

[TABLE]

where ${u_{{k_{j}}}}={x_{{k_{j}}}}-\left[{1+{w_{{k_{j}}}}\left({1/{{\bar{\delta}}_{{k_{j}}}}-1}\right)}\right]{\alpha_{{k_{j}}}}{g_{{k_{j}}}}$. Therefore, we get that

[TABLE]

which implies that $\;{\left({{g_{{k_{j}}}}-g\left({{u_{{k_{j}}}}}\right)}\right)^{T}}\frac{{{g_{{k_{j}}}}}}{{\left\|{{g_{{k_{j}}}}}\right\|}}>\left({1-\sigma}\right)\left\|{{g_{{k_{j}}}}}\right\|.$ According to (47), we know that

[TABLE]

According to (45), (48) and $1\leq{1+{w_{{k_{j}}}}\left({1/{{\bar{\delta}}_{{k_{j}}}}-1}\right)}\leq 10$, we know that

[TABLE]

Since the gradient $g$ is uniformly continuous, for $\frac{{\left({1-\sigma}\right)\bar{\varepsilon}}}{2}$, one can find $\zeta>0$ depending only on $\frac{{\left({1-\sigma}\right)\bar{\varepsilon}}}{2}$ such that $\left\|{{g_{{k_{j}}}}-g\left({{u_{{k_{j}}}}}\right)}\right\|\leq\frac{{\left({1-\sigma}\right)}}{2}\bar{\varepsilon}$ holds whenever $\left\|{{x_{{k_{j}}}}-{u_{{k_{j}}}}}\right\|=\left[{{w_{{k_{j}}}}\left({1/{{\bar{\delta}}_{{k_{j}}}}-1}\right)+1}\right]{\alpha_{{k_{j}}}}\left\|{{g_{{k_{j}}}}}\right\|<\zeta.$ By (51), we know that there exists an integer $j_{1}>0$ such that

[TABLE]

holds for any $j>j_{1}$. As a result, $\left\|{{g_{{k_{j}}}}-g\left({{u_{{k_{j}}}}}\right)}\right\|\leq\frac{{\left({1-\sigma}\right)}}{2}\bar{\varepsilon}$ holds for any $j>j_{1}$, which contradicts (50) when $j\geq\max\left\{j_{0},j_{1}\right\}$. Therefore, there no exists a subsequence $\left\{x_{k_{j}}\right\}$ satisfying (46), which implies (43). The proof is completed. ∎

5 Numerical Experiments

We compare GM$\_$AOS (Reg) with GM_AOS (1.2) Liu2018Several , the BB method, CGOPT (1.0) Dai2014A , CG_DESCENT (5.0) Hager2005A and BBQ method BBQ-HDL (corresponding to Algorithm 3.1 in BBQ-HDL ) in the section. It is widely accepted that CGOPT Dai2014A and CG_DESCENT Hager2005A are the two most famous and efficient conjugate gradient software packages. The codes of the BB method, GM$\_$AOS (1.2) Liu2018Several and GM$\_$AOS (Reg) were implemented by C language, and the C codes of CG$\_$DESCENT (5.0) and CGOPT (1.0) can be downloaded from http://users.clas.ufl.edu/hager/papers/Software and http://coa.amss.ac.cn/wordpress/?page_id=21, respectively. The matlab code of BBQ can be found in Dai’s homepage: http://lsec.cc.ac.cn/~dyh/software.html. The C code of GM$\_$AOS (Reg) and the detailed numerical result will be available in our website finally. Two test sets were used, which include the 145 test problems in the CUTEr library Gould2003CUTEr (we call it CUTEr145 for short) and the 80 test problems mainly from Andrei2008collection collected by Andrei (we call it Andr80 for short), respectively. The two test sets can be found in Hager’s website http://users.clas.ufl.edu/hager/papers/CG/results6.0.txt and Andrei’s homepage http://camo.ici.ro/neculai/AHYBRIDM, respectively. The dimensions of the test problem in the test set CUTEr145 are default and the dimension of each test problems in the test set Andr80 is set to 10,000. All numerical experiments were done in Ubuntu 10.04 LTS in a VMware Workstation 10.0 installed in Win 10.

We choose the following parameters for GM$\_$AOS (Reg): $\varepsilon={10^{-6}},$ ${\alpha_{\min}}={10^{-30}},$ ${\alpha_{\max}}={10^{30}},$ ${\xi_{0}}=1.07,$ ${\xi_{1}}=5\times 10^{-5}/3,$ $\xi_{2}=0.8,\;$ $\xi_{3}=5,\;$ $\sigma_{\min}=10^{-30},\;\sigma_{\max}={10^{3}}$, $\delta={10^{-4}},\;{c_{1}}={10^{-9}},$ ${c_{2}}=10^{-7}$, $c=0.99$ and

[TABLE]

GM_AOS (1.2)Liu2018Several and the BB method used the same line search as that in GM$\_$AOS (Reg). CGOPT(1.0), CG$\_$DESCENT (5.0) and BBQ used all default settings of parameters but the stopping conditions. Each test method is terminated if ${\left\|{{g_{k}}}\right\|_{\infty}}\leq{10^{-6}}$ or the iterations exceeds 140,000.

The performance profiles introduced by Dolan and Moré25 were used to display the performance of these methods. In the following figures, “$N_{iter}$”, “$N_{f}$”, “$N_{g}$” and “$T_{cpu}$” represent the number of iterations, the number of function evaluations, the number of gradient evaluations and CPU time (s), respectively.

We first compare GM$\_$AOS (Reg p=3) with GM$\_$AOS (Reg p=4) on the test set CUTEr145, and use the better one to compare with other test methods. As shown in Figs. 2-4, we see that GM$\_$AOS (Reg p=3) performs better than GM$\_$AOS (Reg p=4) in term of $N_{iter}$, $N_{f}$, $N_{g}$ and $T_{cpu}$. So we select GM$\_$AOS (Reg p=3) to compare with other test methods in the following numerical experiments.

The following numerical experiments are divided into four groups.

In the first group of the numerical experiments, we compare the performance of GM$\_$AOS (Reg p=3) with that of GM$\_$AOS (1.2) Liu2018Several and the BB method on the test set CUTEr145. Figs. 6-8 present the performance profiles on the test set CUTEr145. As shown in Figs. 6-8, we can observe that GM$\_$AOS (Reg p=3) performs better than GM$\_$AOS (1.2) and is superior much to the BB method, and GM$\_$AOS (1.2) outperforms the BB method. The first group of the numerical experiments indicates that the approximately optimal stepsize is extremely efficient.

In the third group of the numerical experiments, we compare the numerical performance of GM$\_$AOS (Reg p=3) and the BBQ method on the test set CUTEr145. In the numerical experiment, we do not compare the performance about the running time due to the fact that the BBQ method was implemented by Matlab code and GM$\_$AOS (Reg p=3) was implented by C code. As shown in Fig. 10, 10 and 11, we can observed that GM$\_$AOS (Reg p=3) is superior to the BBQ method for the test set CUTEr145 in term of the number of iteration, the number of funciton evaluation and the number of gradient evaluation, while the BBQ method has been regarded as the import advance for gradient method.

In the second group of the numerical experiments, we compare the performance of GM_AOS (Reg p=3) with that of CGOPT (1.0) on the two test sets CUTEr145 and Andr80. Figs. 13-15 present the performance profiles on the test set CUTEr145. As shown in Fig. 13, we see that GM_AOS (Reg p=3) performs much better CGOPT (1.0) in term of $N_{f}$, since GM_AOS (Reg p=3) solves successfully about 79$\%$ test problems with the least function evaluations, while the percentage of CGOPT (1.0) is only about 38$\%$. Fig. 13 indicates that GM_AOS (Reg p=3) is at a disadvantage over CGOPT (1.0) in term of $N_{g}$, and Fig. 15 shows that GM_AOS (Reg p=3) outperforms slightly CGOPT (1.0) in term of $N_{f}+3N_{g}$ HagerLMCGDESCENT . We can observe from Fig. 15 that GM_AOS (Reg p=3) is as fast as CGOPT (1.0). Figs. 17-19 present the performance profiles on the test set Andr80. As shown in Figs. 17-19, we observe that GM_AOS (Reg p=3) illustrates huge advantage over CGOPT (1.0) on the test set Andr80. The second group of the numerical experiments indicates that GM_AOS (Reg p=3) is competitive to CGOPT (1.0) on the test set CUTEr145, and has a significant improvement over CGOPT (1.0) on the test set Andr80.

In the fourth group of the numerical experiments, we compare the performance of GM_AOS (Reg p=3) with that of CG_DESCENT (5.0) on the two test sets CUTEr145 and Andr80. Figs. 21-23 present the performance profiles on the test set CUTEr145. As shown in Fig. 21, we see that GM_AOS (Reg p=3) performs better than CG_DESCENT (5.0) in term of $N_{f}$, since GM_AOS (Reg p=3) solves successfully about 65$\%$ test problems with the least function evaluations, while the percentage of CG_DESCENT (5.0) is only about 39$\%$. Fig. 21 shows that GM_AOS (Reg p=3) is at a disadvantage over than CG_DESCENT (5.0) in term of $N_{g}$, and Fig. 23 indicates that GM_AOS (Reg p=3) outperforms slightly CG_DESCENT (5.0) in term of $N_{f}+3N_{g}$ HagerLMCGDESCENT . We can observe from Fig. 23 that GM_AOS (Reg p=3) is as fast as CG_DESCENT (5.0). Figs. 25-27 present the performance profiles on the test set Andr80. As shown in Figs. 25-27, we see that GM_AOS (Reg p=3) is at a little disadvantage over CG_DESCENT (5.0) in term of $N_{iter}$, and has a significant performance boost over CG_DESCENT (5.0) in term of $N_{f}$ and $N_{g}$. We also can see that GM_AOS (Reg p=3) is faster much than CG_DESCENT (5.0). The third group of the numerical experiments indicates that GM_AOS (Reg p=3) is competitive to CG_DESCENT (5.0) on the test set CUTEr145, and has a significant improvement over CG_DESCENT (5.0) on the test set Andr80.

Table 1. The number of test problems

As for the reasons for the suprising numerical effect of GM_AOS (Reg p=3), we think that they lie in two aspects: (i)The approximately optimal stepsize is generated by the approximation models including regularization models and quadratic models at the current iterate $x_{k}$, which implies that it is incorporated

properly into more second order or high order information of the objective function. (ii)The approximately optimal stepsize can readily satisfy Zhang-Hager line search directly in most cases compared to other stepsizes in gradient method, which implies that it require less much function evaluations and thus save much computational cost. This can be observed in Figs. 6, 13, 17, 21 and 25. More results can be seen in Table LABEL:table1. In Table LABEL:table1, $N_{\text{linsear}}$ represents the times that the stepsize is updated by (40) during all iterations of solving a test problem, namely, the times that Zhang-Hager line search is invoked during all iterations of solving a test problem. $N_{\text{linsear}}=0$ indicates the initial stepsize (approximately optimal stepsize or BB stepsize) satisfies Zhang-Hager line search (33) directly at all iterations and thus **Zhang-Hager line search ** is not invoked at all. As shown in Table LABEL:table1, we can see that there are 68 (out of 145) problems for which Zhang-Hager line search is not invoked at all during the solving process, while the number for the BB method is only 41, and there are 90 (out of 145) problems for each of which the times that Zhang-Hager line search is invoked is less than or equal to 3, while the number for the BB method is only 50. We also can see that there are 51 (out of 80) problems for which Zhang-Hager line search is not invoked at all during the solving process. Table LABEL:table1 indicates that the approximately optimal stepsize in GM_AOS (Reg p=3) is easier much to meet Zhang-Hager line search (33) directly.

6 Conclusion and discussion

In this paper, we present two efficient gradient methods with approximately optimal stepsize for unconstrained optimization. In the proposed method, some approximation models including regularization models and quadratic models are exploited carefully to derive approximately optimal stepsize. The convergence of the proposed methods is analyzed. Extensive numerical results indicates that the proposed method GM$\_$AOS (Reg p=3) is superior to the BBQ method and other efficient gradient methods, and is competitive to two quite efficient and well-known conjugate gradient software packages CG$\_$DESCENT (5.0) and CGOPT (1.0) on the 145 test problems in the CUTEr library, has significant improvement over CG$\_$DESCENT (5.0) and CGOPT (1.0) on the 80 test problems collected by Andrei. As for the reason that GM$\_$AOS (Reg p=3) has so important improvement over CG$\_$DESCENT (5.0) and CGOPT (1.0) on Andr80 and is only competitive to CG$\_$DESCENT (5.0) and CGOPT (1.0) on CUTEr145, I think that it lies mainly in that most test problems in CUTEr145 is relatively difficult to solve compared to the test problems in Andr80. It seems that one can draw the following conclusion: Gradient methods with approximately optimal stepsize are sufficient for those test problems that are not very ill-conditioned.

Given that the facts: (i)the search direction $-g_{k}$ has low storage; (ii)the approximately optimal stepsize can be easily computed; (iii)the nonmonotone Armijo line search used can be easily implemented; (iv)the numerical effect is surprisingly nice, the gradient methods with approximately optimal stepsizes can become strong candidates for large scale unconstrained optimization and has potential in constrained optimization and some fields such as machine learning.

Though gradient methods with approximately optimal stepsize is surprisingly efficient, there are still some questions under investigation:

(i)Does gradient method with approximately optimal stepsize based on quadratic approximation model (21) possess Q-linear convergence for convex quadratic minimization? If yes, what conditions should be imposed on the distance $\left\|B_{k}-A\right\|$ ? Here $A$ is the Hessian matrix for strictly convex quadratic function.

(ii)It will be an interesting research for combining approximately optimal stepsize wih Cauchy stepsize in convex quadratic minimization. How should one combine approximately optimal stepsize with Cauchy stepsize for obtaining better convergence rate in convex quadratic minimization?

(iii)Can the type of gradient method with approximately optimal stepsize possess local R-linear convergence or better convergence rate when it is applied to general unconstrained optimization?

(iv)There are still a large room for improving numerical performance of gradient methods with approximately optimal stepsizes by exploiting other adaptive appriximation models based on the properties of the objective function.

Acknowledgements.

We would like to thank Professors Hager W. W. and Zhang, H. C. for their C code of CG$\_$DESCENT, thank Professor Dai Y. H. and Dr. Kou C. X. for their C code of CGOPT (1.0). This research is supported by National Science Foundation of China (No. 11901561) and Guangxi Natural Science Foundation (2018GXNSFBA281180).