An Inexact Newton-like conditional gradient method for constrained nonlinear systems

TL;DR

This paper introduces an inexact Newton-like conditional gradient method for solving constrained nonlinear systems, establishing local convergence and rate under general conditions, with applications to Holder-like and Smale functions, supported by numerical experiments.

Contribution

It presents a novel inexact Newton-like conditional gradient method with convergence analysis for constrained nonlinear systems, applicable to a broad class of functions including analytic ones.

Findings

Method converges locally under general conditions.

Applicable to functions with Holder-like derivatives and Smale conditions.

Numerical experiments demonstrate effectiveness on medium and large problems.

Abstract

In this paper, we propose an inexact Newton-like conditional gradient method for solving constrained systems of nonlinear equations. The local convergence of the new method as well as results on its rate are established by using a general majorant condition. Two applications of such condition are provided: one is for functions whose the derivative satisfies Holder-like condition and the other is for functions that satisfies a Smale condition, which includes a substantial class of analytic functions. Some preliminaries numerical experiments illustrating the applicability of the proposed method for medium and large problems are also presented.

Tables

Table 1. Table 1: Test problems

Problem	Name and souce	n	Box
Pb 1	Chandrasekhar’s H-equation $c=0.99$ [21]	400	$[0,5]$
Pb 2	Discrete boundary value function [32, Problem 28]	500	[-100,100]
Pb 3	Troesch [27, Problem 4.21]	500	[-1,1]
Pb 4	Discrete integral [32, Problem 29]	1000	[-10,10]
Pb 5	Trigexp 1 [27, Problem 4.4]	1000	[-100,100]
Pb 6	Problem 74 [28]	1000	[0,10]
Pb 7	Problem 77 [28]	2000	[0,10]
Pb 8	Function 15 [7]	2000	[-10,0]
Pb 9	Tridiagonal exponential [27, Problem 4.18]	2000	[$e^{-1}$, $e$]
Pb 10	Trigonometric function [7, Problem 8]	2000	[5,15]
Pb 11	Zero Jacobian function [7, Problem 19]	2000	[0,10]
Pb 12	Trigonometric system [27, Problem 4.3]	5000	[$\pi$, $2\pi$]
Pb 13	Five diagonal [27, Problem 4.8]	5000	[1,$\infty$]
Pb 14	Seven diagonal [27, Problem 4.9]	5000	[0,$\infty$]
Pb 15	Countercurrent reactors [27, Problem 4.1]	10000	[-1,10]

Table 2. Table 2: Performance of the FD-INL-CondG, BSU-INL-CondG and BPU-INL-CondG methods and CoDoSol

		FD-INL-CondG		BSU-INL-CondG		BPU-INL-CondG		CoDoSol
Problem	$\gamma$	Iter	${\Vert F\Vert }_{\infty }$	Iter	${\Vert F\Vert }_{\infty }$	Iter	${\Vert F\Vert }_{\infty }$	Iter	${\Vert F\Vert }_{\infty }$
Pb 1	1	5	$1.39e$-$11$	5	$9.22e$-$7$	5	$6.99e$-$7$	7	$1.74e$-$11$
	2	6	$4.07e$-$9$	7	$3.58e$-$8$	7	$2.42e$-$8$	7	$8.25e$-$7$
	3	5	$1.71e$-$7$	7	$5.88e$-$11$	7	$1.13e$-$9$	*
Pb 2	1	9	$3.30e$-$8$	12	$4.78e$-$07$	12	$3.10e$-$8$	14	$3.67e$-$10$
	2	1	$2.59e$-$7$	1	$2.59e$-$7$	1	$2.59e$-$7$	2	$2.19e$-$9$
	3	9	$2.22e$-$7$	12	$2.92e$-$7$	12	$2.06e$-$8$	14	$9.18e$-$9$
Pb 3	1	6	$2.18e$-$7$	13	$7.99e$-$7$	8	$3.60e$-$8$	9	$1.91e$-$9$
	2	7	$7.71e$-$8$	11	$2.35e$-$7$	9	$6.65e$-$8$	6	$5.58e$-$9$
	3	6	$2.18e$-$7$	9	$2.02e$-$7$	8	$4.03e$-$8$	7	$1.56e$-$7$
Pb 4	1	5	$2.66e$-$10$	7	$9.12e$-$12$	7	$2.62e$-$12$	9	$1.34e$-$10$
	2	3	$2.72e$-$11$	3	$2.00e$-$9$	3	$2.00e$-$9$	3	$1.04e$-$7$
	3	6	$4.64e$-$11$	7	$2.23e$-$8$	*		9	$3.49e$-$8$
Pb 5	1	20	$2.60e$-$8$	169	$5.46e$-$9$	28	$2.68e$-$8$	21	$9.01e$-$7$
	2	9	$3.40e$-$11$	13	$1.05e$-$7$	12	$8.00e$-$10$	10	$3.46e$-$9$
	3	13	$5.21e$-$9$	19	$3.64e$-$10$	17	$2.78e$-$8$	23	$2.41e$-$8$
Pb 6	1	5	$5.72e$-$7$	7	$1.67e$-$7$	7	$2.01e$-$8$	7	$3.39e$-$7$
	2	13	$2.72e$-$11$	42	$1.29e$-$8$	69	$2.91e$-$9$	9	$2.32e$-$8$
	3	9	$2.22e$-$8$	25	$1.73e$-$7$	12	$2.27e$-$8$	12	$3.07e$-$8$
Pb 7	1	6	$2.19e$-$10$	7	$9.34e$-$9$	7	$9.95e$-$9$	9	$4.22e$-$7$
	2	8	$6.44e$-$11$	10	$1.34e$-$9$	10	$3.26e$-$8$	12	$3.45e$-$9$
	3	9	$7.83e$-$11$	11	$8.40e$-$8$	12	$1.40e$-$11$	13	$2.73e$-$7$
Pb 8	1	7	$4.75e$-$10$	11	$6.48e$-$8$	9	$4.36e$-$8$	13	$6.70e$-$11$
	2	6	$3.00e$-$7$	10	$1.36e$-$7$	8	$2.61e$-$7$	12	$1.75e$-$7$
	3	6	$2.88e$-$13$	8	$9.36e$-$8$	7	$4.29e$-$7$	11	$1.93e$-$9$
Pb 9	1	2	$4.84e$-$14$	2	$4.84e$-$14$	2	$4.84e$-$14$	8	$6.03e$-$14$
	2	2	$1.39e$-$13$	2	$1.39e$-$13$	2	$1.39e$-$13$	7	$6.23e$-$14$
	3	2	$2.98e$-$14$	2	$2.98e$-$14$	2	$2.98e$-$14$	7	$3.96e$-$14$
Pb 10	1	7	$6.88e$-$10$	9	$4.46e$-$10$	16	$4.74e$-$9$	10	$2.12e$-$11$
	2	3	$1.45e$-$7$	4	$2.22e$-$10$	4	$1.62e$-$10$	4	$3.36e$-$8$
	3	10	$5.65e$-$7$	14	$7.83e$-$8$	73	$6.41e$-$9$	12	$1.28e$-$8$
Pb 11	1	17	$7.28e$-$7$	22	$9.58e$-$7$	27	$7.51e$-$7$	22	$2.56e$-$7$
	2	18	$7.28e$-$7$	24	$6.13e$-$7$	28	$9.21e$-$7$	23	$5.56e$-$7$
	3	19	$4.09e$-$7$	25	$5.39e$-$7$	30	$9.31e$-$7$	24	$4.60e$-$7$
Pb 12	1	*		*		*		18	$2.60e$-$8$
	2	*		*		*		17	$9.00e$-$8$
	3	*		*		*		16	$4.94e$-$9$
Pb 13	1	8	$0.00e$+$0$	7	$0.00e$+$0$	5	$0.00e$+$0$	17	$8.72e$-$10$
	2	12	$0.00e$+$0$	22	$0.00e$+$0$	*		*
	3	16	$0.00e$+$0$	*		7	$0.00e$+$0$	*
Pb 14	0	4	$3.64e$-$10$	5	$1.65e$-$7$	5	$2.47e$-$8$	4	$4.13e$-$10$
	1	12	$3.69e$-$7$	17	$1.32e$-$9$	17	$2.03e$-$11$	17	$1.29e$-$8$
	2	20	$1.09e$-$12$	28	$6.93e$-$10$	*		25	$1.29e$-$11$
Pb 15	1	11	$6.82e$-$9$	*		21	$3.79e$-$7$	17	$2.25e$-$9$
	2	12	$3.06e$-$8$	19	$6.05e$-$7$	*		19	$2.06e$-$9$
	3	13	$5.98e$-$9$	20	$3.24e$-$7$	*		20	$2.10e$-$9$

Equations

$$F(x)=0,x\in C,$$

$$M_k{s}_k=-F(x_k)+r_k,y_k=x_k+s_k.$$

$$x_{k+1}=\text{mbox}CondG(y_k,x_k,{\theta }_k\Vert s_k{\Vert }^2).$$

$$g_t^{\ast }=\underset{u\in C}{\min }\{⟨z_t-y,u-z_t⟩\}.$$

$${\alpha }_t:=\min \left\{1,\frac{-g_t^{\ast }}{\Vert u_t-z_t{\Vert }^2}\right\},z_{t+1}=z_t+{\alpha }_t(u_t-z_t).$$

$$\Vert F^{\prime }(x_{\ast }{)}^{-1}\left[F^{\prime }(x)-F^{\prime }(x_{\ast }+\tau (x-x_{\ast }))\right]\Vert \le f^{\prime }\left(\Vert x-x_{\ast }\Vert \right)-f^{\prime }\left(\tau \Vert x-x_{\ast }\Vert \right),$$

$$0\le \vartheta <1,0\le {\omega }_2<{\omega }_1,{\omega }_1\vartheta +{\omega }_2<1,\lambda \in \left[0,(1-{\omega }_2-{\omega }_1\vartheta )/({\omega }_1(1+\vartheta ))\right).$$

$$\nu :=\sup \{t\in [0,R):f^{\prime }(t)<0\},$$

$$\rho :=\sup \left\{\delta \in (0,\nu ):{\omega }_1(1+\vartheta )(1+\lambda )\left(\frac{f(t)}{t{f}^{\prime }(t)}-1\right)+{\omega }_1[(1+\vartheta )\lambda +\vartheta ]+{\omega }_2<1,t\in (0,\delta )\right\},$$

$$\sigma :=\min \{\kappa ,\rho \}.$$

$$\Vert {M_k}^{-1}{F}^{\prime }(x_k)\Vert \le {\omega }_1,\Vert {M_k}^{-1}{F}^{\prime }(x_k)-I\Vert \le {\omega }_2,k=0,1,\dots ,$$

$$\Vert P_k{r}_k\Vert \le {\eta }_k\Vert P_{k}F(x_k)\Vert ,0\le {\eta }_k\text{mbox}cond(P_k{F}^{\prime }(x_k))\le \vartheta ,k=0,1,\dots ,$$

$$\Vert x_{k+1}-x_{\ast }\Vert <\Vert x_k-x_{\ast }\Vert ,\underset{k\to \infty }{\mathrm{limsup}}\frac{\Vert x_{k+1}-x_{\ast }\Vert }{\Vert x_k-x_{\ast }\Vert }\le {\omega }_1[(1+\vartheta )\sqrt{2\tilde{\theta }}+\vartheta ]+{\omega }_2,$$

$$\begin{gathered} \Vert x_{k+1}-x_{\ast }\Vert \le {\omega }_1(1+\vartheta )(1+\lambda )\left(\frac{f(\Vert x_0-x_{\ast }\Vert )}{f^{\prime }(\Vert x_0-x_{\ast }\Vert )}-\Vert x_0-x_{\ast }\Vert \right){\left(\frac{\Vert x_k-x_{\ast }\Vert }{\Vert x_0-x_{\ast }\Vert }\right)}^{p+1} \\ +({\omega }_1[(1+\vartheta )\lambda +\vartheta ]+{\omega }_2)\Vert x_k-x_{\ast }\Vert . \end{gathered}$$

$$\Vert F^{\prime }(x_{\ast }{)}^{-1}[F^{\prime }(x)-F^{\prime }(x_{\ast }+\tau (x-x_{\ast }))]\Vert \le K(1-{\tau }^p)\Vert x-x_{\ast }{\Vert }^p,\tau \in [0,1],x\in B(x_{\ast },\kappa ).$$

$$\bar{\sigma }:=\min \left\{\kappa ,{\left[\frac{(1-{\omega }_1[(1+\vartheta )\lambda +\vartheta ]-{\omega }_2)(p+1)}{K(p-{\omega }_1[(1+\vartheta )\lambda +\vartheta -p]-{\omega }_2(p+1)+1)}\right]}^{1/p}\right\}.$$

$$\Vert {M_k}^{-1}{F}^{\prime }(x_k)\Vert \le {\omega }_1,\Vert {M_k}^{-1}{F}^{\prime }(x_k)-I\Vert \le {\omega }_2,k=0,1,\dots ,$$

$$\Vert P_k{r}_k\Vert \le {\eta }_k\Vert P_{k}F(x_k)\Vert ,0\le {\eta }_k\text{mbox}cond(P_k{F}^{\prime }(x_k))\le \vartheta ,k=0,1,\dots .$$

$$\Vert x_{k+1}-x_{\ast }\Vert <\Vert x_k-x_{\ast }\Vert ,\underset{k\to \infty }{\mathrm{limsup}}\frac{\Vert x_{k+1}-x_{\ast }\Vert }{\Vert x_k-x_{\ast }\Vert }\le {\omega }_1[(1+\vartheta )\sqrt{2\tilde{\theta }}+\vartheta ]+{\omega }_2,$$

$$\Vert x_{k+1}-x_{\ast }\Vert \le \frac{{\omega }_1(1+\vartheta )(1+\lambda )pK}{(p+1)[1-K\Vert x_0-x_{\ast }{\Vert }^p]}\Vert x_k-x_{\ast }{\Vert }^{p+1}+({\omega }_1[(1+\vartheta )\lambda +\vartheta ]+{\omega }_2)\Vert x_k-x_{\ast }\Vert ,k\ge 0,$$

$$\rho ={\left[\frac{(1-{\omega }_1[(1+\vartheta )\lambda +\vartheta ]-{\omega }_2)(p+1)}{K(p-{\omega }_1[(1+\vartheta )\lambda +\vartheta -p]-{\omega }_2(p+1)+1)}\right]}^{1/p}<\nu ={\left[\frac{1}{K}\right]}^{1/p},$$

$$\gamma :=\underset{n>1}{\sup }{\Vert \frac{F^{\prime }(x_{\ast }{)}^{-1}{F}^{(n)}(x_{\ast })}{n!}\Vert }^{1/(n-1)}<+\infty .$$

$$\bar{\sigma }:=\min \left\{\kappa ,\frac{a-\sqrt{a^2-8b^2}}{4\gamma b}\right\}.$$

$$\Vert {M_k}^{-1}{F}^{\prime }(x_k)\Vert \le {\omega }_1,\Vert {M_k}^{-1}{F}^{\prime }(x_k)-I\Vert \le {\omega }_2,k=0,1,\dots ,$$

$$\Vert P_k{r}_k\Vert \le {\eta }_k\Vert P_{k}F(x_k)\Vert ,0\le {\eta }_k\text{mbox}cond(P_k{F}^{\prime }(x_k))\le \vartheta ,k=0,1,\dots .$$

$$\Vert x_{k+1}-x_{\ast }\Vert <\Vert x_k-x_{\ast }\Vert ,\underset{k\to \infty }{\mathrm{limsup}}\frac{\Vert x_{k+1}-x_{\ast }\Vert }{\Vert x_k-x_{\ast }\Vert }\le {\omega }_1[(1+\vartheta )\sqrt{2\tilde{\theta }}+\vartheta ]+{\omega }_2,$$

$$\Vert x_{k+1}-x_{\ast }\Vert \le \frac{{\omega }_1(1+\vartheta )(1+\lambda )\gamma }{2(1-\gamma \Vert x_0-x_{\ast }\Vert {)}^2-1}\Vert x_k-x_{\ast }{\Vert }^2+({\omega }_1[(1+\vartheta )\lambda +\vartheta ]+{\omega }_2)\Vert x_k-x_{\ast }\Vert ,k\ge 0,$$

$$\rho =\frac{a-\sqrt{a^2-8b^2}}{4\gamma b},\nu =\frac{\sqrt{2}-1}{\sqrt{2}\gamma },\rho <\nu <\frac{1}{\gamma },$$

$$n_f(t)=t-f(t)/f^{\prime }(t)$$

$$n_f(t)<0\text{mbox}forallt\in (0,\nu )\text{mbox}and\underset{t\downarrow 0}{\lim }\frac{|n_f(t)|}{t}=0.$$

Full text

Keywords: constrained nonlinear systems; inexact Newton-like method; conditional gradient method; local convergence.

1 Introduction

Let $\Omega\subset\mathbb{R}^{n}$ be an open set, and $F:\Omega\to\mathbb{R}^{n}$ be a continuously differentiable nonlinear function. Consider the following constrained system of nonlinear equations

[TABLE]

where $C\subset\Omega$ is a nonempty convex compact set. Constrained nonlinear systems such as (1) appear frequently in many important areas, for instance, engineering, chemistry and economy. Due to this fact, the numerical solutions of problem (1) have been the object of intense research in the last years and, consequently, different methods have been proposed in the literature. Many of them are combinations of Newton methods for solving the unconstrained systems with some strategies taking into account the constraint set. Strategies based on projections, trust region, active set and gradient methods have been used; see, e.g., [1, 3, 9, 19, 20, 22, 23, 30, 31, 34, 38, 39, 40].

Recently, paper [15] proposed and established a local convergence analysis of a Newton conditional gradient (Newton-CondG) method for solving (1). Basically, this method consists of computing the Newton step and after applying a conditional gradient (CondG) procedure in order to get the Newton iterative back to the feasible set. It is important to point out that the CondG method, also known as the Frank-Wolfe method, is historically known as one of the earliest first methods for solving convex constrained optimization problems, see [8, 11]. The CondG method and its variants require, at each iteration, to minimizing a linear function over the constraint set, which, in general, is significantly simpler than the projection step arising in many proximal-gradient methods. Indeed, projection problems can be computationally hard for some high-dimensional problems. For instance, in large-scale semidefinite programming, each projection subproblem of the proximal-gradient methods requires to obtain the complete eigenvalue decomposition of a large matrix while each subproblem of the CondG methods requires to compute the leading singular vector of such a matrix. The latter requirement is less computationally expensive (see, for example, [18] for more details). Moreover, depending on the application, linear optimization oracles may provide solutions with specic characteristics leading to important properties such as sparsity and low-rank; see, e.g., [12, 18] for a discussion on this subject. Due to these advantages and others, the CondG method have again received much attention, see for instances [12, 16, 18, 25, 29].

It is well-know that implementations of the Newton method for medium- or large-scale problems may be expensive and difficult due to the necessity to compute all the elements of the Jacobian matrix of $F$, as well as, the exact solution of a linear system for each iteration. For this reason, the main goal of this work is to present an extension of the Newton-CondG method in which the inexact Newton-like method is considered instead of standard Newton method. In each step of this new method, the solution of the linear system and Jacobian matrix can be computed in approximate way; see the INL-CondG method in Section  2 and comments following it. From the theoretical viewpoint, we present a local convergence analysis of the proposed method under a majorant condition. The advantage of using a general condition such as majorant condition in the analyses of Newton methods lies in the fact that it allows to study them in a unified way. Thus, two applications of majorant condition are provided: one is for functions whose the derivative satisfies Hölder-like condition and the other is for functions that satisfies a Smale condition, which includes a substantial class of analytic functions. From the applicability viewpoint, we report some preliminaries numerical experiments of the proposed method for medium and large problems and compare its performance with the constrained dogleg method [2].

This paper is organized as follows. Subsection 1.1 presents some notation and basic assumptions. Section 2 describes the inexact Newton-like conditional gradient method and presents its convergence theorem whose proof is postponed to Section 3. Two applications of the main convergence theorem are also present in Section 2. Section 4 presents some preliminary numerical experiments of the proposed method. We conclude the paper with some remarks.

1.1 Notation and basic assumptions

This subsection presents some notations and assumptions which will be used in the paper. We assume that $F:\Omega\to\mathbb{R}^{n}$ is a continuously differentiable nonlinear function, where $\Omega\subset\mathbb{R}^{n}$ is an open set containing a nonempty convex compact set $C$. The Jacobian matrix of $F$ at $x\in\Omega$ is denoted by $F^{\prime}(x)$. We also assume that there exists $x_{*}\in C$ such that $F(x_{*})=0$ and $F^{\prime}(x_{*})$ is nonsingular. Let the inner product and its associated norm in $\mathbb{R}^{n}$ be denoted by $\langle\cdot,\cdot\rangle$ and $\|\cdot\|$, respectively. The open ball centered at $a\in\mathbb{R}^{n}$ and radius $\delta>0$ is denoted by $B(a,\delta)$. For a given linear operator $T:\mathbb{R}^{n}\to\mathbb{R}^{n}$, we also use $\|\cdot\|$ to denote its norm, which is defined by $\|T\|:=\sup\{\|Tx\|,\;\|x\|\leq 1\}.$ The condition number of a continuous linear operador $A:\mathbb{R}^{n}\to\mathbb{R}^{n}$ is denoted by cond$(A)$ and it is defined as $\mbox{cond(A)}:=\|A^{-1}\|\|A\|$.

2 The method and its local convergence analysis

In this section, we present the inexact Newton-like conditional gradient (INL-CondG) method for solving (1) as well as its local convergence theorem whose proof is postponed to Section 3. Our analysis is done by using a majorant condition, which allows to unify the convergence results for two classes of nonlinear functions, namely, one satisfying a Hölder-like condition and another one satisfying a Smale condition. The convergence results for these special cases are established in this section.

The INL-CondG method is formally described as follows.

INL-CondG method

** Step 0.**

Let $x_{0}\in C$ and $\{\theta_{j}\}\subset[0,\infty)$ be given. Set $k=0$ and go to step 1.

** Step 1.**

If $F(x_{k})=0$, then stop; otherwise, choose an invertible approximation $M_{k}$ of $F^{\prime}(x_{k})$ and compute a triple $(s_{k},r_{k},y_{k})\in\mathbb{R}^{n}\times\mathbb{R}^{n}\times\mathbb{R}^{n}$ such that

$M_{k}s_{k}=-F(x_{k})+r_{k},\quad y_{k}=x_{k}+s_{k}.$

(2)

** Step 2.**

Use CondG procedure to obtain $x_{k+1}\in C$ as

$x_{k+1}=\mbox{CondG}(y_{k},x_{k},\theta_{k}\|s_{k}\|^{2}).$

** Step 3.**

Set $k\leftarrow k+1$, and go to step 1.

end

We now describe the subroutine CondG procedure.

CondG procedure $z=\mbox{CondG}(y,x,\varepsilon)$

** P0.**

Set $z_{1}=x$ and $t=1$.

** P1.**

Use the LO oracle to compute an optimal solution $u_{t}$ of

$g_{t}^{*}=\min_{u\in C}\{\langle z_{t}-y,u-z_{t}\rangle\}.$

** P2.**

If $g^{*}_{t}\geq-\varepsilon$, set $z=z_{t}$ and stop the procedure; otherwise, compute $\alpha_{t}\in\,(0,1]$ and $z_{t+1}$ as

${\alpha}_{t}:=\min\left\{1,\frac{-g^{*}_{t}}{\|u_{t}-z_{t}\|^{2}}\right\},\quad z_{t+1}=z_{t}+\alpha_{t}(u_{t}-z_{t}).$

** P3.**

Set $t\leftarrow t+1$, and go to P1.

end procedure

Remarks. 1) In our local analysis of the INL-CondG method, the invertible approximation $M_{k}$ of $F^{\prime}(x_{k})$ and the residual $r_{k}$ will satisfy classic conditions (see (5) and (6)). The inexact Newton-like method with these conditions on $M_{k}$ and $r_{k}$ was proposed in [33] and, subsequently, also studied in, for example, [6, 10]. 2) The INL-CondG method can be seen as a class of methods, depending on the choices of the invertible approximation $M_{k}$ of $F^{\prime}(x_{k})$ and residual $r_{k}$. Indeed, by letting $M_{k}=F^{\prime}(x_{k})$ and $r_{k}=0$, the INL-CondG method corresponds to Newton conditional gradient method which was studied in [15]. Another classical choice of $M_{k}$ would be $M_{k}=F^{\prime}(x_{0})$. We also emphasize that there are some approach to built $M_{k}$ that do not involve derivatives, see, for example, [24, 26, 30]. 3) Due to existence of constraint set $C$, the point $y_{k}$ in Step 1 may be infeasible and hence the INL-CondG method use an inexact conditional gradient method in order to obtain the new iterate $x_{k+1}$ in C. 4) The CondG procedure requires an oracle which is assumed to be able to minimize linear functions over the constraint set. 5) Finally, in the CondG procedure, if $g^{*}_{t}\geq-\varepsilon$, then $z_{t}\in C$ and stop. However, if $g^{*}_{t}<-\varepsilon\leq 0$, the procedure continues. In this case, the stepsize $\alpha_{t}$ is well defined and belong to (0,1].

In the following, we state our main local convergence result for the INL-CondG method whose proof is given in Section 3.

Theorem 1**.**

Let $x_{*}\in C$, $R>0$ and $\kappa:=\kappa(\Omega,R)=\sup\left\{t\in[0,R):B(x_{*},t)\subset\Omega\right\}.$ Suppose that there exist a $f:[0,R)\to\mathbb{R}$ continuously differentiable function such that

[TABLE]

for all $\tau\in[0,1]$ and $x\in B(x_{*},\kappa)$, where

• h1.

$f(0)=0$* and $f^{\prime}(0)=-1$;*

• h2.

$f^{\prime}$* is strictly increasing.*

Take the constants $\vartheta$, $\omega_{1}$, $\omega_{2}$ and $\lambda$ such that

[TABLE]

Let the scalars $\nu$, $\rho$ and $\sigma$ defined as

[TABLE]

[TABLE]

[TABLE]

Let $\{\theta_{k}\}$ and $x_{0}$ be given in step 0 of the INL-CondG method and let also $\{M_{k}\}$ and $\{(x_{k},r_{k})\}$ be generated by the INL-CondG method. Assume that the invertible approximation $M_{k}$ of $F^{\prime}(x_{k})$ satisfies

[TABLE]

and the residual $r_{k}$ is such that

[TABLE]

where $\{P_{k}\}$ is a sequence of invertible matrix (preconditioners for the linear system in (2)) and $\{\eta_{k}\}$ is a forcing sequence. If $x_{0}\in C\cap B(x_{*},\sigma)\backslash\{x_{*}\}$ and $\{\theta_{k}\}\subset[0,\lambda^{2}/2]$, then $\{x_{k}\}$ is contained in $B(x_{*},\sigma)\cap C$, converges to $x_{*}$ and there holds

[TABLE]

where $\tilde{\theta}=\limsup_{k\to\infty}\theta_{k}$. Additionally, given $0<p\leq 1$, assume that the following condition holds:

• h3.

the function (0,\,\nu)\ni t\mapsto[f(t)\big{/}f^{\prime}(t)-t]/t^{p+1} is strictly increasing.

Then, for all integer $k\geq 0$, we have

[TABLE]

Remark 1**.**

As mentioned before, the INL-CondG method as be viewed as a class of methods. Hence, the above Theorem implies, in particular, the convergence of some new methods, which are named below. We obtain, from Theorem 2, the convergence for the inexact modified Newton conditional gradient method if $M_{k}=F^{\prime}(x_{0})$, the inexact Newton conditional gradient method if $\omega_{1}=1$ and $\omega_{2}=0$ (i.e., $M_{k}=F^{\prime}(x_{k})$), and the Newton-like conditional gradient method if $\vartheta=0$ (in this case $\eta_{k}\equiv 0$ and $r_{k}\equiv 0$). We also mention that when $\omega_{1}=1$, $\omega_{2}=0$ and $\vartheta=0$ (i.e., $M_{k}=F^{\prime}(x_{k})$, $\eta_{k}\equiv 0$ and $r_{k}\equiv 0$), Theorem 2 is similar to Theorem 6 in [15].

Remark 2**.**

It is worth pointing out that when $\tilde{\theta}=0$, then (7) implies that the sequence $\{x_{k}\}$ converge linearly to $x_{*}$. Additionally, if $\omega_{1}=1$, $\omega_{2}=0$ and $\vartheta=0$, then $\{x_{k}\}$ converge superlinear to $x_{*}$. On the other hand, if $f^{\prime}$ is convex, i.e., h3 holds with $p=1$, it follows from (8), the first inequality in (7), definition (4) and the fact that $x_{0}\in C\cap B(x_{*},\sigma)\backslash\{x_{*}\}$ that $\{x_{k}\}$ converge linearly to $x_{*}$. Additionally, if $\omega_{1}=1$, $\omega_{2}=\vartheta=\lambda=0$, it follows from (8), that $\{x_{k}\}$ converge quadratically to $x_{*}$.

We now specialize Theorem 1 for two important classes of functions. In the first one, $F^{\prime}$ satisfies a Hölder-like condition [13, 14, 17], and in the second one, $F$ is an analytic function satisfying a Smale condition [36, 37].

Corollary 2**.**

Let $\kappa=\kappa(\Omega,\infty)$ as defined in Theorem 1. Assume that there exist a constant $K>0$ and $0<p\leq 1$ such that

[TABLE]

Take $0\leq\vartheta<1$, $0\leq\omega_{2}<\omega_{1}$ such that $\omega_{1}\vartheta+\omega_{2}<1$ and $\lambda\in\left[0,{(1-\omega_{2}-\omega_{1}\vartheta)}/{(\omega_{1}(1+\vartheta))}\right)$. Let

[TABLE]

Let $\{\theta_{k}\}$ and $x_{0}$ be given in step 0 of the INL-CondG method and let also $\{M_{k}\}$ and $\{(x_{k},r_{k})\}$ be generated by the INL-CondG method. Assume that the invertible approximation $M_{k}$ of $F^{\prime}(x_{k})$ satisfies

[TABLE]

and the residual $r_{k}$ is such that

[TABLE]

where $\{P_{k}\}$ is a sequence of invertible matrix (preconditioners for the linear system in (2)) and $\{\eta_{k}\}$ is a forcing sequence. If $x_{0}\in C\cap B(x_{*},\sigma)\backslash\{x_{*}\}$ and $\{\theta_{k}\}\subset[0,\lambda^{2}/2]$, then $\{x_{k}\}$ is contained in $B(x_{*},\sigma)\cap C$, converges to $x_{*}$ and there hold

[TABLE]

[TABLE]

where $\tilde{\theta}=\limsup_{k\to\infty}\theta_{k}$.

Proof.

It is immediate to prove that $F$, $x_{*}$ and $f:[0,\infty)\to\mathbb{R}$ defined by $f(t)=Kt^{p+1}/(p+1)-t$ satisfy the inequality (3), conditions h1, h2 and h3 in Theorem 1. Moreover, in this case, it is easily seen that $\nu$ and $\rho$, as defined in Theorem 1, satisfy

[TABLE]

as a consequence, $\bar{\sigma}=\min\{\kappa,\;\rho\}=\sigma$ (see (4)). Therefore, the statements of the corollary follow directly from Theorem 1. ∎

Remarks. 1) If a function $F$ is such that its derivative is L-Lipschitz continuous, i.e., $\|F^{\prime}(x)-F^{\prime}(y)\|\leq L\|x-y\|$, for all $x,y\in B(x_{*},\kappa)$ where $L>0$, then it also satisfies condition (9) with $p=1$ and $K=L\|F^{\prime}(x_{*})^{-1}\|$. Hence, we obtain the convergence of the INL-CondG method under a Lipschitz condition. In this case, $\{x_{k}\}$ converges linearly to $x_{*}$, and if additionally $\omega_{1}=1$ and $\omega_{2}=\vartheta=\lambda=0$, it converges to $x_{*}$ quadratically. 2) It is worth mentioning that if $\omega_{1}=1$ and $\omega_{2}=\vartheta=0$ in the previous corollary, we obtain the convergence of the Newton-CondG method under a Hölder-like condition, as obtained in [15, Theorem 7].

We next specialize Theorem 1 for the class of analytic functions satisfying a Smale condition.

Corollary 3**.**

Let $\kappa=\kappa(\Omega,1/\gamma)$ as defined in Theorem 1. Assume that $F:{\Omega}\to\mathbb{R}^{n}$ is an analytic function and

[TABLE]

Take $0\leq\vartheta<1$, $0\leq\omega_{2}<\omega_{1}$ such that $\omega_{1}\vartheta+\omega_{2}<1$ and $\lambda\in\left[0,{(1-\omega_{2}-\omega_{1}\vartheta)}/{(\omega_{1}(1+\vartheta))}\right)$. Let $a:=\omega_{1}(1+\vartheta)(1-3\lambda)+4(1-\omega_{1}\vartheta-\omega_{2})$, $b:=1-\omega_{1}[(1+\vartheta)\lambda+\vartheta]-\omega_{2}$ and

[TABLE]

Let $\{\theta_{k}\}$ and $x_{0}$ be given in step 0 of the INL-CondG method and let also $\{M_{k}\}$ and $\{(x_{k},r_{k})\}$ be generated by the INL-CondG method. Assume that the invertible approximation $M_{k}$ of $F^{\prime}(x_{k})$ satisfies

[TABLE]

and the residual $r_{k}$ is such that

[TABLE]

where $\{P_{k}\}$ is a sequence of invertible matrix (preconditioners for the linear system in (2)) and $\{\eta_{k}\}$ is a forcing sequence. If $x_{0}\in C\cap B(x_{*},\sigma)\backslash\{x_{*}\}$ and $\{\theta_{k}\}\subset[0,\lambda^{2}/2]$, then $\{x_{k}\}$ is contained in $B(x_{*},\sigma)\cap C$, converges to $x_{*}$ and there holds

[TABLE]

[TABLE]

where $\tilde{\theta}=\limsup_{k\to\infty}\theta_{k}$.

Proof.

Under the assumptions of the corollary, the real function $f:[0,1/\gamma)\to\mathbb{R}$, defined by $f(t)={t}/{(1-\gamma t)}-2t$, is a majorant function for $F$ on $B(x_{*},1/\gamma)$; see for instance, [10, Theorem 15]. Since $f^{\prime}$ is convex, it satisfies h3 in Theorem 1 with $p=1$; see [10, Proposition 7]. Moreover, in this case, it is easily seen that $\nu$ and $\rho$, as defined in Theorem 1, satisfy

[TABLE]

and, as a consequence, $\bar{\sigma}=\min\{\kappa,\;\rho\}=\sigma$ (see (4)). Therefore, the statements of the corollary follow from Theorem 1. ∎

Remark. The convergence of the Newton-CondG method under a Smale condition, as obtained in [15, Theorem 8], follows from the previous corollary with $\omega_{1}=1$ and $\omega_{2}=\vartheta=0$.

3 Proof of Theorem 1

The main goal of this section is to prove Theorem 1. First, we establish some properties involving the majorant function and its Newton iteration map. Then, some properties of the CondG procedure are discussed. Finally, the desired proof is presented.

From now on, we assume that all the assumptions of Theorem 1 hold, with the exception of h3, which will be considered to hold only when explicitly stated.

Proposition 4**.**

The constant $\nu$ is positive and $f^{\prime}(t)<0$ for all $t\in[0,\nu).$ As a consequence, the Newton iteration map $n_{f}:[0,\,\nu)\to\mathbb{R}$ defined by

[TABLE]

is well defined and satisfies

[TABLE]

Moreover, the constants $\rho$ and $\sigma$ are positive and

[TABLE]

Proof.

Firstly, since $f^{\prime}$ is continuous and $f^{\prime}(0)=-1$, it follows that the constant $\nu$ is positive. Hence, h2 implies that $f^{\prime}(t)<0$ for all $t\in[0,\nu)$, from which we conclude that $n_{f}$ is well defined. On the other hand, in view of h2, we have $f$ is strictly convex in $[0,R)$. Therefore, using $\nu\leq R$, we obtain $f(0)>f(t)+f^{\prime}(t)(0-t),$ for any $t\in(0,\,\nu)$ which, combined with $f(0)=0$ and $f^{\prime}(t)<0$ for any $t\in(0,\nu)$, proves the inequality in (11). Now, using the fact that $f(0)=0$ and $n_{f}(t)<0$ for all $t\in\,(0,\,\nu)$, we obtain

[TABLE]

Since $f^{\prime}(0)\neq 0$, the second statement in (11) follows by taking limit in (13), as $t$ $\downarrow 0$.

It remains to prove the last part of the proposition. First, as $\lambda<[1-\omega_{2}-\omega_{1}\vartheta]/\omega_{1}(1+\vartheta)$, we have $[1-\omega_{1}(1+\vartheta)\lambda-\omega_{1}\vartheta-\omega_{2}]/\omega_{1}(1+\vartheta)(1+\lambda)>0$. Hence, using (11), we conclude that there exists $\delta>0$ such that

[TABLE]

or, equivalently,

[TABLE]

Hence, $\rho$ is positive which in turn implies that $\sigma$ is positive and (12) holds. ∎

The following lemma gives the some relationships between the majorant function $f$ and the nonlinear operator $F$.

Lemma 5**.**

Let $x\in B\left(x_{*},\min\{\kappa,\nu\}\right)$. Then the function $F^{\prime}(x)$ is invertible and the following estimates hold:

• a)

$\|F^{\prime}(x)^{-1}F^{\prime}(x_{*})\|\leqslant 1/|f^{\prime}(\|x-x_{*}\|)|;$**

• b)

$\|F^{\prime}(x)^{-1}F(x)\|\leq f(\|x-x_{*}\|)/f^{\prime}(\|x-x_{*}\|);$**

• c)

$\|F^{\prime}(x_{*})^{-1}\left[F(x_{*})-F(x)-F^{\prime}(x)(x_{*}-x)\right]\|\leq f^{\prime}(\|x-x_{*}\|)\|x-x_{*}\|-f(\|x-x_{*}\|)$.

Proof.

The proof follows the same pattern as the proofs of Lemmas 10, 11 and 12 in [10]. ∎

The next result presents a basic property of the CondG procedure whose the proof can be found in [15, lemma 4].

Lemma 6**.**

For any $y,\tilde{y}\in\mathbb{R}^{n}$, $x,\tilde{x}\in C$ and $\mu\geq 0$, we have

[TABLE]

Before presenting the proof of Theorem 1, we first establish a technical result which will be used to prove that the sequence $\{x_{k}\}$ is contained in $B(x_{*},\sigma)\cap C$ and the sequence $\{\|x_{k}-x_{*}\|\}$ is strictly decreasing.

Lemma 7**.**

Assume that $x_{k}\in C\cap B(x_{*},\sigma)\backslash\{x_{*}\}$. Then, for every $k\geq 0$,

[TABLE]

where $n_{f}$ is as in (10). As a consequence,

[TABLE]

Proof.

First of all, since $\mbox{CondG}(x,x,0)=x,$ for all $x\in C$, it follows from INL-CondG method that

[TABLE]

Hence, using the Lemma 6 with

[TABLE]

we obtain

[TABLE]

Now, simple calculus yields

[TABLE]

Since, $x_{k}\in C\cap B(x_{*},\sigma)\backslash\{x_{*}\}$, it follows from Lemma 5 that $F^{\prime}(x_{k})$ is invertible. Thus, combining the last two inequalities, we obtain

[TABLE]

which, combined with (5) and (6), yields

[TABLE]

On the other hand, using the third inequality in (6), we find

[TABLE]

Hence, it follows from (16) that

[TABLE]

Combining the last inequality with items (a), (b) and (c) of Lemma 5, we conclude that

[TABLE]

The latter inequality, definition of $n_{f}$ in (10) and the fact that $f^{\prime}(||x_{k}-x_{*}||)<0$ imply that

[TABLE]

Hence, inequality (14) now follows by simple calculus. Since $\sqrt{2\theta_{k}}\leq\lambda$ and $0<\|x_{k}-x_{*}\|<\sigma\leq\rho$, it follows from (12) with $t=\|x_{k}-x_{*}\|$ that

[TABLE]

which, combined with (14), yields (15). ∎

We are now ready to prove Theorem 1.

Proof of Theorem 1:.

Since $x_{0}\in C\cap B(x_{*},\sigma)\backslash\{x_{*}\}$, combining the first statement of Lemma 5, inequality (15) and an induction argument, it is immediate to conclude that the sequence $\{x_{k}\}$ is contained in $B(x_{*},\sigma)\cap C$.

Let us prove that $\{x_{k}\}$ converges to $x_{*}$. Since, $\|x_{k}-x_{*}\|<\sigma\leq\rho$ for all $k\geq 0$, the first inequality in (7) follows trivially from (15). Therefore, $\{\|x_{k}-x_{*}\|\}$ is a bounded strictly decreasing sequence and hence it converges to some $\ell_{*}\in[0,\rho)$. Moreover, taking into account that $n_{f}(\cdot)$ is continuous in $[0,\rho)$, in particular, from (14) we have

[TABLE]

which, combined with (12), implies that $\ell_{*}=0$ and consequently $x_{k}\rightarrow x_{*}$.

Now, from (14) we obtain

[TABLE]

In order to prove the asymptotic rate in (7), just take the limit superior in the last inequality as $k\to\infty$ and use $\|x_{k}-x_{*}\|\to 0$, equality in (11) and $\limsup_{k\to\infty}\theta_{k}=\tilde{\theta}$.

It remains to show the last part of the theorem. For this purpose, let us assume that h3 holds. It follows from (15) and (14) respectively that, for all $k\geq 0$, $\|x_{k}-x_{*}\|<\|x_{0}-x_{*}\|$ and

[TABLE]

Therefore, the inequality (8) follows due to assumption h3, $\sqrt{2\theta_{k}}\leq\lambda$ and the definition of $n_{f}$ in  (10). ∎

Remark 3**.**

Similar to the analysis in [15], we could have defined a scalar sequence $\{t_{k}\}$, associated to the majorant function $f$, such that

[TABLE]

Indeed, if $\{t_{k}\}$ is defined as

[TABLE]

it is possible to prove that (17) holds, and $\{t_{k}\}$ is well defined, is strictly decreasing and converges to [math]. Moreover, $\limsup_{k\to\infty}t_{k+1}/t_{k}=\omega_{1}[(1+\vartheta)\sqrt{2\tilde{\theta}}+\vartheta]+\omega_{2},$ where $\tilde{\theta}=\limsup_{k\to\infty}{\theta_{k}}$.

4 Numerical experiments

In this section, we present the results of some preliminaries numerical tests which show the computational feasibility of the INL-CondG method. The experiments were carried out on a set of 15 well-known box-constrained nonlinear systems (i.e., problem (1) with $C=\{x\in\mathbb{R}^{n}:l\leq x\leq u\},$ where $l\in\mathbb{R}^{n}$ and $u\in(\mathbb{R}\cup\{\infty\})^{n}$) with dimensions between $n=400$ and $n=10000$ (see Table 1). We made three implementations of our INL-CondG method which differ in the way that the approximation matrices $M_{k}$’s are built. In the first implementation, the matrices $M_{k}$’s were approximated by finite difference (FD), whereas in the second and third ones, we used the Broyden-Schubert Update (BSU) [5, 35] and Bogle-Perkins Update (BPU) [4], respectively. The three resulting methods described above are denoted by FD-INL-CondG, BSU-INL-CondG and BPU-INL-CondG, respectively. In the implementations of the BSU-INL-CondG and BPU-INL-CondG methods, the matrices $M_{k}$’s were approximated by finite difference when $k=0$ and mod$(k-1,5)=0$. This strategy to periodically compute the Jacobian matrix $F^{\prime}$ seems to be crucial for the robustness of these derivative-free methods. For a comparison purpose, we also run the constrained Dogleg solver (CoDoSol) which is a MATLAB package based on the constrained Dogleg method [2], and available on the web site http://codosol.de.unifi.it.

The computational results were obtained using MATLAB R2016a on a 2.5GHz Intel(R) i5 with 6GB of RAM and Windows 7 ultimate system. The stopping criterion $\|F(x_{k})\|_{\infty}\leq 10^{-6}$ was used, and a failure was declared if the number of iterations was greater than $300$ or no progress was detected. The starting points were defined as $x_{0}(\gamma)=l+0.25\gamma(u-l)$ where $\gamma=1,2,3$ for problems having finite lower and upper bounds and $x_{0}(\gamma)=10^{\gamma}(1,\ldots,1)^{T}$ with $\gamma=0,1,2$, for problems with infinite upper bound. However, since $x_{0}(0)$ is a solution of Pb13, we used $x_{0}(3)$ instead. In the implementations of the INL-CondG method, the error parameter $\theta_{k}$ was set equal to $10^{-5}$ for all $k$ and the CondG Procedure stopped when either the required accuracy was obtained or the maximum of $300$ iterations were performed. The parameters of CoDoSol were set as the default choice recommended by the authors (see [2, Subsection 4.1]). It worth pointing out that the Jacobian matrices in the latter solver are approximated by finite difference.

Table 2 reports the performance of the FD-INL-CondG, BSU-INL-CondG and BPU-INL-CondG methods, and CoDoSol for solving the 15 problems considered. In table 2, “$\gamma$” is the scalar used to compute the starting point $x_{0}(\gamma)$, “Iter” is the number of iterations of the methods and “$\|F\|_{\infty}$” is the infinity norm of $F$ at the final iterate $x_{k}$. Finally, the symbol “$*$” indicates a failure.

From Table 2, we see that the FD-INL-CondG, BSU-INL-CondG and BPU-INL-CondG methods and CoDoSol successfully ended 42, 40, 37 and 42 times, respectively, on a total of 45 runs. The FD-INL-CondG method is comparable to or even slightly better than CoDoSol, because it required less iterations in 35 cases in which both methods successfully ended. This behavior also has been observed in [15] for some small and medium scale problems. Regarding the methods whose the $F^{\prime}$ is not evaluated at each iteration, the BSU-INL-CondG method solved 3 problems more than BPU-INL-CondG method, while the BPU-INL-CondG method required less (resp. more) iterations than BSU-INL-CondG method in 11 (resp. 7) cases in which both methods successfully ended. Hence, we may say that latter two methods had similar numerical performance and, for some problems, they are comparable to the methods in which all $M_{k}$’s are approximated by finite difference. Finally, based on the previous discussion, we conclude that the INL-CondG method seems to be a promising tool for solving medium and large box-constrained systems of nonlinear equations.

Final remarks

We proposed a method for solving constrained systems of nonlinear equations which is a combination of inexact Newton-like and conditional gradient methods. Under appropriate hypotheses and using a majorant condition, it was showed that the sequence generated by new method converge locally. Additionally, we were able to provide convergence results for two important classes of nonlinear functions, namely, one is for functions whose the derivative satisfies Hölder-like condition and the other is for functions that satisfies a Smale condition. In order to show the practical behavior of the proposed method, we tested it on medium- and large-scale problems from the literature. The numerical experiments showed that it works quite well and compares favorably with the constrained dogleg method [2].