On inexact relative-error hybrid proximal extragradient, forward-backward and Tseng's modified forward-backward methods with inertial effects

TL;DR

This paper introduces an inertial under-relaxed relative-error hybrid proximal extragradient method with convergence guarantees, extending to inertial forward-backward and Tseng's methods for structured monotone inclusions, under flexible assumptions.

Contribution

It develops a novel inertial under-relaxed HPE method with convergence analysis and applies it to advanced forward-backward algorithms for monotone problems.

Findings

Proves asymptotic convergence of the proposed method.

Establishes nonasymptotic iteration-complexity bounds.

Demonstrates effectiveness on structured monotone inclusion problems.

Abstract

In this paper, we propose and study the asymptotic convergence and nonasymptotic global convergence rates (iteration-complexity) of an inertial under-relaxed version of the relative-error hybrid proximal extragradient (HPE) method for solving monotone inclusion problems. We analyze the proposed method under more flexible assumptions than existing ones on the extrapolation and relative-error parameters. As applications, we propose and/or study inertial under-relaxed forward-backward and Tseng's modified forward-backward type methods for solving structured monotone inclusions.

Figures

Equations

$$0\in T(z)$$

$$0\in F(z)+B(z)$$

$$z_k:=({\lambda }_{k}T+I{)}^{-1}{z}_{k-1}\forall k\ge 1,$$

$$\displaystyle\left\{\begin{array}[]{ll}w_{k-1}:=z_{k-1}+\alpha_{k-1}(z_{k-1}-z_{k-2}),\\[11.38109pt] z_{k}:=(\lambda_{k}T+I)^{-1}w_{k-1},\end{array}\right.$$

$$0\le {\alpha }_{k-1}\le {\alpha }_k\le \alpha <1/3\forall k\ge 1.$$

$$\Vert z_k-({\lambda }_{k}T+I{)}^{-1}{z}_{k-1}\Vert \le e_k,\sum _{k=1}^{\infty }{e}_k<\infty ,$$

$$v_k\in T(z_k),{\lambda }_k{v}_k+z_k-z_{k-1}=0,$$

$$\displaystyle\left\{\begin{array}[]{ll}v_{k}\in T^{\varepsilon_{k}}(\tilde{z}_{k}),\quad\|{\lambda_{k}v_{k}+\tilde{z}_{k}-z_{k-1}}\|^{2}+2\lambda_{k}\varepsilon_{k}\leq\sigma^{2}\|{\tilde{z}_{k}-z_{k-1}}\|^{2},\\[8.53581pt] z_{k}:=z_{k-1}-\lambda_{k}v_{k},\\ \end{array}\right.$$

$$z_k:=({\lambda }_{k}B+I{)}^{-1}(z_{k-1}-{\lambda }_{k}F(z_{k-1})),$$

$$\displaystyle\left\{\begin{array}[]{ll}\tilde{z}_{k}:=(\lambda_{k}B+I)^{-1}(z_{k-1}-\lambda_{k}F(z_{k-1})),\\[8.53581pt] z_{k}:=\tilde{z}_{k}-\lambda_{k}(F(\tilde{z}_{k})-F(z_{k-1})).\\ \end{array}\right.$$

$$T^{\epsilon }(z):=\{v\in \mathcal{H}|⟨z-z^{\prime },v-v^{\prime }⟩\ge -\epsilon \forall (z^{\prime },v^{\prime })\in G(T)\}\forall z\in \mathcal{H}.$$

$$v_{\ell }\in T^{{\epsilon }_{\ell }}({\tilde{z}}_{\ell }),\ell =1,\dots ,k,\sum _{\ell =1}^k{\alpha }_{\ell }=1,$$

$${\tilde{z}}_k^a:=\sum _{\ell =1}^k{\alpha }_{\ell }{\tilde{z}}_{\ell },v_k^a:=\sum _{\ell =1}^k{\alpha }_{\ell }{v}_{\ell },{\epsilon }_k^a:=\sum _{\ell =1}^k{\alpha }_{\ell }\left({\epsilon }_{\ell }+⟨{\tilde{z}}_{\ell }-{\tilde{z}}_k^a,v_{\ell }-v_k^a⟩\right).$$

$$v_k^a\in T^{{\epsilon }_k^a}({\tilde{z}}_k^a).$$

$$\Vert pw+(1-p)z{\Vert }^2=p\Vert w{\Vert }^2+(1-p)\Vert z{\Vert }^2-p(1-p)\Vert w-z{\Vert }^2.$$

$$0\in T(z)$$

$$v\in T^{\epsilon }(z),\Vert v\Vert \le \rho ,\epsilon \le ϵ.$$

$$w_{k-1}:=z_{k-1}+{\alpha }_{k-1}(z_{k-1}-z_{k-2}).$$

$$v_k\in T^{{\epsilon }_k}({\tilde{z}}_k),\Vert {\lambda }_k{v}_k+{\tilde{z}}_k-w_{k-1}{\Vert }^2+2{\lambda }_k{\epsilon }_k\le {\sigma }^2\Vert {\tilde{z}}_k-w_{k-1}{\Vert }^2.$$

$$z_k:=w_{k-1}-\tau {\lambda }_k{v}_k.$$

$$s_k:=\max \left\{\eta \Vert z_k-w_{k-1}{\Vert }^2,(1-{\sigma }^2)\tau \Vert {\tilde{z}}_k-w_{k-1}{\Vert }^2\right\}$$

$$\eta :=\eta (\sigma ,\tau ):=\frac{2}{(1+\sigma )\tau }-1>0.$$

$$\Vert z_k-z^{\ast }{\Vert }^2+s_k\le \Vert w_{k-1}-z^{\ast }{\Vert }^2\forall k\ge 1.$$

$$\Vert z_k-z^{\ast }{\Vert }^2+(1-{\sigma }^2)\tau \Vert {\tilde{z}}_k-w_{k-1}{\Vert }^2+\tau (1-\tau )\Vert {\lambda }_k{v}_k{\Vert }^2\le \Vert w_{k-1}-z^{\ast }{\Vert }^2.$$

$${\tau }^{-1}\Vert z_k-w_{k-1}\Vert =\Vert {\lambda }_k{v}_k\Vert$$

$$\le (1+\sigma )\Vert {\tilde{z}}_k-w_{k-1}\Vert ,$$

$$(1-{\sigma }^2)\tau \Vert {\tilde{z}}_k-w_{k-1}{\Vert }^2$$

$$\tau (1-\tau )\Vert {\lambda }_k{v}_k{\Vert }^2={\tau }^{-1}(1-\tau )\Vert \tau {\lambda }_k{v}_k{\Vert }^2={\tau }^{-1}(1-\tau )\Vert z_k-w_{k-1}{\Vert }^2.$$

$$\Vert w_{k-1}-z{\Vert }^2=(1+{\alpha }_{k-1})\Vert z_{k-1}-z{\Vert }^2-{\alpha }_{k-1}\Vert z_{k-2}-z{\Vert }^2+{\alpha }_{k-1}(1+{\alpha }_{k-1})\Vert z_{k-1}-z_{k-2}{\Vert }^2.$$

$$(\forall k\ge -1){\phi }_k:=\Vert z_k-z^{\ast }{\Vert }^2\text{mbox}and(\forall k\ge 1){\delta }_k:={\alpha }_{k-1}(1+{\alpha }_{k-1})\Vert z_{k-1}-z_{k-2}{\Vert }^2.$$

$${\phi }_k-{\phi }_{k-1}+s_k\le {\alpha }_{k-1}({\phi }_{k-1}-{\phi }_{k-2})+{\delta }_k\forall k\ge 1,$$

Full text

2000 Mathematics Subject Classification: 90C25, 90C30, 47H05.

Key words: inertial, relaxed, proximal point method, HPE method, pointwise, ergodic, iteration-complexity, forward-backward algorithm, Tseng’s modified forward-backward algorithm.

Introduction

Inertial proximal point-type algorithms for monotone inclusions gained a lot of attention in research recently (see, e.g., [3, 4] and the references therein). The first method of this type – the inertial proximal point (PP) method – for solving generalized equations with monotone operators was proposed and studied by Alvarez and Attouch in [2]. The intense research activity in the subject in the last years is in part due to its connections with fast first-order algorithms for convex programming (see, e.g., [3, 4, 5, 6, 7, 23]).

Since the inertial PP method of Alvarez and Attouch has been used as the hidden engine for the design and analysis of various first-order proximal algorithms with inertial effects, including inertial versions of ADMM, forward-backward and Douglas-Rachford algorithms (see, e.g., [3, 4, 10, 15, 16]), it is natural to attempt to design inexact versions of it. In [9], Bot and Csetnek proposed and studied the asymptotic convergence of an inertial version of the hybrid proximal extragradient (HPE) method of Solodov and Svaiter [30, 35]. The HPE method is an inexact PP algorithm for which, at each iteration, the corresponding proximal subproblems are supposed to be (inexactly) solved within a relative error criterion (this contrasts to the summable error criterion proposed by Rockafellar [34]).

In this paper, we propose and study the asymptotic convergence and nonasymptotic global convergence rates (iteration-complexity) of an inertial under-relaxed HPE method for solving monotone inclusions. The proposed method (Algorithm 1) differs from the existing inertial HPE-type method of Bot and Csetnek in the sense it is based on a different mechanism of iteration. Moreover, we prove its convergence and iteration-complexity under more flexible assumptions than those proposed in [9] on the extrapolation and relative-error parameters. As applications, we study inertial (under-relaxed) versions of the Tseng’s modified forward-backward and forward-backward algorithms (see Algorithms 3 and 4) for solving structured monotone inclusions problems.

The main contributions of this paper will be further discussed in Section 1.

This paper is organized as follows. In Section 1, we present some preliminaries and basic results, review some existing algorithms and discuss in detail the main contributions of this paper. The inertial under-relaxed HPE method (Algorithm 1) is presented in Section 2; the main results are Theorems 2.5 (asymptotic convergence), and 2.7 and 2.8 (iteration-complexity). Sections 3 is devoted to present and study the inertial versions of the Tseng’s modified forward-backward and forward-backward algorithms; the main results are Theorems 3.2 and 3.4. We finish the paper in Section 4 with some concluding remarks.

The inertial proximal point (PP) method is a modification of the Rockafellar’s PP method for which, at each iteration, past information is used to extrapolate the current iterate by an extrapolation factor $\alpha\in[0,1[$. The method was proposed and studied by Alvarez and Attouch and since then it The convergence of the inertial PP method was proved in under the assumption that $\alpha$ is within the range $0\leq\alpha<1/3$, which has become a standard assumption in the analysis of different variants and special instances of Alvarez–Attouch’s method.

The hybrid proximal extragradient (HPE) method of Solodov and Svaiter is an inexact PP algorithm for which, at each iteration, the proximal subproblems are supposed to be (inexactly) solved within a relative error criterion determined by a tolerance $\sigma\in[0,1[$. When $\sigma=0$ it reduces to the exact Rockafellar’s PP method and, on the other hand, $\sigma=0.99$ has been successfully used in many applications. Recently, an inertial version of the HPE method was proposed and studied by Bot and Csetnek . In this case, the extrapolation factor $\alpha$ depends on $\sigma$ and it is close to zero for large values of $\sigma\in[0,1[$ .

In this paper, we propose an inertial under-relaxed HPE method, which generalizes the under-relaxed HPE method of Svaiter . In contrast to the inertial HPE method of Bot and Csetnek, we obtain convergence and iteration-complexity of the proposed algorithm for the extrapolation factor $\alpha$ within the standard range $0\leq\alpha<1/3$ at the price of performing an under-relaxed step with factor with $\tau\geq 0.5$ (uniformly on $\sigma\in[0,1[$). Simple numerical experiments indicate that this strategy is promising even in the exact case, i.e., when $\sigma=0$. Beyond to that, we explicitly compute the corresponding under-relaxation factor $\tau=\tau(\sigma,\beta)$ to implement the method with extrapolation factor within the range $0\leq\alpha<\beta<1$ ($\tau=1$ when $\beta=1/3$ and $\sigma=0$). As an application, we propose and study an (under-relaxed) inertial version of the forward-backward-forward method of Tseng. We show that our propsed version of the latter algorithm deals differently and better with parameters than existing ones.

1 Preliminaries, basic results and general notation

1.1 Problem statement

Let $\mathcal{H}$ be a real Hilbert space and consider the general monotone inclusion problem (MIP) of finding $z\in\mathcal{H}$ such that

[TABLE]

as well as the structured MIP

[TABLE]

where $T$ and $B$ are (set-valued) maximal monotone operators on $\mathcal{H}$ and $F:D(F)\subset\mathcal{H}\to\mathcal{H}$ is a (point-to-point) monotone operator which is either Lipschitz continuous or cocoercive (see Subsections 3.1 and 3.2 for the precise statement). Problems (1) and (2) appear in different fields of applied mathematics and optimization including convex optimization, signal processing, PDEs, inverse problems, among others (see, e.g.,[8, 19]). We mention that under mild conditions on the operators $F$ and $B$, problem (2) becomes a special instance of (1) with $T:=F+B$.

In this paper, we propose and study the asymptotic convergence and the iteration-complexity of inertial under-relaxed versions of the hybrid proximal extragradient (HPE) method (Algorithm 1), and Tseng’s modified forward-backward (Algorithm 3) and forward-backward (Algorithm 4) methods for solving (1), and (2), respectively.

The main contributions of (as well as the most related works with) this paper will be discussed along the next subsections, the main contributions being further summarized in Subsection 1.5.

1.2 The Alvarez–Attouch’s inertial proximal point method

The proximal point (PP) method is an iterative scheme for seeking approximate solutions of (1). It was first proposed by Martinet [25] for solving monotone variational inequalities (with point-to-point operators) and further studied and developed by Rockafellar in his pioneering work [34]. In its exact formulation, an iteration of the PP method can be described by

[TABLE]

where $\lambda_{k}>0$ is a stepsize parameter and $z_{k-1}$ is the current iterate.

The inertial PP method is a modification of (3) proposed and studied by Alvarez and Attouch in [2] as follows: for all $k\geq 1$,

[TABLE]

where $\{\alpha_{k}\}$ is a sequence of extrapolation parameters; note that if $\alpha_{k}\equiv 0$, then it follows that (6) reduces to the Rockafellar’s PP method (3). Inertial PP-type methods deserve a lot of attention in nowadays research due the possibility of extending this methodology to different practical algorithms and, in part, as we mentioned earlier, due to its connections with fast first-order methods in convex programming. Asymptotic (weak) convergence of $\{z_{k}\}$ generated in (6) to a solution of (1) was first obtained in [2] under the assumptions that $\lambda_{k}\geq\underline{\lambda}>0$ and

[TABLE]

The above upper bound $1/3$ on $\{\alpha_{k}\}$ has become standard in the analysis of inertial-like proximal algorithms (see, e.g., [15, 16, 23, 32]). It seems that (7) was first improved by Alvarez in [1, Proposition 2.5] in the setting of projective-proximal point-type methods and, more recently, by Attouch and Cabot in [4] with relaxation playing a central role. One of the main goals of this contribution is the analysis of an inertial under-relaxed HPE-type method under the assumption (actually more general than) (7) on $\{\alpha_{k}\}$; see Assumption ${\bf(A)}$.

1.3 The hybrid proximal extragradient method of Solodov and Svaiter

It is of course important to design and study inexact versions of known (exact) numerical algorithms, and this also applies to (3). In [34], Rockafellar proved that if, at each iteration $k\geq 1$, $z_{k}$ is computed satisfying

[TABLE]

and $\{\lambda_{k}\}$ is bounded away from zero, then $\{z_{k}\}$ converges (weakly) to a solution of (1). Many modern inexact versions of the PP method (3), as opposed to the summable error criterion (8), use relative error tolerances for solving the associated subproblems. The first methods of this type were proposed by Solodov and Svaiter in [35, 36] and subsequently studied in [29, 30, 31, 37, 38]. The key idea consists of observing that (3) can be decoupled as

[TABLE]

and then relaxing (9) within relative error tolerance criteria. Among these new methods, the HPE method [35] has been shown to be very effective as a framework for the design and analysis of many concrete algorithms (see, e.g., [9, 14, 17, 20, 21, 24, 26, 27, 28, 31, 35, 37, 38]). It can be described as follows: for all $k\geq 1$,

[TABLE]

where $\sigma\in[0,1[$. (see Subsection 1.6 for the general notation on $\varepsilon$-enlargements $T^{\varepsilon}(\cdot)$.) Note that if $\sigma=0$, then it follows that (12) reduces to the exact PP method (3). As we mentioned before, recently Bot and Csetnek [9] proposed and studied an inertial proximal-like algorithm which combines ideas from (6) and (12). They have proved asymptotic convergence of their method under the assumption $\alpha(5+4\sigma^{2})+\sigma^{2}<1$ on $\alpha$ and $\sigma$, where $\sigma\in[0,1[$ is as in (12) and $0\leq\alpha_{k-1}\leq\alpha_{k}\leq\alpha<1$ for all $k\geq 1$ (cf. (7)). This condition enforces $\alpha\approx 0$ whenever $\sigma\approx 1$. This would, in particular, degenerate the desired inertial effect in many important applications of HPE-type methods for which $\sigma=0.99$ is known (experimentally) to be the best choice among all possible $\sigma\in[0,1[$ (see, e.g., [17, 18, 26, 27]).

In this paper, we propose an inertial under-relaxed HPE-type method (Algorithm 1) with guarantee of asymptotic convergence and iteration-complexity (both pointwise and ergodic) under the assumption (actually more general than) (7) on $\{\alpha_{k}\}$; see Assumption ${\bf(A)}$. The price to pay is to perform, in addition to inertial, under-relaxed steps. On the other hand, the under-relaxed parameter $\tau\in]0,1]$ is explicitly computed and, in the case of (7), $\tau\geq 0.5$, the latter lower bound being uniform on $\sigma\in[0,1[$ (see the third remark following Assumption ${\bf(A)}$). We also emphasize that our algorithm is different of the corresponding one in [9], in the sense it is based on a different mechanism of iteration.

The main convergence results on Algorithm 1 are Theorems 2.5, 2.7 and 2.8. It seems it is the first time in the literature that global (ergodic) $\mathcal{O}(1/k)$ convergence rates are obtained for inertial-like proximal algorithms (see Theorem 2.8).

1.4 Forward-backward and Tseng’s modified forward-backward methods

With its roots in the projected gradient algorithm for convex optimization, the forward-backward method (see, e.g., [22, 33]) is one of the most popular numerical algorithms for solving the structured monotone inclusion problem (2), having numerous applications in modern applied mathematics (see, e.g., [8]). It can be described as follows: for all $k\geq 1$,

[TABLE]

where $\lambda_{k}>0$ is a stepsize parameter and $z_{k-1}$ is the current iterate. Under the assumption that $F:\mathcal{H}\to\mathcal{H}$ is cocoercive and $\{\lambda_{k}\}$ is within a certain range, it follows that the sequence $\{z_{k}\}$ generated in (13) is weakly convergent to a solution of (2) (see, e.g., [8]). In the seminal paper [41], Tseng proposed and studied the following modification of (13) – known as the Tseng’s modified forward-backward method: for all $k\geq 1$,

[TABLE]

We clearly see that in contrast to (13), (16) performs an additional forward step to define the next iterate $z_{k}$. This is crucial to obtain convergence under the (weaker than cocoercivity) assumption of Lipschitz continuity on $F$ (see, e.g., [8, 41]). Since both forward-backward and Tseng’s modified forward-backward methods are known to be special instances of the HPE method (12) for solving (1) with $T:=F+B$ (see, e.g., [30, 35, 39]), we have managed to propose and/or study inertial under-relaxed versions of (13) and (16) – namely, Algorithms 3 and 4, respectively – as special instances of the proposed inertial under-relaxed HPE method (Algorithm 1). As a by-product of the results obtained for Algorithm 1, we prove their asymptotic convergence as well as their global $\mathcal{O}(1/\sqrt{k})$ pointwise and $\mathcal{O}(1/k)$ ergodic convergence rates/iteration-complexity (see Theorems 3.2 and 3.4). We discuss some existing inertial/relaxed variants of (13) and (16) as well as how they are related to Algorithms 3 and 4 in the remarks following them. We also emphasize that, since Algorithms 3 and 4 will be analyzed within the framework of Algorithm 1, they will automatically inherit all the possible benefits from the proposed policy of choosing the upper bound on the sequence of inertial parameters and the relaxation parameter (see Assumption (A), the remarks following it, and the remarks following Algorithms 3 and 4).

1.5 The main contributions of this work

We summarize the main contributions of this work are as follows:

• (i)

Asymptotic convergence and nonasymptotic global $\mathcal{O}(1/\sqrt{k})$ pointwise and $\mathcal{O}(1/k)$ ergodic convergence rates (iteration-complexity) of an inertial under-relaxed HPE method (Algorithm 1) for solving (1) under more flexible than existing assumptions on the choice of inertial $\{\alpha_{k}\}$ and relative-error $\sigma\in[0,1[$ parameters (see Assumption (A) and the remarks following it). We show, in particular, that it is possible to assume the upper bound $1/3$ on the sequence of inertial parameters $\{\alpha_{k}\}$, which became standard in the analysis of inertial-type proximal algorithms, at the price of performing under-relaxed iterations with explicitly computed parameter $\tau\geq 0.5$, where the latter lower bound is uniform on the relative-error parameter $\sigma\in[0,1[$. We also emphasize that, up to the authors knowledge, it is the first time in the literature that an iteration-complexity analysis is performed for inertial HPE-type methods (see Theorems 2.7 and 2.8) and it seems it is also the first time that ergodic iteration-complexity results are established for inertial proximal-type algorithms.

• (ii)

Asymptotic convergence and pointwise and ergodic iteration-complexity of inertial under-relaxed versions of the Tseng’s modified forward-backward method (Algorithm 3) and forward-backward method (Algorithm 4) for solving (2) under the assumption that $F$ is monotone and either Lipschitz continuous or cocoercive. Analogously to (i), in this case, the proposed methods also benefit from the more flexible than standard assumptions on the choice of inertial parameters (see Subsections 3.1 and 3.2 for a discussion).

1.6 General notation and basics on monotone operators and $\varepsilon$–enlargements

Let $\mathcal{H}$ be a real Hilbert space with inner product $\langle\cdot,\cdot\rangle$ and induced norm $\|\cdot\|=\sqrt{\langle\cdot,\cdot\rangle\textbf{}}$. The weak limit of a sequence $\{z_{k}\}$ in $\mathcal{H}$ (whenever it exists) will be denoted by $w-\lim_{k\to\infty}\,z_{k}$. A set-valued map $T:\mathcal{H}\rightrightarrows\mathcal{H}$ is said to be a monotone operator if $\langle z-z^{\prime},v-v^{\prime}\rangle\geq 0$ for all $v\in T(z)$ and $v^{\prime}\in T(z^{\prime})$. On the other hand, $T:\mathcal{H}\rightrightarrows\mathcal{H}$ is maximal monotone if $T$ is monotone and its graph $G(T):=\{(z,v)\in\mathcal{H}\times\mathcal{H}\,|\,v\in T(z)\}$ is not properly contained in the graph of any other monotone operator on $\mathcal{H}$. The inverse of $T:\mathcal{H}\rightrightarrows\mathcal{H}$ is $T^{-1}:\mathcal{H}\rightrightarrows\mathcal{H}$, defined at any $z\in\mathcal{H}$ by $v\in T^{-1}(z)$ if and only if $z\in T(v)$. The resolvent of a maximal monotone operator $T:\mathcal{H}\rightrightarrows\mathcal{H}$ is $(T+I)^{-1}$ and $z=(T+I)^{-1}x$ if and only if $x-z\in T(z)$. The operator $\gamma T:\mathcal{H}\rightrightarrows\mathcal{H}$, where $\gamma>0$, is defined by $(\gamma T)z:=\gamma T(z):=\{\gamma v\,|\,v\in T(z)\}$.

For $T:\mathcal{H}\rightrightarrows\mathcal{H}$ maximal monotone and $\varepsilon\geq 0$, the $\varepsilon$-enlargement [11] of $T$ is the operator $T^{\varepsilon}:\mathcal{H}\rightrightarrows\mathcal{H}$ defined by

[TABLE]

Note that $T(z)\subset T^{\varepsilon}(z)$ for all $z\in\mathcal{H}$.

The following summarizes some useful properties of $T^{\varepsilon}$ (see, e.g., [13, Lemma 3.1 and Proposition 3.4(b)]).

Proposition 1.1**.**

Let $T,S:\mathcal{H}\rightrightarrows\mathcal{H}$ be set-valued maps. Then,

• (a)

If $\varepsilon\leq\varepsilon^{\prime}$, then $T^{\varepsilon}(z)\subseteq T^{\varepsilon^{\prime}}(z)$ for every $z\in\mathcal{H}$.

• (b)

$T^{\varepsilon}(z)+S^{\,\varepsilon^{\prime}}(z)\subseteq(T+S)^{\varepsilon+\varepsilon^{\prime}}(z)$* for every $z\in\mathcal{H}$ and $\varepsilon,\varepsilon^{\prime}\geq 0$.*

• (c)

$T$* is monotone, if and only if $T\subseteq T^{0}$.*

• (d)

$T$* is maximal monotone, if and only if $T=T^{0}$. *

• (e)

If $T$ is maximal monotone, $\{(\tilde{z}_{k},v_{k},\varepsilon_{k})\}$ is such that $v_{k}\in T^{\varepsilon_{k}}(\tilde{z}_{k})$, for all $k\geq 1$, $w-\lim_{k\to\infty}\,\tilde{z}_{k}=z$, $\lim_{k\to\infty}\,v_{k}=v$ and $\lim_{k\to\infty}\,\varepsilon_{k}=\varepsilon$, then $v\in T^{\varepsilon}(z)$.

Proposition 1.2**.**

(see, e.g., [13, Lemma 3.1 and Proposition 3.4(b)])*

Assume $\{(\tilde{z}_{k},v_{k},\varepsilon_{k})\}$ is a sequence in $\mathcal{H}\times\mathcal{H}\times\mathbb{R}_{+}$ such that $v_{k}\in T^{\varepsilon_{k}}(\tilde{z}_{k})$ for all $k\geq 1$. If, $w-\lim_{k\to\infty}\,\tilde{z}_{k}=z$, $\lim_{k\to\infty}\,v_{k}=v$ and $\lim_{k\to\infty}\,\varepsilon_{k}=\varepsilon$, then $v\in T^{\varepsilon}(z)$.*

Next we present the transportation formula for $\varepsilon$-enlargements.

Theorem 1.3**.**

(see, e.g., [12, Theorem 2.3])*

Suppose $T:\mathcal{H}\rightrightarrows\mathcal{H}$ is maximal monotone and let $\tilde{z}_{\ell},v_{\ell}\in\mathcal{H}$, $\varepsilon_{\ell},\alpha_{\ell}\in\mathbb{R}_{+}$, for $\ell=1,\dots,k$, be such that*

[TABLE]

and define

[TABLE]

Then, $\varepsilon_{k}^{a}\geq 0$ and

[TABLE]

The following well-known property will be also useful in this paper. For any $w,z\in\mathcal{H}$ and $p\in\mathbb{R}$, we have

[TABLE]

2 An inertial under-relaxed hybrid proximal extragradient method

Consider the monotone inclusion problem (1), i.e., the problem of finding $z\in\mathcal{H}$ such that

[TABLE]

where $T$ is a maximal monotone operator on $\mathcal{H}$ for which $T^{-1}(0)\neq\emptyset$.

In this section, we propose and study the asymptotic convergence and nonasymptotic global convergence rates (iteration-complexity) of an inertial under-relaxed hybrid proximal extragradient (HPE) method (Algorithm 1) for solving (19). Regarding the iteration-complexity analysis, we consider the following notion of approximate solution for (19): given tolerances $\rho,\epsilon>0$, find $z,v\in\mathcal{H}$ and $\varepsilon\geq 0$ such that

[TABLE]

Note that $\rho=\epsilon=0$ in (20) gives $0\in T(z)$, i.e., in this case $z\in\mathcal{H}$ is a solution of (19) (for a more detailed discussion on (20), see, e.g., [30]).

The main results in this section are Theorems 2.5, 2.7 and 2.8. We refer the reader to the remarks and comments following each of the above mentioned theorems for a discussion regarding the contribution of each of them in the light of related results available in the current literature.

Algorithm 1**.**

An inertial under-relaxed HPE method for solving (19)**

Input: $z_{0}=z_{-1}\in\mathcal{H}$ and $0\leq\alpha,\sigma<1$ and $0<\tau\leq 1$.

1:

for $k=1,2,\dots$, do

2:

Choose $\alpha_{k-1}\in[0,\alpha]$ and define

$\displaystyle w_{k-1}:=z_{k-1}+\alpha_{k-1}(z_{k-1}-z_{k-2}).$

(21)

3:

Find $(\tilde{z}_{k},v_{k},\varepsilon_{k})\in\mathcal{H}\times\mathcal{H}\times\mathbb{R}_{+}$ and $\lambda_{k}>0$ such that

$\displaystyle v_{k}\in T^{\varepsilon_{k}}(\tilde{z}_{k}),\quad\|{\lambda_{k}v_{k}+\tilde{z}_{k}-w_{k-1}}\|^{2}+2\lambda_{k}\varepsilon_{k}\leq\sigma^{2}\|{\tilde{z}_{k}-w_{k-1}}\|^{2}.$

(22)

4:

Define

$\displaystyle z_{k}:=w_{k-1}-\tau\lambda_{k}v_{k}.$

(23)

Remarks.

• (i)

Algorithm 1 clearly combines the inertial proximal point (PP) and the HPE methods (6) and (12), respectively. It reduces to (6) when $\sigma=0$ and $\tau=1$. Indeed, in this case, using (22), (23) and Proposition 1.1(d), we find $0\in\lambda_{k}T(z_{k})+z_{k}-[z_{k-1}+\alpha_{k-1}(z_{k-1}-z_{k-2})]$ for all $k\geq 1$ (cf. iteration $(\mathcal{A}_{0})$–$(\mathcal{A}_{2})$ in [2]).

• (ii)

A similar inertial relaxed relative-error PP algorithm was proposed and analyzed by Alvarez in [1]. We emphasize that in contrast to Algorithm 1, the algorithm proposed by Alvarez is a projective-type algorithm (see, e.g., [36]) and it is based on a different mechanism of iteration.

• (iii)

Algorithm 1 generalizes the HPE method of Solodov and Svaiter [30] and (a special instance of) the under-relaxed HPE method of Svaiter [40]. Indeed, the HPE method is obtained by letting $\alpha=0$ and $\tau=1$, in which case $w_{k-1}=z_{k-1}$, while the under-relaxed HPE method (with $t_{k}\equiv\tau$, in the notation of the latter reference) appears whenever $\alpha=0$ in Algorithm 1.

• (iv)

As we mentioned in Subsection 1.3, an inertial HPE-type method was recently proposed and studied by Bot and Csetnek in [9]. We refer the reader to Subsection 1.3 for a discussion of the contributions of this paper in the light of the latter reference, regarding the HPE-type methods.

• (v)

We emphasize that, in contrast to the analysis presented in this work – see Theorems 2.7 and 2.8 –, in all cases of inertial-type algorithms which were mentioned in remarks (i)–(iv) no iteration-complexity analysis has been obtained.

• (vi)

Step 3 of Algorithm 1 does not specify how to compute $\lambda_{k}>0$ and the triple $(\tilde{z}_{k},v_{k},\varepsilon_{k})$ satisfying (22), their computation depending on the instance of the method under consideration. In this regard, Proposition 3.3 shows, in particular, how the evaluation of a cocoercive (monotone) point-to-point operator naturally produces such triples.

The next three results, especially Proposition 2.3, will be important for proving the main results on convergence and iteration-complexity of Algorithm 1.

Proposition 2.1**.**

Let $\{z_{k}\}$, $\{\tilde{z}_{k}\}$ and $\{w_{k}\}$ be generated by Algorithm 1 and define, for all $k\geq 1$,

[TABLE]

where

[TABLE]

Then, for any $z^{*}\in T^{-1}(0)$,

[TABLE]

Proof.

Using (22), (23) and Lemma A.2(b) we obtain

[TABLE]

Note now that from (23) and (22) we have

[TABLE]

which, in turn, gives

[TABLE]

On the other hand, (23) yields

[TABLE]

To finish the proof, note that (26) is a direct consequence of (24), (27)–(29) and (25). ∎

Lemma 2.2**.**

Let $\{z_{k}\}$, $\{w_{k}\}$ and $\{\alpha_{k}\}$ be generated by Algorithm 1 and let $z\in\mathcal{H}$. Then, for all $k\geq 1$,

[TABLE]

Proof.

From (21) we have $z_{k-1}-z=(1+\alpha_{k-1})^{-1}(w_{k-1}-z)+\alpha_{k-1}(1+\alpha_{k-1})^{-1}(z_{k-2}-z)$ and $w_{k-1}-z_{k-2}=(1+\alpha_{k-1})(z_{k-1}-z_{k-2})$, which combined with the property (18) yield the desired identity. ∎

Proposition 2.3**.**

Let $\{z_{k}\}$, $\{w_{k}\}$ and $\{\alpha_{k}\}$ be generated by Algorithm 1 and let $\{s_{k}\}$ be as in (24). Let also $z^{*}\in T^{-1}(0)$ and define

[TABLE]

Then, $\varphi_{0}=\varphi_{-1}$ and

[TABLE]

i.e., the sequences $\{\varphi_{k}\}$, $\{s_{k}\}$, $\{\alpha_{k}\}$ and $\{\delta_{k}\}$ satisfy the assumptions of Lemma A.5.

Proof.

Using Lemma 2.2 with $z=z^{*}$ and (30) we obtain, for all $k\geq 1$,

[TABLE]

which combined with Proposition 2.1 and the definition of $\varphi_{k}$ in (30) yields (31). The identity $\varphi_{0}=\varphi_{-1}$ follows from the fact that $z_{0}=z_{-1}$ and the first definition in (30). ∎

Next we present the first result on the asymptotic convergence of Algorithm 1.

Theorem 2.4** (first result on the weak convergence of Algorithm 1).**

Let $\{z_{k}\}$, $\{\lambda_{k}\}$ and $\{\alpha_{k}\}$ be generated by Algorithm 1. If the following holds

[TABLE]

and, additionally, $\lambda_{k}\geq\underline{\lambda}>0$, for all $k\geq 1$, then the sequence $\{z_{k}\}$ converges weakly to a solution of the monotone inclusion problem (19).

Proof.

Using Proposition 2.3, (32), the fact that $\alpha_{k}\leq\alpha$ for all $k\geq 1$ and Lemma A.5, one concludes that (i) $\lim_{k\to\infty}\,\|z_{k}-z^{*}\|$ exist for every $z^{*}\in\Omega:=T^{-1}(0)$, and $\sum_{k=1}^{\infty}\,s_{k}<+\infty$, which gives (ii) $\lim_{k\to\infty}\,s_{k}=0$, where $\{s_{k}\}$ is as in (24). In particular, $\{z_{k}\}$ is bounded. Using (ii), (22)–(24) and the assumption $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$, we find

[TABLE]

Now let $z^{\infty}\in\mathcal{H}$ be a weak cluster point of $\{z_{k}\}$ (recall that it is bounded). Note that it follows from (33) that $z^{\infty}$ is also a (weak) cluster point of $\{\tilde{z}_{k}\}$ and let $\{\tilde{z}_{k_{j}}\}$ be such that $w-\lim_{j\to\infty}\,\tilde{z}_{k_{j}}=z^{\infty}$. Using (33) and the inclusion in (22) we obtain

[TABLE]

which, in turn, combined with Proposition 1.1(e) yields $z^{\infty}\in\Omega=T^{-1}(0)$, and so the desired result follows from (i) and Lemma A.4. ∎

Remark. Condition (32) appeared for the first time in [2], and since then it has become a standard assumption in the asymptotic convergence analysis of different inertial PP-type algorithms. Next, we present a sufficient condition on the input parameters $(\alpha,\sigma,\tau)$ in Algorithm 1 to ensure (32) holds (see Theorems 2.5, 2.7 and 2.8).

Assumption (A): $(\alpha,\sigma,\tau)\in[0,1[\times[0,1[\times]0,1]$ and $\{\alpha_{k}\}$ satisfy the following (for some $\beta>0$):

[TABLE]

and

[TABLE]

where

[TABLE]

Remarks.

• (i)

Conditions (35)–(37) will be crucial to prove convergence and iteration-complexity of the algorithms presented and studied in this paper; see, e.g., Theorems 2.5, 2.7 and 2.8, and Section 3.

• (ii)

Note that by letting $\sigma=0$, which by the first remark following Algorithm 1 means that it reduces to an under-relaxed version of the (exact) Alvarez–Attouch’s inertial PP method, we obtain that (35)–(37) are now simply given by: $0\leq\alpha_{k-1}\leq\alpha_{k}\leq\alpha<\beta<1$, for all $k\geq 1$, and

[TABLE]

In particular, in this case, we have $\tau=\tau(0,1/3)=1$ whenever $\beta=1/3$ in (35), which corresponds to the standard upper bound on $\{\alpha_{k}\}$ which has been used in different works in the current literature (see Subsection 1.2 for a discussion). Hence, even in the setting of exact inertial PP methods, conditions (35)–(37) generalize the usual assumption (7). See Figure 1.

• (iii)

As we mentioned earlier, an inertial HPE-type method was proposed and studied by Bot and Csetnek in [9], where asymptotic convergence is proved under the assumption $\alpha(5+4\sigma^{2})+\sigma^{2}<1$ on $\alpha,\sigma\in[0,1[$. Note that, in this case, $\alpha\approx 0$ whenever $\sigma\approx 1$. This contrasts to the conditions (35)–(37), which, in particular yield $\tau=\tau(\sigma,1/3)=1/(1+\sigma)>0.5$ (uniformly on $\sigma$) when $\beta=1/3$ in (35). This may become especially useful in numerical implementations of Algorithm 1, since $\sigma=0.99$ has been usually employed in the recent literature on HPE-type methods (see, e.g., [17, 18, 26, 27]). Further, (35)–(37) allow the upper bound $\alpha$ on $\{\alpha_{k}\}$ to be chosen arbitrarily close to 1, at the price of performing under-relaxed steps with the explicitly computed $\tau=\tau(\sigma,\beta)$ as in (36). See Figure 1.

Theorem 2.5** (second result on the weak convergence of Algorithm 1).**

Under the Assumption ${\bf(A)}$ on Algorithm 1, let $\eta>0$ be as in (25) and define the quadratic real function:

[TABLE]

Then, $q(\alpha)>0$ and, for every $z^{*}\in T^{-1}(0)$,

[TABLE]

*As a consequence, it follows that under the assumption ${\bf(A)}$ the sequence $\{z_{k}\}$ generated by Algorithm 1 converges weakly to a solution of the monotone inclusion problem (19) whenever $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$. *

Proof.

Using (21), the Cauchy-Schwarz inequality and the Young inequality $2ab\leq a^{2}+b^{2}$ with $a:=\|{z_{k}-z_{k-1}}\|$ and $b:=\|{z_{k-1}-z_{k-2}}\|$ we find

[TABLE]

which combined with (31) and (24), and after some algebraic manipulations, yields

[TABLE]

where

[TABLE]

Define,

[TABLE]

where $\varphi_{k}$ is as in (30). Using (39), the assumption that $\{\alpha_{k}\}$ is nondecreasing (see (35)) and (41)–(43) we obtain, for all $k\geq 1$,

[TABLE]

Note now that from (36) and Lemma A.3 we have

[TABLE]

which, in turn, combined with the definition of $\eta>0$ in (25), and after some algebraic calculations, gives

[TABLE]

The latter identity implies, in particular, that $\beta^{\prime}$ is either the smallest or the largest root of the quadratic function $q(\cdot)$. Hence, from (35) and the fact that $\beta^{\prime}\geq\beta$ (see (37)) we obtain

[TABLE]

The above inequalities combined with (2) yield

[TABLE]

which, in turn, combined with (35) and the definition of $\mu_{k}$ in (43), gives

[TABLE]

Note now that (45), (35) and (43) also yield

[TABLE]

and so,

[TABLE]

Hence, (40) follows directly from (2), (47), the definition of $\mu_{0}$ in (43) and the definition of $\varphi_{0}$ in (30). On the other hand, the second statement of the theorem follows from (40) and Theorem 2.4 (recall that $\alpha_{k}\leq\alpha<1$ for all $k\geq 0$). ∎

Remark. A quadratic function similar to $q(\cdot)$, as defined in (39), was also considered by Alvarez in [1]. As we mentioned in the second remark following Algorithm 1, the algorithm studied in the later reference is different of the corresponding algorithm presented in this work, namely Algorithm 1. Moreover, note that if $\eta=1$, then $q(\alpha^{\prime})=1-3\alpha^{\prime}$ (cf. [2]).

Corollary 2.6**.**

Under the Assumption ${\bf(A)}$ on Algorithm 1, let $\eta>0$ and $q(\cdot)$ be as in (25) and (39), respectively, and let $z^{*}\in T^{-1}(0)$. Then, for all $k\geq 1$,

[TABLE]

Proof.

Using Proposition 2.3 and Lemma A.5(a) we conclude that (85) holds with $\{s_{k}\}$, $\{\varphi_{k}\}$ and $\{\delta_{k}\}$ as in (24) and (30), which gives that the desired result follows from (85) and (40). ∎

Next we present the first result on nonasymptotic global convergence rates/iteration-complexity of Algorithm 1.

Theorem 2.7** (global $\mathcal{O}(1/\sqrt{k})$ pointwise convergence rate of Algorithm 1).**

Under the Assumption ${\bf(A)}$ on Algorithm 1, let $\eta>0$ and $q(\cdot)$ be as in (25) and (39), respectively, and let $d_{0}$ denote the distance of $z_{0}$ to $T^{-1}(0)$. Assume that $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$. Then, for every $k\geq 1$, there exists $i\in\{1,\dots,k\}$ such that

[TABLE]

Proof.

Let $z^{*}\in T^{-1}(0)$ be such that $d_{0}=\|{z_{0}-z^{*}}\|$. It follows from Corollary 2.6 that, for every $k\geq 1$, there exists $i\in\{1,\dots,k\}$ such that

[TABLE]

which combined with the assumption $\lambda_{i}\geq\underline{\lambda}>0$ and (22), and after some simple algebraic manipulations, yields the desired result. ∎

Remarks.

• (i)

Theorem 2.7 provides a global $\mathcal{O}(1/\sqrt{k})$ pontwise convergence rate and ensures, in particular, that for given tolerances $\rho,\epsilon>0$, Algorithm 1 finds a triple $(z,v,\varepsilon)$ satisfying (20) after performing at most

[TABLE]

iterations.

• (ii)

If $\alpha=0$ and $\tau=1$, in which case Algorithm 1 reduces to the HPE method of Solodov and Svaiter, then it follows that Theorem 2.7 reduces to [30, Theorem 4.4(a)].

• (iii)

Analogous global $\mathcal{O}(1/\sqrt{k})$ pontwise convergence rates were also obtained in [15, 16] for inertial-type algorithms for variational inequality and convex optimization problems.

In order to study the ergodic iteration-complexity of Algorithm 1, we need to define the following.

The aggregate stepsize sequence $\{\Lambda_{k}\}$ and the ergodic sequences $\{\tilde{z}_{k}^{a}\}$, $\{\tilde{v}_{k}^{a}\}$, $\{\varepsilon_{k}^{a}\}$ associated to $\{\lambda_{k}\}$ and $\{\tilde{z}_{k}\}$, $\{v_{k}\}$, and $\{\varepsilon_{k}\}$ are, respectively, for $k\geq 1$,

[TABLE]

Next we study the ergodic iteration-complexity of Algorithm 1 under the assumption that $\alpha_{k}\equiv\alpha$ in (21).

Theorem 2.8** (global $\mathcal{O}(1/k)$ ergodic convergence rate of Algorithm 1).**

Under the Assumption ${\bf(A)}$ on Algorithm 1 and, additionally, the assumption that $\alpha_{k}\equiv\alpha$, let $\{\tilde{z}_{k}^{a}\}$, $\{v_{k}^{a}\}$ and $\{\varepsilon_{k}^{a}\}$ be as in (51) and let $d_{0}$ denote the distance of $z_{0}$ to $T^{-1}(0)$. Let also $\eta>0$ and $q(\cdot)$ be as in (25) and (39), respectively, and assume that $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$.

Then, for all $k\geq 1$,

[TABLE]

Proof.

Let $z^{*}\in T^{-1}(0)$ be such that $d_{0}=\|{z_{0}-z^{*}}\|$. Using Algorithm 1’s definition and Lemma A.2(a) with $z=\tilde{z}_{k}^{a}$ we find, for all $j\geq 1$,

[TABLE]

On the other hand, Lemma 2.2 yields

[TABLE]

which, in turn, combined with (55) gives, for all $j\geq 1$,

[TABLE]

where the sequence $\{\delta_{j}\}$ is as in (30). Summing the latter inequality over all $j=1,\dots,k$ and using (51) as well as the assumption $\alpha_{k}\equiv\alpha$, we obtain

[TABLE]

which combined with the definition of $\{\delta_{j}\}$ and (40) yields (recall that $z_{0}=z_{-1}$)

[TABLE]

where we have also used the inequality $\|{a}\|^{2}-\|{b}\|^{2}\leq 2\|{a}\|\|{a-b}\|$ for all $a,b\in\mathcal{H}$. Now, define

[TABLE]

From Corollary 2.6, the first definition in (57), (51), (23) and the convexity of $\|{\cdot}\|^{2}$ we find

[TABLE]

[TABLE]

and

[TABLE]

From (58), (59), the convexity of $\|\cdot\|^{2}$ and the inequality $\|{a-b}\|^{2}\leq 2\left(\|{a}\|^{2}+\|{b}\|^{2}\right)$ (for all $a,b\in\mathcal{H}$), we find

[TABLE]

Using the above inequality and (2) we obtain, for all $\ell\geq 0$,

[TABLE]

Hence, (2), (58) with $\ell=0,k-1$ and $j=k$, and (2) with $\ell=0,k-1$ yield

[TABLE]

which, combined with the assumption $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$, clearly finishes the proof of (54).

Now note that using (23), (21) and the assumption $\alpha_{k}\equiv\alpha$ we find

[TABLE]

Summing the above identity over $j=1,\dots,k$ and using (51) and (58) with $\ell=0$ and $j=k-1,k$ we find (recall that $z_{0}=z_{-1}$)

[TABLE]

which, combined with the assumption $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$, yields (53). To finish the proof of the theorem, note that (52) is a direct consequence of the inclusion in (22) and Theorem 1.3(a). ∎

Remark. We mention that, up to the authors knowledge, this is the first time in the literature that $\mathcal{O}(1/k)$ global convergence rates are established for inertial PP-type algorithms.

2.1 On the under-relaxed inertial proximal point method

In this subsection, we analyze the convergence and iteration-complexity of the under-relaxed inertial proximal point (PP) method (see, e.g., [1, 4]) with constant under-relaxation (Algorithm 2) for solving (19). The analysis is performed by viewing Algorithm 2 within the framework of Algorithm 1, for which asymptotic convergence and iteration-complexity were obtained in Theorems 2.5, 2.7 and 2.8.

Algorithm 2**.**

Under-relaxed inertial proximal point method for solving (19)**

Input: $z_{0}=z_{-1}\in\mathcal{H}$ and $0\leq\alpha<1$ and $0<\tau\leq 1$.

1:

for $k=1,2,\dots$, do

2:

Choose $\alpha_{k-1}\in[0,\alpha]$ and define

$\displaystyle w_{k-1}:=z_{k-1}+\alpha_{k-1}(z_{k-1}-z_{k-2}).$

(62)

3:

Compute

$\displaystyle\tilde{z}_{k}=(\lambda_{k}T+I)^{-1}w_{k-1}.$

(63)

4:

Define

$\displaystyle z_{k}:=\tau\tilde{z}_{k}+(1-\tau)w_{k-1}.$

(64)

Proposition 2.9**.**

Algorithm 2* is a special instance of Algorithm 1 with $\sigma=0$ in the Input, in which case $\varepsilon_{k}=0$ and $v_{k}=(w_{k-1}-\tilde{z}_{k})/\lambda_{k}\in T(\tilde{z}_{k})$ for all $k\geq 1$.*

Proof.

The proof follows from the well-known fact that $\tilde{z}=(\lambda T+I)^{-1}w$ if and only if $v:=(w-\tilde{z})/\lambda\in T(\tilde{z})$ and Algorithms 2 and 1’s definitions. ∎

Theorem 2.10** (convergence and iteration-complexity of Algorithm 2).**

Under the Assumption (A) with $\sigma=0$ on Algorithm 2, let $\{z_{k}\}$, $\{v_{k}\}$, $\{\tilde{z}_{k}\}$ and $\{\lambda_{k}\}$ be generated by Algorithm 2 and let the ergodic sequences $\{\tilde{z}_{k}^{a}\}$, $\{v_{k}^{a}\}$ and $\{\varepsilon_{k}^{a}\}$ be as in (51). Let also $q(\cdot)$ be as in (39) and let $d_{0}$ denote the distance of $z_{0}$ to $T^{-1}(0)$. Assume that $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$. Then, the following statements hold:

• (a)

The sequence $\{z_{k}\}$ converges weakly to a solution of the monotone inclusion problem (19).

• (b)

For all $k\geq 1$, there exists $i\in\{1,\dots,k\}$ such that

[TABLE]

• (c)

If, additionally, $\alpha_{k}\equiv\alpha$, then, for all $k\geq 1$,

[TABLE]

Proof.

The results in (a), (b) and (c) follow directly from Proposition 2.11 and Theorems 2.5, 2.7 and 2.8. ∎

Proposition 2.11**.**

The following statements hold:

• (a)

Algorithm 2* is a special instance of Algorithm 1 with $\sigma=0$ in the Input, in which case $\varepsilon_{k}\equiv 0$, $z_{k}\equiv\tilde{z}_{k}$ and $v_{k}=(w_{k-1}-z_{k})/\lambda_{k}\in T(z_{k})$.*

• (b)

The assertions of Theorems 2.5, 2.7 (Eqs. (48) and (49))* and 2.8 are valid, with $\sigma=0$, for Algorithm 2.*

Proof.

(a) This follows from the well-known fact that $\tilde{z}=(\lambda T+I)^{-1}w$ if and only if $v:=(w-\tilde{z})/\lambda\in T(\tilde{z})$ and Algorithms 2 and 1’s definitions.

(b) This follows trivially from Item (a). Note that, since $\varepsilon_{k}\equiv 0$, it follows that in this case (50) is irrelevant. ∎

Proposition 2.12**.**

Let $\{\tilde{z}_{k}\}$, $\{w_{k}\}$ and $\{\lambda_{k}\}$ be generated by Algorithm 2 and define, for all $k\geq 1$,

[TABLE]

The triple $(\tilde{z}_{k},v_{k},\varepsilon_{k})$ and $\lambda_{k}>0$ satisfy condition (22) with $\sigma=0$ and condition (23). As a consequence, it follows that Algorithm 2 is a special instance of Algorithm 1 with input $(\alpha,\sigma:=0,\tau)$.

Theorem 2.13** (weak convergence of Algorithm 2).**

Let $\{z_{k}\}$, $\{\alpha_{k}\}$ and $\{\lambda_{k}\}$ be generated by Algorithm 2 with input $(\alpha,\beta,\tau)\in[0,1[\times]0,1[\times]0,1]$ such that

[TABLE]

where

[TABLE]

Assume that $\{\alpha_{k}\}$ is nondecreasing and $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$. Then,

[TABLE]

As a consequence, under the above assumptions the sequence $\{z_{k}\}$ converges weakly to a solution of the monotone inclusion problem (19).

3 Inertial under-relaxed forward-backward and Tseng’s modified forward-backward methods

Consider the structured monotone inclusion problem (2), i.e., the problem of finding $z\in\mathcal{H}$ such that

[TABLE]

where $F:D(F)\subset\mathcal{H}\to\mathcal{H}$ is point-to-point monotone and $B:\mathcal{H}\rightrightarrows\mathcal{H}$ is a (set-valued) maximal monotone operator for which $T^{-1}(0)\neq\emptyset$ (precise assumption on $F$ and $B$ will be stated later).

In this section, we study the convergence and iteration-complexity of inertial (under-relaxed) versions of the forward-backward and Tseng’s modified forward-backward methods (13) and (16), respectively, for solving (73), by viewing them within the framework of Algorithm 1, for which asymptotic convergence and iteration-complexity were studied in Section 2.

3.1 An inertial under-relaxed Tseng’s modified forward-backward method

In this subsection, we consider the monotone inclusion problem (73) where the following assumptions are assumed to hold:

• (C1)

$F:D(F)\subset\mathcal{H}\to\mathcal{H}$ is monotone and $L$-Lipschitz continuous on a (nonempty) closed convex set $\Omega$ such that $D(B)\subset\Omega\subset D(F)$, i.e., $F$ is monotone on $\Omega$ and there exists $L\geq 0$ such that

[TABLE]

• (C2)

$B$ is a (set-valued) maximal monotone operators on $\mathcal{H}$.

• (C3)

The solution set of (73) is nonempty.

We mention that it was proved in [29, Proposition A.1] that under assumptions $(C1)$–$(C3)$ the operator $T(\cdot)$ defined in (73) is maximal monotone, which guarantee that (73) is a special instance of (19). In particular, it follows that Algorithm 1 can be used to solving the structured monotone inclusion (73).

As we mentioned earlier, in this subsection, we shall study the convergence and iteration-complexity of the following inertial under-relaxed version of the Tseng’s modified forward-backward method for solving (73).

Algorithm 3**.**

An inertial under-relaxed Tseng’s modified forward-backward method for solving (73)**

Input: $z_{0}=z_{-1}\in\mathcal{H}$, $0\leq\alpha<1$, $0<\sigma<1$ and $0<\tau\leq 1$.

1:

for $k=1,2,\dots$, do

2:

Choose $\alpha_{k-1}\in[0,\alpha]$ and define

$\displaystyle w_{k-1}:=z_{k-1}+\alpha_{k-1}(z_{k-1}-z_{k-2}).$

3:

Choose $\lambda_{k}\in]0,\sigma/L]$, let $w^{\prime}_{k-1}=P_{\Omega}(w_{k-1})$ and compute

$\displaystyle\tilde{z}_{k}=(\lambda_{k}B+I)^{-1}(w_{k-1}-\lambda_{k}F(w^{\prime}_{k-1})),$

$\displaystyle\hat{z}_{k}=\tilde{z}_{k}-\lambda_{k}\left(F(\tilde{z}_{k})-F(w^{\prime}_{k-1})\right).$

4:

Define

$\displaystyle z_{k}:=(1-\tau)w_{k-1}+\tau\hat{z}_{k}.$

Remarks.

• (i)

Algorithm 3 reduces to the Tseng’s modified forward-backward method [41] for solving (73) if $\alpha=0$ and $\tau=1$, in which case $w_{k-1}=z_{k-1}$ and $z_{k}=\hat{z}_{k}$.

• (ii)

An inertial Tseng’s modified forward-backward-type method (based on a different mechanism of iteration) was proposed and studied in [9]. The proposed Tseng’s modified forward-backward type method in the latter reference tends to suffer from similar limitations as the inertial HPE-type method proposed in [9], as we discussed in the third remark following Assumption ${\bf(A)}$. Moreover, in contrast to this paper which performs the iteration-complexity analysis of Algorithm 3 (see Theorem 3.2), [9] has focused on asymptotic convergence.

Since the proof of the next proposition follows the same outline of [30, Proposition 6.1], we omit it here.

Proposition 3.1**.**

Let $\{w_{k}\}$, $\{w^{\prime}_{k}\}$, $\{z_{k}\}$, $\{\alpha_{k}\}$, $\{\tilde{z}_{k}\}$ and $\{\lambda_{k}\}$ be generated by Algorithm 3 and define

[TABLE]

Then, the sequences $\{w_{k}\}$, $\{z_{k}\}$, $\{\alpha_{k}\}$, $\{\tilde{z}_{k}\}$, $\{v_{k}\}$, $\{\varepsilon_{k}\}$ and $\{\lambda_{k}\}$ satisfy the conditions (21)–(23) in Algorithm 1. As a consequence, it follows that Algorithm 3 is a special instance of Algorithm 1 for solving (73).

Next we present the convergence and iteration-complexity of Algorithm 3 under the Assumption ${\bf(A)}$ on the Input $(\alpha,\sigma,\tau)\in[0,1[\times]0,1[\times]0,1]$ and on the sequence $\{\alpha_{k}\}$. We also mention that the observations regarding the parameter $\tau$ in the third remark following Assumption ${\bf(A)}$ obviously apply to Algorithm 3.

Theorem 3.2** (convergence and iteration-complexity of Algorithm 3).**

Under the Assumption ${\bf(A)}$ on $(\alpha,\sigma,\tau)\in[0,1[\times]0,1[\times]0,1]$ and $\{\alpha_{k}\}$, let $\{z_{k}\}$, $\{\tilde{z}_{k}\}$ and $\{\lambda_{k}\}$ be generated by Algorithm 3, let $\{v_{k}\}$ and $\{\varepsilon_{k}\}$ be as in (74) and let the ergodic sequences $\{\tilde{z}_{k}^{a}\}$, $\{v_{k}^{a}\}$ and $\{\varepsilon_{k}^{a}\}$ be as in (51). Let also $\eta>0$ and $q(\cdot)$ be as in (25) and (39), respectively, let $d_{0}$ denote the distance of $z_{0}$ to $(F+B)^{-1}(0)$ and assume that $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$. Then, the following statements hold:

• (a)

The sequence $\{z_{k}\}$ converges weakly to a solution of the monotone inclusion problem (73).

• (b)

For all $k\geq 1$, there exists $i\in\{1,\dots,k\}$ such that

[TABLE]

• (c)

If, additionally, $\alpha_{k}\equiv\alpha$, then, for all $k\geq 1$,

[TABLE]

Proof.

The proof follows directly from Proposition 3.1 and Theorems 2.5, 2.7 and 2.8. ∎

Remarks.

• (i)

Itens (b) and (c) ensure, respectively, global pointwise $\mathcal{O}(1/\sqrt{k})$ and ergodic $\mathcal{O}(1/k)$ convergence rates for Algorithm 3. On the other hand, note that the inclusion in (LABEL:eq:seg088) is potentially weaker than the corresponding one in (75).

• (ii)

If $\lambda_{k}\equiv\sigma/L$ in Step 3 of Algorithm 3, in which case $\underline{\lambda}=\sigma/L$, then $d_{0}/\underline{\lambda}$ in (75) and (LABEL:eq:seg088) can be replaced by $d_{0}L/\sigma$. In this case, Item (b) gives that for a given tolerance $\rho>0$, Algorithm 3 finds a pair $(z,v)$ such that (cf. (20))

[TABLE]

in at most

[TABLE]

iterations, an analogous remark also holding for Item (c).

3.2 On the inertial under-relaxed forward-backward method

Similarly to Subsection 3.1, in this subsection, we consider the monotone inclusion problem (73) but now assume the following: (C2) and (C3) as in Subsection 3.1 and instead of (C1):

• (C1*′*)

$F:\mathcal{H}\to\mathcal{H}$ is $(1/L)-$cocoercive, i.e., there exists $L>0$ such that

[TABLE]

We observe that it follows from (77) that $F$ is, in particular, $L$–Lipschitz continuous.

Algorithm 4**.**

Inertial under-relaxed forward-backward method for solving (73)**

Input: $z_{0}=z_{-1}\in\mathcal{H}$, $0\leq\alpha<1$, $0<\sigma<1$ and $0<\tau\leq 1$.

1:

for $k=1,2,\dots$, do

2:

Choose $\alpha_{k-1}\in[0,\alpha]$ and define

$\displaystyle w_{k-1}:=z_{k-1}+\alpha_{k-1}(z_{k-1}-z_{k-2}).$

3:

Choose $\lambda_{k}\in]0,2\sigma^{2}/L]$ and compute

$\displaystyle\tilde{z}_{k}=(\lambda_{k}B+I)^{-1}(w_{k-1}-\lambda_{k}F(w_{k-1})).$

4:

Define

$\displaystyle z_{k}:=(1-\tau)w_{k-1}+\tau\tilde{z}_{k}.$

Remarks.

• (i)

If $\alpha=0$ and $\tau=1$, then it follows that Algorithm 4 reduces to the forward-backward [22, 33] method for solving (73).

• (ii)

Inertial versions of the forward-backward method were previously proposed and studied in [32], [23] and [3]. Asymptotic convergence of the forward-backward method proposed in [23] was proved in the latter reference, in particular, under the assumption: $0\leq\alpha_{k-1}\leq\alpha_{k}\leq\alpha<1$, for all $k\geq 1$, and

[TABLE]

for some $\varepsilon\in]0,(9-4\gamma)/(2\gamma)[$, where $\gamma\in(0,2)$ and $\lambda_{k}\equiv\lambda:=\gamma/L$ ($\gamma=2\sigma^{2}$ in the notation of the present paper). The apparent limitation of this approach is that $\alpha\to 0$ if $\gamma\to 2$, i.e., the inertial effect degenerates for large values of the stepsize (see Fig. 1 in [23]). This contrasts to the approach proposed in this paper, where the under-relaxation parameter $\tau\in[0,1[$ is crucial to allowing $\alpha$ sufficiently close to 1, even for large stepsize values, i.e., when $\sigma\approx 1$ (see Assumption (A) and part of the discussion in the third remark following it).

• (iii)

Algorithm 4 is a special instance (with constant relaxation) of the RIFB algorithm in [3]. We refer the reader to [3] (see, e.g., Theorems 3.8 and 3.15, and Remark 3.13) for a comprehensive discussion of the interplay and benefits of inertia and relaxation.

Next proposition shows that Algorithm 4 is also a special instance of Algorithm 1 for solving (73). Since the proof follows the same outline of [39, Proposition 5.3], we omit it here too.

Proposition 3.3**.**

Let $\{\tilde{z}_{k}\}$, $\{z_{k}\}$, $\{w_{k}\}$ and $\{\lambda_{k}\}$ be generated by Algorithm 4, let $T=F+B$ be as in (73) and define, for all $k\geq 1$,

[TABLE]

Then, the following hold for all $k\geq 1$:

[TABLE]

As a consequence of (LABEL:eq:0607) and Algorithm 4’s definition, it follows that Algorithm 4 is a special instance of Algorithm 1 for solving (73).

We finish this section by presenting the convergence and iteration-complexity of Algorithm 4, which are a direct consequence of Proposition 3.3 and Theorems 2.5, 2.7 and 2.8. We mention that analogous remarks to those made in the Remarks following Theorem 3.2 also apply here.

Theorem 3.4** (convergence and iteration-complexity of Algorithm 4).**

Under the Assumption ${\bf(A)}$ on $(\alpha,\sigma,\tau)\in[0,1[\times]0,1[\times]0,1]$ and $\{\alpha_{k}\}$, let $\{z_{k}\}$, $\{\tilde{z}_{k}\}$ and $\{\lambda_{k}\}$ be generated by Algorithm 4, let $\{v_{k}\}$ and $\{\varepsilon_{k}\}$ be as in (78) and let the ergodic sequences $\{\tilde{z}_{k}^{a}\}$, $\{v_{k}^{a}\}$ and $\{\varepsilon_{k}^{a}\}$ be as in (51). Let also $\eta>0$ and $q(\cdot)$ be as in (25) and (39), respectively, let $d_{0}$ denote the distance of $z_{0}$ to $(F+B)^{-1}(0)$ and assume that $\lambda_{k}\geq\underline{\lambda}>0$ for all $k\geq 1$. Then, the following statements hold:

• (a)

The sequence $\{z_{k}\}$ converges weakly to a solution of the monotone inclusion problem (73).

• (b)

For all $k\geq 1$, there exists $i\in\{1,\dots,k\}$ such that

[TABLE]

• (c)

If, additionally, $\alpha_{k}\equiv\alpha$, then, for all $k\geq 1$,

[TABLE]

4 Concluding remarks

In this paper, we proposed and studied the asymptotic convergence and iteration-complexity of an inertial under-relaxed HPE-type method. As applications, we proposed and/or studied inertial (under-relaxed) versions of the Tseng’s modified forward-backward and forward-backward methods for solving structured monotone inclusion problems with either Lipschitz continuous or cocoercive operators. All the proposed and/or studied algorithms, namely Algorithms 1, 2, 3 and 4 potentially benefit from a specific policy for choosing the upper bound on the sequence of extrapolation parameters, in which case (under) relaxation plays a central role (see Assumption ${\bf(A)}$ and Theorems 2.7, 2.8, 3.2 and 3.4); see also the recent work [3] of Attouch and Cabot. We also emphasize that, up to the authors knowledge, this is the first time in the literature that nonasymptotic global convergence rates (iteration-complexity) are provided for inertial HPE-type methods, in particular for the proposed inertial Tseng’s modified forward-backward method.

Appendix A Auxiliary results

Lemma A.1**.**

([39, Lemma 2.2])*

Let $F:\mathcal{H}\to\mathcal{H}$ be $\gamma$–cocoercive, for some $\gamma>0$, and let $w,\widetilde{z}\in\mathcal{H}$. Then,*

[TABLE]

Next lemma was proved in [40, Lemma 2.1]. Here, we present a short and direct proof for the convenience of the reader.

Lemma A.2** (Svaiter).**

Let $\tilde{z},v,w\in\mathcal{H}$ and $\lambda>0$, $\varepsilon\geq 0$ and $\sigma\in[0,1[$ be such that

[TABLE]

Let $\tau\in[0,1]$ and define $z_{+}:=w-\tau\lambda v$. Then, the following hold:

• (a)

For any $z\in\mathcal{H}$,

[TABLE]

• (b)

For any $z^{*}\in T^{-1}(0)$,

[TABLE]

Proof.

(a) Using the inequality in (82) and some algebraic manipulations we find, for any $z\in\mathcal{H}$,

[TABLE]

The fact that $z_{+}=(1-\tau)w+\tau(w-\lambda v)$ and (18) yield

[TABLE]

Multiplying (83) by $\tau\in[0,1]$ and using the latter identity we obtain the desired inequality in (a).

(b) This is a direct consequence of Item (a), (17), the inclusion in (82) and the fact that $0\in T(z^{*})$. ∎

Lemma A.3**.**

For any $\sigma\in[0,1[$, the inverse function of the scalar map

[TABLE]

is given by

[TABLE]

Lemma A.4** (Opial).**

Let $\emptyset\neq\Omega\subset\mathcal{H}$ and $\{z_{k}\}$ be a sequence in $\mathcal{H}$ such that $\lim_{k\to\infty}\,\|{z_{k}-z^{*}}\|$ exist for every $z^{*}\in\Omega$. If every (sequential) weak cluster point of $\{z_{k}\}$ belongs to $\Omega$, then $\{z_{k}\}$ converges weakly to a point in $\Omega$.

The following lemma was essentially proved by Alvarez and Attouch in [2, Theorem 2.1].

Lemma A.5**.**

Let the sequences $\{\varphi_{k}\}$, $\{s_{k}\}$, $\{\alpha_{k}\}$ and $\{\delta_{k}\}$ in $[0,+\infty[$ and $\alpha\in\mathbb{R}$ be such that $\varphi_{0}=\varphi_{-1}$, $0\leq\alpha_{k-1}\leq\alpha<1$ and

[TABLE]

The following hold:

1. (a)

For all $k\geq 1$,

[TABLE] 2. (b)

If $\sum^{\infty}_{k=1}\delta_{k}<+\infty$, then $\lim_{k\to\infty}\,\varphi_{k}$ exist, i.e., the sequence $\{\varphi_{k}\}$ converges to some element in $[0,\infty[$.

Proof.

It was proved in [2, Theorem 2.1] that $\mathcal{M}:=(1-\alpha)^{-1}\sum_{j=1}^{k}\delta_{j}\geq\sum_{j=1}^{k}\,[\varphi_{j}-\varphi_{j-1}]_{+}$, where $[\cdot]_{+}=\max\{\cdot,0\}.$ Using this, the assumptions $\varphi_{0}=\varphi_{-1}$, $0\leq\alpha_{k-1}\leq\alpha<1$ and (84), and some algebraic manipulations we find

[TABLE]

which proves (a). To finish the proof of the lemma, note that (b) was proved inside the proof of [2, Theorem 2.1]. ∎