Fast Reflected Forward-Backward algorithm: achieving fast convergence rates for convex optimization with linear cone constraints

TL;DR

The paper introduces a new optimization algorithm that achieves faster convergence for convex optimization problems with linear constraints.

Contribution

The novel Fast Reflected Forward-Backward algorithm incorporates Nesterov momentum and a correction term to improve convergence rates.

Findings

The algorithm achieves a convergence rate of o(1/k) for discrete velocity and tangent residual.

It outperforms existing methods in solving minimax problems with nonsmooth regularizers.

Numerical experiments confirm the improved convergence behavior of the proposed algorithm.

Abstract

In this paper, we derive a Fast Reflected Forward-Backward (Fast RFB) algorithm to solve the problem of finding a zero of the sum of a maximally monotone operator and a monotone and Lipschitz continuous operator in a real Hilbert space. Our approach extends the class of reflected forward-backward methods by introducing a Nesterov momentum term and a correction term, resulting in enhanced convergence performance. The iterative sequence of the proposed algorithm is proven to converge weakly, and the Fast RFB algorithm demonstrates impressive convergence rates, achieving \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$o\left( \frac{1}{k} \right) $$\end{document}o1k as \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$k \rightarrow +\infty $$\end{document}k→+∞ for both the discrete velocity and the tangent residual at the last-iterate. When applied to minimax problems with a smooth coupling term and nonsmooth convex regularizers, the resulting algorithm demonstrates significantly improved convergence properties compared to the current state of the art in the literature. For convex optimization problems with linear cone constraints, our approach yields a fully splitting primal-dual algorithm that ensures not only the convergence of iterates to a primal-dual solution, but also a last-iterate convergence rate of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$o\left( \frac{1}{k} \right) $$\end{document}o1k as \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$k \rightarrow +\infty $$\end{document}k→+∞ for the objective function value, feasibility measure, and complementarity condition. This represents the most competitive theoretical result currently known for algorithms addressing this class of optimization problems. Numerical experiments are performed to illustrate the convergence behavior of Fast RFB.

Figures

The impact of the parameter *c* on the convergence behavior of the discrete velocity, the tangent residual, the primal-dual gap, and the function values when $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =3$$\end{document}$

The impact of the parameter *c* on the convergence behavior of the discrete velocity, the tangent residual, the primal-dual gap, and the function values when $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =5$$\end{document}$ Fig. 3The impact of the parameter *c* on the convergence behavior of the discrete velocity, the tangent residual, the primal-dual gap, and the function values when $\documentclass[12pt

A comparison of various methods in terms of discrete velocity, tangent residual, primal-dual gap, and function values for $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=1000$$\end{document}$

A comparison of various methods in terms of the norm of the residual $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\Vert V \left( x_{k} , \lambda _{k} \right) \right\Vert $$\end{document}$ for $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \beg

Full text

Introduction

Problem formulation

In recent years, there has been a significant surge in research on minimax problems, primarily driven by their emerging applications in machine learning and robust optimization. Notable instances include generative adversarial networks (GANs) [2, 27], which use minimax frameworks to improve data generation and adversarial training methods, and distributionally robust optimization [36, 37], which employs minimax principles to ensure model performance under varying distributional shifts. Beyond these areas, minimax problems have also found applications in online learning [7], where they help develop algorithms that adapt to dynamic environments, and in reinforcement learning [4, 20], contributing to more efficient decision-making processes. This highlights the versatility and fundamental significance of minimax approaches across a wide range of domains.

Consider a minimax problem of the form

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{x\in {\mathcal {X}}}\max _{\lambda \in {\mathcal {Y}}} \Psi \left( x , \lambda \right) {:=} f(x) + \Phi \left( x , \lambda \right) - g(\lambda ), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {X}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ {\mathcal {Y}}$$\end{document} are two real Hilbert spaces, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f :{\mathcal {X}}\rightarrow {\mathbb {R}}\cup \left\{ + \infty \right\} $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g :{\mathcal {Y}}\rightarrow {\mathbb {R}}\cup \left\{ + \infty \right\} $$\end{document} are proper, convex, and lower semicontinuous functions, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi :{\mathcal {X}}\times {\mathcal {Y}}\rightarrow {\mathbb {R}}$$\end{document} is a convex-concave and differentiable coupling function with Lipschitz continuous gradient. We are interested in finding saddle points of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Psi $$\end{document} , which are pairs \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{*} , \lambda _{*} \right) \in {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} fulfilling

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Psi (x_{*},\lambda ) \leqslant \Psi \left( x_{*} , \lambda _{*} \right) \leqslant \Psi (x,\lambda _{*}) \text { for every } \left( x , \lambda \right) \in {\mathcal {X}}\times {\mathcal {Y}}. \end{aligned}$$\end{document}

The minimax setting (1.1) is highly versatile, providing a general framework for studying a wide range of problems, including unconstrained composite convex minimization, mixed variational inequalities, and constrained convex minimization problems, see [12, 17, 19, 24, 38, 45, 51, 59].

An element \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{*} , \lambda _{*} \right) \in {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} is a saddle point of (1.1) if and only if it is a solution of the system of optimality conditions

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{pmatrix}0 \\ 0 \end{pmatrix} \in \begin{pmatrix} \partial f(x) \\ \partial g(\lambda ) \end{pmatrix} + \begin{pmatrix} \nabla _{x} \Phi \left( x, \lambda \right) \\ -\nabla _{\lambda } \Phi \left( x, \lambda \right) \end{pmatrix}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\partial f : {\mathcal {X}}\rightarrow 2^{{\mathcal {X}}} $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\partial g : {\mathcal {Y}}\rightarrow 2^{{\mathcal {Y}}}$$\end{document} denote the convex subdifferentials of f and g, respectively.

This motivates us to develop solution methods for solving the following monotone inclusion problem

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 0\in M(z) + F(z), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {H}}$$\end{document} is a real Hilbert space, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M :{\mathcal {H}}\rightarrow 2^{{\mathcal {H}}}$$\end{document} is a (possibly set-valued) maximally monotone operator and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F :{\mathcal {H}}\rightarrow {\mathcal {H}}$$\end{document} is a single-valued monotone and L-Lipschitz continuous operator. We assume \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{Zer}(M + F):=\{z \in {\mathcal {H}}: 0 \in M(z) + F(z)\} \ne \emptyset $$\end{document} .

The graph of a set-valued operator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M :{\mathcal {H}}\rightarrow 2^{{\mathcal {H}}}$$\end{document} is defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{Graph}(M):=\{(z,u) \in {\mathcal {H}}\times {\mathcal {H}}: u \in M(z)\}.$$\end{document} The operator M is said to be monotone if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\langle v-u, y-z \rangle \geqslant 0$$\end{document} for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(z,u), (y,v) \in \textrm{Graph}(M)$$\end{document} . A monotone operator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M :{\mathcal {H}}\rightarrow 2^{{\mathcal {H}}}$$\end{document} is said to be maximal monotone if there exists no other monotone operator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M' :{\mathcal {H}}\rightarrow 2^{{\mathcal {H}}}$$\end{document} such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{Graph}(M) \subsetneq \textrm{Graph}(M')$$\end{document} . The convex subdifferential of a proper, convex and lower semicontinuous function defined on a real Hilbert space is a maximally monotone operator [5, 8]. For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {H}}:= {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} , and

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&M :{\mathcal {H}}\rightarrow 2^{{\mathcal {H}}}, \ \left( x , \lambda \right) \mapsto \big (\partial f \left( x \right) , \partial g\left( \lambda \right) \big ), \end{aligned}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&F :{\mathcal {H}}\rightarrow {\mathcal {H}}, \ \left( x , \lambda \right) \mapsto (\nabla _{x} \Phi \left( x, \lambda \right) ,-\nabla _{\lambda } \Phi \left( x, \lambda \right) ), \end{aligned}$$\end{document}

both maximally monotone operators, the system of optimality conditions (1.2) reduces to the monotone inclusion problem (1.3).

If \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M:=N_C$$\end{document} with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C \subseteq {\mathcal {H}}$$\end{document} being a nonempty convex and closed set, (1.3) becomes

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 0\in N_{C}(z) + F(z). \end{aligned}$$\end{document}

This nothing else than the variational inequality problem

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {find}\,\,\, z\in C\,\,\, \text {such that}\,\,\, \langle F(z), u-z\rangle \geqslant 0 \,\,\,\text {for all}\,\,\, u\in C, \end{aligned}$$\end{document}

that has been extensively studied in the literature – see, for instance, [22, 25, 26, 28, 48, 58].

In the following, we will review several methods for solving monotone inclusions of the form (1.3), as well as saddle point problems of the form (1.1).

Numerical methods for monotone inclusions with monotone and Lipschitz continuous operators

In this subsection, we will survey the most prominent numerical methods for solving (1.3), excluding algorithms that rely on F being cocoercive and thus belong to the framework of the classical Forward-Backward (FB) method.

The Extragradient (EG) method was introduced by Korpelevich [29] and Antipin [1] and is one of the first and most famous algorithms for solving (1.2). Based on the EG method, Tran-Dinh [52] has recently developed the following algorithm to solve (1.3)

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ (\forall k \geqslant 0) \ \left\{ \begin{aligned} w_{k}&= J_{\frac{\gamma }{\eta }M} \left( z_{k} - \frac{\gamma }{\eta } F(z_{k}) \right) , \\ z_{k+1}&= J_{\gamma M} \left( z_{k} - \gamma F(w_{k}) \right) , \end{aligned}\right. \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J_{\gamma M} {:=} \left( \textrm{Id}+ \gamma M \right) ^{-1} : {\mathcal {H}}\rightarrow {\mathcal {H}}$$\end{document} denotes the resolvent of M with parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma >0$$\end{document} , which plays the role of a step size, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta $$\end{document} is a scaling factor. The scaling factor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta $$\end{document} allows to provide a unified framework for different methods. For example, the classical EG method is obtained from (1.7) for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta = 1$$\end{document} , M given by (1.4), with f and g the indicator functions of two nonempty, convex and closed subsets of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {X}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {Y}}$$\end{document} , respectively, F given by (1.5), and the step size required to satisfy \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0<\gamma <\frac{1}{L}$$\end{document} (see, also, [23]). Assuming that M is maximally 3-cyclically monotone, the author demonstrated convergence for the iterates generated by (1.7), and that the tangent residual achieves best-iterate and last-iterate convergence rates of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{\sqrt{k}} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} .

By tangent residual we mean the quantity

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_{tan}(z) {:=} \textrm{dist}(0, M(z) + F(z)) = \inf _{\xi \in M(z)} \left\Vert \xi +F(z) \right\Vert . \end{aligned}$$\end{document}

Since (1.3) can be rewritten as the following fixed point problem

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} z = J_{\gamma M} \left( z - \gamma F \left( z \right) \right) , \end{aligned}$$\end{document}

another widely used measure of optimality is the so-called fixed-point residual

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_{fix}(z) {:=} \left\Vert z - J_{\gamma M} \left( z - \gamma F \left( z \right) \right) \right\Vert . \end{aligned}$$\end{document}

However, the derivation of convergence rates in terms of the tangent residual is more desirable. This is not only because it gives an upper bound on the fixed point residual, i.e. (see, for instance, [14])

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 0 \leqslant r_{fix}(z) \leqslant r_{tan}(z) \quad \forall z \in {\mathcal {H}}, \end{aligned}$$\end{document}

but also because it allows the convergence rates to be transferred to function values when applied to convex optimization problems and minimax problems such as (1.1).

In order to reduce the computational cost of the EG method caused by evaluating the operator F at two different points in each iteration, Popov introduced in [47] the Optimistic Gradient Descent Ascent (OGDA) method, which requires only one evaluation of the operator per iteration. Its extension to solving (1.3) provided in [52] is as follows

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ (\forall k \geqslant 1) \ \left\{ \begin{aligned} w_{k}&= J_{\frac{\gamma }{\eta }M} \left( z_{k} - \frac{\gamma }{\eta } F(w_{k-1}) \right) , \\ z_{k+1}&= J_{\gamma M} \left( z_{k} - \gamma F(w_{k}) \right) , \end{aligned}\right. \end{aligned}$$\end{document}

and differs from (1.7) in that in the first block \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F(w_{k-1})$$\end{document} replaces \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F(z_k)$$\end{document} , see also [10]. The classical OGDA method is obtained from (1.9) in the same way as described above, as is the EG method from (1.7), but with the step size required to satisfy \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0<\gamma <\frac{1}{{2L}}$$\end{document} . Assuming that M is maximally 3-cyclically monotone, in [52], convergence for the iterates generated by (1.9), and that the tangent residual achieves best-iterate and last-iterate convergence rates of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{\sqrt{k}} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} are shown.

Both the EG and OGDA methods require two evaluations of the resolvent \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J_{\gamma M}$$\end{document} per iteration. To address this computational demand, Tseng [56] proposed the Forward-Backward-Forward (FBF) method, inspired by the EG framework. The FBF method reduces the per-iteration complexity by requiring only a single evaluation of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J_{\gamma M}$$\end{document} per iteration, making it particularly advantageous when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J_{\gamma M}$$\end{document} is expensive to compute. Its iterative scheme is as follows

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ (\forall k \geqslant 0) \ \left\{ \begin{aligned} w_{k}&= J_{\gamma M} \left( z_{k} - \gamma F(z_{k}) \right) , \\ z_{k+1}&= w_{k} - \gamma F(w_{k}) + \gamma F(z_{k}), \end{aligned}\right. \end{aligned}$$\end{document}

and generates a sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(z_k)_{k \geqslant 0}$$\end{document} that converges to a solution of (1.3) for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ 0< \gamma < \frac{1}{L} $$\end{document} . If \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M=0$$\end{document} , the FBF method reduces to the classical EG method [29]. For cases where M represents the convex subdifferential of a proper, convex and lower semicontinuous function, an ergodic convergence rate of  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k} \right) $$\end{document} for the restricted gap function as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} was established in [11]. Furthermore, the best-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{\sqrt{k}} \right) $$\end{document} for the tangent residual as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} has been shown in two distinct contexts: using a potential function approach for the star-co-monotone case in [39], and through an alternative method in [52] that leverages results from [23] combined with the concept of star co-hypomonotonicity.

By replacing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F(z_{k})$$\end{document} with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F(w_{k-1})$$\end{document} in (1.10), the following Past Forward-Backward-Forward (PFBF) method is obtained

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ (\forall k \geqslant 1) \ \left\{ \begin{aligned} w_{k}&= J_{\gamma M} \left( z_{k} - \gamma F(w_{k-1}) \right) , \\ z_{k+1}&= w_{k} - \gamma F(w_{k}) + \gamma F(w_{k-1}), \end{aligned}\right. \end{aligned}$$\end{document}

which requires only one forward estimation per iteration. For this method, the best-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{\sqrt{k}} \right) $$\end{document} for the tangent residual as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} was established in [39] under the assumption that M is star-co-monotone. By interchanging the roles of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{k}$$\end{document} of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{k}$$\end{document} in (1.11) and simplifying to a single sequence, one arrives at the Forward-Reflected-Backward (FRB) method proposed by Malitsky and Tam in [41], described as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (\forall k \geqslant 1) \ z_{k+1} = J_{\gamma M} \left( z_{k} - 2 \gamma F(z_{k}) + \gamma F(z_{k-1}) \right) , \end{aligned}$$\end{document}

which converges to a solution of (1.3) provided \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0< \gamma < \frac{1}{2L}$$\end{document} . This iterative scheme can also be derived from (1.10) by reusing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F(w_{k-1})$$\end{document} instead of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F(z_{k})$$\end{document} in the first line, similar to how the OGDA method is derived from the EG method. For variational inequalities, [14] demonstrated that the FRB method achieves a last-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{\sqrt{k}} \right) $$\end{document} for the tangent residual as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} .

In [15], Cevher and Vũ proposed the following Reflected Forward-Backward (RFB) method

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (\forall k \geqslant 1) \ z_{k+1} = J_{\gamma M} \left( z_{k} - \gamma F(2 z_{k} - z_{k-1}) \right) , \end{aligned}$$\end{document}

which converges to a solution of (1.3) provided \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0< \gamma < \frac{\sqrt{2}-1}{L} $$\end{document} . In (1.13), the evaluation of F through a second forward step is circumvented by using a suitable linear combination of the iterates. If F is linear, (1.13) is equivalent to (1.12). In [52], best-iterate and last-iterate convergence rates of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{\sqrt{k}} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the tangent residual were established. For the variational inequality problem (1.6), (1.13) reduces to the Projected Reflected Gradient (PRG) method introduced in [40]. Last-iterate convergence rates for the PRG were provided in [14].

In recent years, there has been significant interest in the development of numerical methods with fast convergence properties for solving monotone inclusions. Using the performance estimation problem framework, Kim introduced in [30] an Accelerated Proximal Point (APP) method for solving (1.3) in the special case where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F \equiv 0$$\end{document} . This method achieves a convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k +\infty $$\end{document} for the fixed-point residual, thereby outperforming the classical Proximal Point method [5]. For problems without a set-valued operator ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M \equiv 0$$\end{document} in (1.3)), a Fast Optimistic Gradient Descent Ascent (Fast OGDA) method was proposed in [9]. This method, derived as a discretization of a fast-converging continuous time model, not only ensures the convergence of the iterates but also achieves a last-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$o \left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the operator norm of F.

The Extra-Anchored Gradient (EAG) method, inspired by Halpern iteration [31], has been developed to address (1.3). This algorithm achieves a last-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the tangent residual when M is maximally 3-cyclically monotone [52]. The method extends an earlier algorithm proposed by Yoon and Ryu in [60]. Building on these ideas, the Past Extra-Anchored Gradient (PEAG) method was introduced in [49], leveraging concepts from [54] and the Optimistic Gradient Descent Ascent (OGDA) method [47]. The PEAG method is designed to solve (1.3) under the assumption that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M+F$$\end{document} is co-hypomonotone. It also guarantees a last-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k} \right) $$\end{document} for the tangent residual as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} .

Also building on Halpern iteration [31], Cai and Zheng proposed in [14] an Accelerated Reflected Gradient (ARG) method. This method, which can be viewed as an acceleration of the RFB method, addresses the monotone inclusion (1.3) under the assumptions that M is maximally monotone, F is Lipschitz continuous, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M+F$$\end{document} is negatively comonotone. The ARG method achieves a convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the tangent residual.

Additional fast methods that exploit either Nesterov momentum [46] or Halpern iteration can be found in [13, 42, 43, 49]. A more detailed discussion of these approaches, along with comparisons to our proposed method, will be presented in the next section.

Numerical methods for saddle point problems with convex regularizes

In the literature, several attempts have been made to solve the saddle point problem (1.1) directly, without relying on the more general formulation (1.3).

A particularly well-studied instance of this problem involves a bilinear coupling term, namely, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi (x, \lambda ):= \langle Ax, \lambda \rangle $$\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A:{\mathcal {X}}\rightarrow {\mathcal {Y}}$$\end{document} is a linear continuous operator. In this case, the problem (1.1) reduces to

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{x \in {\mathcal {X}}}\max _{\lambda \in {\mathcal {Y}}} f(x) + \langle Ax, \lambda \rangle - g(\lambda ). \end{aligned}$$\end{document}

To address (1.14), a primal-dual approach was first proposed in [3] and further developed in [61], with convergence properties guaranteed under the assumption that f is strongly convex. Later, Chambolle and Pock [16] introduced a fully splitting primal-dual algorithm to solve (1.14) in the finite-dimensional setting. They demonstrated that the sequence of iterates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k} , \lambda _{k} \right) _{k\geqslant 0}$$\end{document} converges to a saddle point of (1.14), and they established an ergodic convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k}\right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the so-called restricted primal-dual gap. When f is strongly convex, an accelerated version of this primal-dual algorithm achieves an improved ergodic convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k^2}\right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} again with respect to the restricted primal-dual gap [17]. The Chambolle-Pock algorithm has since inspired a variety of primal-dual methods for solving (1.14), including those proposed in [17, 21, 34, 50, 53, 55, 57], to name just a few.

Compared to the bilinear case, the study of (1.1) in its general form has been less extensive. Nemirovski and Juditsky [35, 44] introduced the Mirror-Prox method to address (1.1) in the absence of regularizers ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f=g=0$$\end{document} ) achieving an ergodic convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the restricted gap function. This method was later extended in [33] to handle convex regularizers using Bregman distances, while maintaining the same ergodic convergence rate. Hamedani and Aybat [32] proposed an Accelerated Primal-Dual (APD) algorithm incorporating a Nesterov momentum term, which generalizes the Chambolle-Pock approach [16] to the saddle point problem (1.1). The APD algorithm is defined as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}\ (\forall k \geqslant 1) \ \left\{ \begin{aligned} s_{k}&= (1+ \theta _k)\nabla _{\lambda } \Phi (x_k,\lambda _k)-\theta _k \nabla _{\lambda } \Phi (x_{k-1},\lambda _{k-1}),\\ \lambda _{k+1}&= \textrm{prox}_{\sigma _k g} \left( \lambda _{k} + \sigma _k s_k \right) , \\ x_{k+1}&= \textrm{prox}_{\tau _k f} \left( x_{k} - \tau _k \nabla _{x} \Phi (x_k,\lambda _{k+1})\right) . \end{aligned}\right. \end{aligned}$$\end{document}

Under suitable conditions on the parameter sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( \tau _{k} , \sigma _{k} \right) _{k \geqslant 0}$$\end{document} , the authors proved that the iterates converge to a saddle point. Moreover, the ergodic sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\bar{x}}_{k} , {\bar{\lambda }}_{k} \right) _{k\geqslant 0}$$\end{document} satisfied \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Psi \left( {\bar{x}}_{k},\lambda _{*} \right) -\Psi \left( x_{*},{\bar{\lambda }}_{k} \right) \rightarrow 0$$\end{document} with a convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k} \right) $$\end{document} in the general convex setting and of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k^2} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi (x,\cdot )$$\end{document} is linear for each fixed x and f is strongly convex.

Recently, Chang, Yang, and Zhang [18] introduced an enhancement of the APD algorithm by employing adaptive linesearch techniques, which only assume local Lipschitz continuity for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla _x \Phi $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla _\lambda \Phi $$\end{document} . This enhanced method retains similar convergence and convergence rate properties as the original APD algorithm.

Our contribution

This paper introduces an accelerated first-order method for solving the monotone inclusion problem (1.3). The proposed approach ensures the weak convergence of iterates to a solution of (1.3) and achieves last-iterate convergence rates of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$o \left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for both the discrete velocity and the tangent residual. We demonstrate the versatility of the algorithm by applying it to the minimax problem (1.1) and convex optimization problems with linear cone constraints. Finally, we validate the theoretical results and explore the impact of algorithm parameters through comprehensive numerical experiments.

The contributions of the paper are as follows:

• We propose a Fast Reflected Forward-Backward (Fast RFB) algorithm, which incorporates a Nesterov momentum term and a correction term, for solving the monotone inclusion problem (1.3). This method requires only a single operator evaluation and one resolvent computation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J_{\gamma M}$$\end{document} per iteration. The iterative sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(z_k)_{k \geqslant 0}$$\end{document} generated by the algorithm weakly converges to a solution of (1.3). Furthermore, the Fast RFB algorithm achieves a last-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$o \left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for both the discrete velocity \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\Vert z_k - z_{k-1} \right\Vert $$\end{document} and the tangent residual \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_{\textrm{tan}} \left( z_k \right) = \textrm{dist}(0, M(z_k) + F(z_k)) = \inf _{\xi \in M(z_k)} \left\Vert \xi + F \left( z_k \right) \right\Vert $$\end{document} .
• Building on the Fast RFB method, we develop a primal-dual full-splitting algorithm for solving the saddle point problem (1.1). The proposed algorithm ensures the weak convergence of the sequence of primal-dual iterates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k},\lambda _{k} \right) _{k \geqslant 0}$$\end{document} to a saddle point. Additionally, it achieves last-iterate convergence rates of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$o \left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} or the discrete primal and dual velocities, the tangent residual, and the primal-dual gap.
• As a particular instance of the saddle problem (1.1), we apply the proposed primal-dual full splitting algorithm to solve optimization problems of the form

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \min \ &f \left( x \right) +h \left( x \right) ,\\ \text {subject to} \&Ax - b \in -{\mathcal {K}}, \end{aligned} \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {X}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {Y}}$$\end{document} are real Hilbert spaces, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {K}}$$\end{document} is a nonempty, convex and closed cone in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {Y}}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f :{\mathcal {X}}\rightarrow {\mathbb {R}}\cup \left\{ + \infty \right\} $$\end{document} is a proper, convex, and lower semicontinuous function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h :{\mathcal {X}}\rightarrow {\mathbb {R}}$$\end{document} is a convex and differentiable function such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla h$$\end{document} is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{\nabla h}$$\end{document} -Lipschitz continuous, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A :{\mathcal {X}}\rightarrow {\mathcal {Y}}$$\end{document} is a linear continuous operator.We generate a sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k},\lambda _{k} \right) _{k \geqslant 0}$$\end{document} of primal-dual iterates which converges weakly to a primal-dual solution of (1.15). In addition, we achieve as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} convergence rates for the velocities

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert x_{k} - x_{k-1} \right\Vert = o \left( \dfrac{1}{k} \right) \qquad \text { and } \qquad \left\Vert \lambda _{k} - \lambda _{k-1} \right\Vert = o \left( \dfrac{1}{k} \right) , \end{aligned}$$\end{document}

the tangent residual

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert u_k + \nabla h(x_k) + A^{*} \lambda _k \right\Vert = o \left( \dfrac{1}{k} \right) \qquad \text { and } \qquad \left\Vert v_k - Ax_{k} + b \right\Vert = o \left( \dfrac{1}{k} \right) , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u_k \in \partial f(x_k)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_k \in N_{\mathcal{K}^*}(\lambda _k)$$\end{document} , for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 0$$\end{document} , the primal-dual gap

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{{\mathcal {L}}}} \left( x_{k} , \lambda _{*} \right) - {{{\mathcal {L}}}} \left( x_{*} , \lambda _{k} \right) = o \left( \dfrac{1}{k} \right) , \end{aligned}$$\end{document}

the complementarity condition

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} |\langle \lambda _k, Ax_{k} - b \rangle | = o \left( \dfrac{1}{k} \right) , \end{aligned}$$\end{document}

and the objective function values

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} |\left( f+h \right) \left( x_{k} \right) - \left( f+h \right) \left( x_{*} \right) |= o \left( \dfrac{1}{k} \right) . \end{aligned}$$\end{document}

Here, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\mathcal {L}}}}$$\end{document} denotes the Lagrangian attached to (1.15), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{*} , \lambda _{*} \right) $$\end{document} a primal-dual optimal solution.

• The approach we consider involves the minimization of the sum of a nonsmooth convex and a smooth convex function subject to linear equality constraints. While there is an extensive body of work on full splitting primal-dual methods for solving this class of problems, few results exist regarding fast-converging methods in terms of objective function values and feasibility measures, while also ensuring the convergence of the iterates. Our approach contributes to filling this gap.

A Fast Reflected Forward-Backward algorithm for monotone inclusions

The Fast RFB algorithm

In this section, we formulate the algorithm and conduct a thorough analysis of its convergence.

In the following we give an equivalent formulation for Algorithm 1, which will play a central role in the convergence analysis.

Proposition 2.1

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{0}, y_{0}, w_{0} \in {\mathcal {H}}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{1} = J_{\gamma M} \left( y_{0} -\gamma F \left( w_{0} \right) \right) $$\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _{1} = \frac{1}{\gamma } \left( y_{0} - z_{1} \right) - F \left( w_{0} \right) \in M \left( z_{1} \right) $$\end{document} . Then the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(z_k)_{k \geqslant 0}$$\end{document} generated in Algorithm 1 can also be generated equivalently by the following iterative scheme

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (\forall k \geqslant 1) \ \left\{ \begin{aligned} w_{k}&= z_{k} + \left( 1 - \dfrac{\alpha }{k + \alpha } \right) \left( z_{k} - z_{k-1} \right) - \frac{c}{k+\alpha } \gamma \big (\xi _k + F \left( w_{k-1} \right) \big ), \\ z_{k+1}&= J_{\gamma M} \big ( w_{k} - \gamma F \left( w_{k} \right) + \gamma \left( F \left( w_{k-1} \right) + \xi _{k}\right) \big ),\\ \xi _{k+1}&= \frac{1}{\gamma }\left( w_{k}-z_{k+1}\right) - F \left( w_{k} \right) + F \left( w_{k-1} \right) + \xi _{k}. \end{aligned} \right. \end{aligned}$$\end{document}

In addition, it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \xi _{k}\in M \left( z_{k} \right) \qquad \quad \forall k\geqslant 1. \end{aligned}$$\end{document}

Proof

Given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{0}, w_{0}, y_{0} \in {\mathcal {H}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{1} = J_{\gamma M} \left( y_{0} -\gamma F \left( w_{0} \right) \right) $$\end{document} , it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \xi _{1} = \frac{1}{\gamma } \left( y_{0} - z_{1} \right) - F \left( w_{0} \right) \in M \left( z_{1} \right) . \end{aligned}$$\end{document}

In the same way, by invoking also the third update block of Algorithm 1, we obtain

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \xi _{k+1} := \frac{1}{\gamma }\left( y_{k}-z_{k+1}\right) - F \left( w_{k} \right) \in M(z_{k+1}) \quad \forall k \geqslant 0. \end{aligned}$$\end{document}

Therefore, the iterative scheme in Algorithm 1 can be equivalently written as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (\forall k \geqslant 1) \ \left\{ \begin{aligned} y_{k}&= z_{k} + \left( 1 - \dfrac{\alpha }{k + \alpha } \right) \left( z_{k} - z_{k-1} \right) + \left( 1 - \frac{c}{k+\alpha } \right) \gamma \left( \xi _k + F \left( w_{k-1} \right) \right) , \\ w_{k}&= y_{k} - \gamma \left( \xi _k + F \left( w_{k-1} \right) \right) ,\\ z_{k+1}&= J_{\gamma M} \left( y_{k} -\gamma F \left( w_{k} \right) \right) ,\\ \xi _{k+1}&= \frac{1}{\gamma }\left( y_{k}-z_{k+1}\right) - F \left( w_{k} \right) , \end{aligned}\right. \end{aligned}$$\end{document}

which, after some simplifications, it transforms into (2.2).

Conversely, starting from (2.2), we can define a sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( y_{k} \right) _{k \geqslant 1}$$\end{document} following the first update block in (2.3). Consequently, for every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document} it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \gamma \left( \xi _{k+1} + F \left( w_{k} \right) \right) = y_k-z_{k+1} , \end{aligned}$$\end{document}

leading to the transformation of (2.2) into (2.1). \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Remark 2.2

If the momentum term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{k} - z_{k-1}$$\end{document} and the correction term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{k-1} - z_{k}$$\end{document} are removed from the first update block of (2.1), the resulting algorithm reduces to the Reflected Forward-Backward (RFB) method

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ (\forall k \geqslant 1) \ \left\{ \begin{aligned} w_{k}&= 2z_{k} - z_{k-1},\\ z_{k+1}&= J_{\gamma M} \left( z_{k} -\gamma F \left( w_{k} \right) \right) , \end{aligned}\right. \end{aligned}$$\end{document}

as introduced in [15]. This method extends the PRG method proposed by Malitsky in [40] for variational inequalities. For (2.4), best-iterate and last-iterate convergence rates of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{\sqrt{k}} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the tangent residual were established in [52].

When F is linear, (2.4) simplifies to the Forward-Reflected-Backward (FRB) method (1.12) by Malitsky and Tam [41]

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (\forall k \geqslant 1) \ z_{k+1} := J_{\gamma M} \left( z_{k} -2\gamma F \left( z_{k} \right) + \gamma F \left( z_{k-1} \right) \right) . \end{aligned}$$\end{document}

While convergence results for the iterates were established in [41] in the general setting, a last-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{\sqrt{k}} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the tangent residual was demonstrated in [14] in the specific case of variational inequalities.

Remark 2.3

The Accelerated Reflected Gradient (ARG) method, introduced by Cai and Zheng in [14], is defined as follows for initial points \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{0}=z_{1} \in {\mathcal {H}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{2}=J_{\gamma M}\left( z_{1} -\gamma F(z_{1})\right) $$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}\ (\forall k \geqslant 1) \ \left\{ \begin{aligned} x_{k}&= 2 z_{k} - z_{k-1} + \frac{1}{k+1}\left( z_{0} - z_{k} \right) - \frac{1}{k}\left( z_{0} - z_{k-1} \right) , \\ z_{k+1}&= J_{\gamma M} \left( z_{k} - \gamma F\left( x_{k} \right) + \frac{1}{k+1} \left( z_{0} -z_{k} \right) \right) , \end{aligned}\right. \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0<\gamma \leqslant \frac{1}{2\sqrt{6}L}$$\end{document} . The ARG method builds on the anchoring technique employed in (2.4), a concept originating in [31]. It was shown to achieve a last-iterate convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k}\right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the tangent residual.

The Accelerated Extragradient (AEG) method was introduced by Tran-Dinh in [49] accelerates the FBF method by employing the momentum term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{k+1}{k+3}\left( x_{k}-x_{k-1} \right) $$\end{document} alongside additional correction terms. It is defined as follows for initial points \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_0, x_1, w_0, z_1 \in {\mathcal {H}}$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (\forall k \geqslant 1) \ \left\{ \begin{aligned} x_{k}&= J_{\gamma M} \left( z_{k} - \gamma F(z_{k}) + \frac{k+1}{k+2}\gamma w_{k-1} \right) , \\ w_{k}&= \frac{1}{\gamma } \left( z_{k} - x_{k} + \frac{k+1}{k+2} \gamma w_{k-1}\right) + F (x_{k}) - F(z_{k}),\\ z_{k+1}&= x_{k} + \frac{k+1}{k+3}\left( x_{k}-x_{k-1} \right) - \frac{k+2}{k+3}\gamma \left( F (x_{k}) - F(z_{k}) \right) , \end{aligned}\right. \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0<\gamma < \frac{1}{L}$$\end{document} .

To further reduce the number of evaluations of F, Tran-Dinh proposed the Accelerated Past Extragradient (APEG) method in [49], defined as follows for initial points \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_0, x_1, w_0, z_1 \in {\mathcal {H}}$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}\ (\forall k \geqslant 1) \ \left\{ \begin{aligned} x_{k}&= J_{\gamma M} \left( z_{k} -\gamma F(z_{k}) + \frac{k+1}{k+2}\gamma w_{k-1} \right) , \\ w_{k}&= \frac{1}{\gamma } \left( z_{k} - x_{k} + \frac{k+1}{k+2} \gamma w_{k-1}\right) ,\\ z_{k+1}&= x_{k} + \frac{k+1}{k+3}\left( x_{k}-x_{k-1} \right) + \frac{5(k+2)}{6(k+3)}\gamma w_{k} -\frac{5(k+1)}{6(k+3)}\gamma w_{k-1}, \end{aligned}\right. \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0<\gamma \leqslant \frac{3}{2\sqrt{29}L}$$\end{document} .

Both the AEG and APEG methods have been shown to achieve last-iterate convergence rates of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k}\right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the tangent residual.

In contrast, the Fast RFB algorithm introduces both momentum and correction terms to the Reflected Forward-Backward framework (2.4). This design not only ensures weak convergence to a solution of (1.3) but also achieves a superior convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k}\right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the discrete velocity and the tangent residual.

Convergence analysis

In the following, we will use the notation

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} v_{k} := F(w_{k-1}) + \xi _{k} \quad \forall k \geqslant 1. \end{aligned}$$\end{document}

With this notation, from Algorithm 1 we obtain

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} w_{k} = z_{k} + \dfrac{k}{k + \alpha } \Big ( z_{k} - z_{k-1} \Big ) - \dfrac{c}{k + \alpha } \gamma v_{k} \quad \text{ and } \quad z_{k+1} = w_{k} - \gamma \Big ( v_{k+1} - v_{k} \Big ) \quad \forall k \geqslant 1.\nonumber \\ \end{aligned}$$\end{document}

Since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ 0< \gamma < \frac{1}{2L}$$\end{document} , the definition of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_{k}$$\end{document} together with the Lipschitz continuity of F give for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert \xi _{k+1} + F \left( z_{k+1} \right) - v_{k+1} \right\Vert&= \left\Vert F \left( z_{k+1} \right) - F \left( w_{k} \right) \right\Vert \leqslant L \left\Vert z_{k+1} - w_{k} \right\Vert \nonumber \\&= \gamma L \left\Vert v_{k+1} - v_{k} \right\Vert \leqslant \frac{1}{2} \left\Vert v_{k+1} - v_{k} \right\Vert \leqslant \left\Vert v_{k+1} - v_{k} \right\Vert . \end{aligned}$$\end{document}

By summing up the two equations in (2.5), we obtain that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document} it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( k + \alpha \right) \left( z_{k+1} - z_{k} \right) - k \left( z_{k} - z_{k-1} \right) = -c \gamma v_{k+1} - \gamma (k+\alpha -c) \left( v_{k+1} - v_{k} \right) . \end{aligned}$$\end{document}

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{*} \in \textrm{Zer}(M+F)$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \leqslant \lambda \leqslant \alpha - 1$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1< s < 2$$\end{document} . In the lines of [9], we denote for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} u_{\lambda ,s,k} :=&2 \lambda \left( z_{k} - z_{*} \right) + 2k \left( z_{k} - z_{k-1} \right) + s\gamma k v_{k}, \nonumber \\ {\mathcal {E}}_{\lambda ,s,k} :=&\dfrac{1}{2} \left\Vert u_{\lambda ,s,k} \right\Vert ^{2} + 2 \lambda \left( \alpha - 1 - \lambda \right) \left\Vert z_{k} - z_{*} \right\Vert ^{2} \nonumber \\ &+ 2 \lambda \gamma \left( (2-s)k+2(\alpha -c)\right) \left\langle z_{k} - z_{*}, v_{k} \right\rangle \nonumber \\&+ \frac{1}{2}\gamma ^2\left( (2-s)k+2(\alpha -c)\right) \left( s k + 2 c \right) \left\Vert v_{k} \right\Vert ^{2}, \end{aligned}$$\end{document}

and for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 2$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {G}}_{\lambda ,s,k}&:= {{\mathcal {E}}_{\lambda ,s,k}} - 2 \gamma \big ((2-s)k+2(\alpha -c)\big )k \left\langle z_{k} - z_{k-1}, F \left( z_{k} \right) - F \left( w_{k-1} \right) \right\rangle \nonumber \\&\quad \phantom {:} + \gamma ^3 L\big ((2-s)k+2(\alpha -c)\big ) \left( k+\alpha -c +c \gamma L \sqrt{(2-s)k+2(\alpha -c )} \right) \nonumber \\ &\qquad \left\Vert v_{k} - v_{k-1} \right\Vert ^{2}. \end{aligned}$$\end{document}

The proof of the following lemma is given in Appendix B.

Lemma 2.4

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{*} \in \textrm{Zer}(M+F)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} be the sequence generated by Algorithm 1. For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \leqslant \lambda \leqslant \alpha - 1$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1< s < 2$$\end{document} , the following identity holds for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathcal {E}}_{\lambda ,s,k+1} - {\mathcal {E}}_{\lambda ,s,k} =&\ -4 \left( c - 1 \right) \lambda \gamma \left\langle z_{k+1} - z_{*}, v_{k+1} \right\rangle \\ &+ 2 \left( \lambda + 1 - \alpha \right) \left( 2k + \alpha + 1 \right) \left\Vert z_{k+1} - z_{k} \right\Vert ^{2} \\&+ 2\gamma \Bigl (\big ((2-s)\lambda + s \left( \lambda + 1 - \alpha \right) +s - 2c\big ) k + 2 \lambda \alpha +s -2\alpha c \Bigr )\\&\quad \left\langle z_{k+1} - z_{k}, v_{k+1} \right\rangle \\&- 2\gamma \left( k + \alpha \right) \Big ((2-s)k + 2\left( \alpha - c \right) \Big ) \left\langle z_{k+1} - z_{k}, v_{k+1} - v_{k} \right\rangle \\&- \gamma ^2 \left( k + \alpha \right) \Big ((2-s)k + 2\left( \alpha - c \right) \Big ) \left\Vert v_{k+1} - v_{k} \right\Vert ^{2} \\&+\gamma ^2 \Big ( (1-c)(2sk + 2c +s) + s(\alpha - c )\Big ) \left\Vert v_{k+1} \right\Vert ^{2} . \end{aligned} \end{aligned}$$\end{document}

In the next result we prove a quasi-Féjer monotone property together with a lower bound for the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({\mathcal {G}}_{\lambda ,s,k})_{k \geqslant 1}$$\end{document} . For the proof we also refer the reader to Appendix B.

Lemma 2.5

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{*} \in \textrm{Zer}(M+F)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} be the sequence generated by Algorithm 1. For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \leqslant \lambda \leqslant \alpha - 1$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1< s < 2$$\end{document} , the following statements are true:

• (i)for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 2$$\end{document} , it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned}&{\mathcal {G}}_{\lambda ,s,k+1} - {\mathcal {G}}_{\lambda ,s,k}\\&\quad \leqslant \frac{4\left( c - 1 \right) ^2}{\left( k+1 \right) \sqrt{k+1}} \lambda ^{2} \left\Vert z_{k+1} - z_{*} \right\Vert ^{2}- 4 \left( c - 1 \right) \lambda \gamma \left\langle z_{k+1} - z_{*}, \xi _{k+1} + F \left( z_{k+1} \right) \right\rangle \\&\qquad + 2\gamma \left( \omega _{1} k + \omega _{2} \right) \left\langle z_{k+1} - z_{k}, v_{k+1} \right\rangle - \mu _{k} \gamma ^{2} \left\Vert v_{k+1} - v_{k} \right\Vert ^{2} \\&\qquad + 2\left( \omega _{3} k + \sqrt{\omega _{5}(k+1)+\omega _{7}}\right) \left\Vert z_{k+1} - z_{k} \right\Vert ^{2}\\&\qquad + \gamma ^2 \left( \omega _{4} k + c \sqrt{\omega _{0}k+2 \omega _{6}} + s \omega _{6}\right) \left\Vert v_{k+1} \right\Vert ^{2}, \end{aligned} \end{aligned}$$\end{document}

where

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \omega _{0} =&\ 2-s> 0, \quad \omega _{1} = \omega _{0} \lambda +s \left( \lambda + 1 - \alpha \right) + s - 2c, \quad \omega _{2} = 2\lambda \alpha +s -2 \alpha c, \\ \omega _{3} =&\ 2 \left( \lambda + 1 - \alpha \right) \leqslant 0, \quad \omega _{4} = 2s( 1 - c) < 0, \quad \omega _{5} = (\alpha - 2)\omega _{0}> 0, \\ \omega _{6} =&\ \alpha - c> 0, \quad \omega _{7} = \ (\alpha -1)(2\omega _{6} - \omega _{0}) > 0,\\ \mu _{k} =&\ \omega _{0} \left( 1 - 2\gamma L\right) k^2 + \left( 2 \omega _{6} + \omega _{0}\alpha \right) k + 2 \omega _{6} \alpha \\&\ - 2 \gamma L\left( \left( 2(\omega _{0} + 2 \omega _{6})- s \omega _{6}\right) k + \left( \omega _{0} + 2 \omega _{6}\right) (\alpha +1 -c)\right) - (k + 1)\sqrt{k + 1}\\&\ -(\omega _{5}(k+1) + \omega _{7})\\ &\times \sqrt{\omega _{5}(k+1) + \omega _{7}} -\gamma ^2 L^2 c\big (\omega _{0}(k + 1) + 2 \omega _{6}\big )\sqrt{\omega _{0}(k + 1) + 2 \omega _{6}}. \end{aligned} \end{aligned}$$\end{document}

• (ii)there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k_1 \geqslant 2$$\end{document} such that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant k_1$$\end{document} it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathcal {G}}_{\lambda ,s, k} \geqslant&\ \dfrac{1}{4 s } \omega _{0} \left\Vert 4 \lambda \left( z_{k} - z_{*} \right) + 2 k \left( z_{k} - z_{k-1} \right) + 2s \gamma k v_{k} \right\Vert ^{2} \\&+ \frac{1}{4s}\omega _{0}^2k^{2} \left\Vert z_{k} - z_{k-1} \right\Vert ^{2} + 2 \lambda \left( \alpha - 1 - \dfrac{4\left( \alpha - 1\right) }{s\alpha } \lambda \right) \left\Vert z_{k} - z_{*} \right\Vert ^{2}. \end{aligned} \end{aligned}$$\end{document}

The following lemma will play an essential role in the proof of the main convergence result of this section. Its proof is also given in Appendix B.

Lemma 2.6

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{*} \in \textrm{Zer}(M+F)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} be the sequence generated by Algorithm 1. The following statements are true:

• (i)if s and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta $$\end{document} are such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 1+ \frac{\alpha }{4 c - \alpha }<s < 2 \end{aligned}$$\end{document}

and

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \max \left\{ \sqrt{\frac{s\left( \alpha - 2\right) + 2\left( 2c -s \right) }{ 4 s \left( c - 1\right) }}, \sqrt{\frac{-\left( 2 -s \right) \left( \alpha - 1 \right) - s +2c }{s\left( c - 1 \right) }}\right\}< \delta <1, \end{aligned}$$\end{document}

then there exist

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 0 \leqslant {\underline{\lambda }} \left( \alpha ,c,s \right) < {\overline{\lambda }} \left( \alpha ,c,s \right) \leqslant \dfrac{s \alpha }{4}, \end{aligned}$$\end{document}

such that for every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda $$\end{document} satisfying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\underline{\lambda }} \left( \alpha ,c,s \right)< \lambda < {\overline{\lambda }} \left( \alpha ,c,s \right) $$\end{document} one can find an integer \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k_{\lambda } \geqslant 1$$\end{document} with the property that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant k_{\lambda }$$\end{document} the following inequality holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} R_{k} :=&\ 2\gamma \left( \omega _{1} k + \omega _{2} \right) \left\langle z_{k+1} - z_{k}, v_{k+1} \right\rangle + \delta \gamma ^2 \left( \omega _{4} k + c\sqrt{\omega _{0}k + 2 \omega _{6}}+ s \omega _{6}\right) \left\Vert v_{k+1} \right\Vert ^{2}\\&\ + 2\delta \left( \omega _{3} k + \sqrt{\omega _{5}(k + 1) + \omega _{7}} \right) \left\Vert z_{k+1} - z_{k} \right\Vert ^{2} \\ \leqslant&\ 0; \end{aligned} \end{aligned}$$\end{document}

• (ii)there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k_2 \geqslant 2$$\end{document} such that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant k_2$$\end{document} it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mu _{k} \geqslant 0. \end{aligned}$$\end{document}

We are now in a position to prove a proposition that allows us to make initial statements about convergence rates.

Proposition 2.7

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{*} \in \textrm{Zer}(M+F)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} be the sequence generated by Algorithm 1. The following statements are true:

• (i)it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\sum _{k \geqslant 1} \left\langle z_{k} - z_{*}, F (z_{k}) + \xi _{k} \right\rangle< + \infty , \quad \sum _{k \geqslant 1} k^{2} \left\Vert v_{k+1} - v_{k} \right\Vert ^{2}< + \infty ,\\&\sum _{k \geqslant 1} k \left\Vert F(w_{k}) + \xi _{k+1} \right\Vert ^{2}< + \infty , \qquad \sum _{k \geqslant 1} k \left\Vert z_{k+1} - z_{k} \right\Vert ^{2} < + \infty ; \end{aligned}$$\end{document}

• (ii)the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} is bounded and it holds as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ k \rightarrow +\infty $$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\left\Vert z_{k} - z_{k-1} \right\Vert = {\mathcal {O}}\left( \dfrac{1}{k} \right) , \qquad \left\Vert \xi _{k} + F \left( w_{k-1} \right) \right\Vert = {\mathcal {O}}\left( \dfrac{1}{k} \right) , \\&\left\Vert \xi _{k} + F \left( z_{k} \right) \right\Vert = {\mathcal {O}}\left( \dfrac{1}{k} \right) , \quad \left\langle z_{k} - z_{*}, \xi _{k} + F \left( z_{k} \right) \right\rangle = {\mathcal {O}}\left( \dfrac{1}{k} \right) ; \end{aligned}$$\end{document}

• (iii)for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s \in \left( 1+ \frac{\alpha }{4 c - \alpha }, 2 \right) $$\end{document} , there exist \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \leqslant {\underline{\lambda }} \left( \alpha ,c, s \right) < {\overline{\lambda }} \left( \alpha ,c ,s \right) \leqslant \frac{s\alpha }{4}$$\end{document} such that, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\underline{\lambda }} \left( \alpha ,c,s \right)< \lambda < {\overline{\lambda }} \left( \alpha ,c,s \right) $$\end{document} , the sequences \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\mathcal {E}}_{\lambda ,s,k} \right) _{k \geqslant 1}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\mathcal {G}}_{\lambda ,s,k} \right) _{k \geqslant 2}$$\end{document} are convergent.

Proof

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s \in \left( 1+ \frac{\alpha }{4 c - \alpha }, 2 \right) $$\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta \in (0,1)$$\end{document} such that (2.14) is satisfied. According to Lemma 2.6(i) there exist \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \leqslant {\underline{\lambda }} \left( \alpha ,c,s \right) < {\overline{\lambda }} \left( \alpha ,c,s \right) \leqslant \frac{s\alpha }{4}$$\end{document} such that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \lambda \in \left( {\underline{\lambda }} \left( \alpha ,c,s \right) , {\overline{\lambda }} \left( \alpha ,c,s \right) \right) $$\end{document} there exists an integer \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ k_{\lambda }} \geqslant 1$$\end{document} with the property that (2.15) holds for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant { k_{\lambda }}$$\end{document} . In addition, according to Lemma 2.6(ii), we get a positive integer \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ k_{2}} \geqslant 2$$\end{document} such that (2.16) holds for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant { k_{2}}$$\end{document} .

This means that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant { k_{0}} := \max \left\{ { k_{\lambda }},k_{1}, { k_{2}} \right\} $$\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k_{1}$$\end{document} is the positive integer given by Lemma 2.5(ii), according to Lemma 2.5(i) it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned}&{\mathcal {G}}_{\lambda ,s,k+1} - {\mathcal {G}}_{\lambda ,s,k}\\&\quad \leqslant \frac{4\left( c - 1 \right) ^2}{\left( k+1 \right) \sqrt{k+1}} \lambda ^{2} \left\Vert z_{k+1} - z_{*} \right\Vert ^{2}- 4 \left( c - 1 \right) \lambda \gamma \left\langle z_{k+1} - z_{*}, \xi _{k+1} + F \left( z_{k+1} \right) \right\rangle \\&\qquad - \mu _{k} \gamma ^{2} \left\Vert v_{k+1} - v_{k} \right\Vert ^{2} + \left( 1 - \delta \right) \gamma ^2 \left( \omega _{4} k + c\sqrt{\omega _{0}k + 2 \omega _{6}}+ s \omega _{6}\right) \left\Vert v_{k+1} \right\Vert ^{2}\\&\qquad + 2\left( 1 - \delta \right) \left( \omega _{3} k + \sqrt{\omega _{5}(k + 1) + \omega _{7}} \right) \left\Vert z_{k+1} - z_{k} \right\Vert ^{2}. \end{aligned} \end{aligned}$$\end{document}

Since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega _{3}, \omega _{4} < 0$$\end{document} , there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k_{3} \geqslant k_{0}$$\end{document} such that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant k_{3}$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathcal {G}}_{\lambda ,s,k+1} \leqslant&\ {\mathcal {G}}_{\lambda ,s,k} + \frac{4\left( c - 1 \right) ^2}{\left( k+1 \right) \sqrt{k+1}} \lambda ^{2} \left\Vert z_{k+1} - z_{*} \right\Vert ^{2}- 4 \left( c - 1 \right) \lambda \gamma \left\langle z_{k+1} - z_{*}, \xi _{k+1}\right. \\ &\left. + F \left( z_{k+1} \right) \right\rangle \ - \mu _{k} \gamma ^{2} \left\Vert v_{k+1} - v_{k} \right\Vert ^{2} + \frac{1}{2}\left( 1 - \delta \right) \gamma ^2 \omega _{4} k \left\Vert v_{k+1} \right\Vert ^{2} \\ &+ \left( 1 - \delta \right) \omega _{3} k\left\Vert z_{k+1} - z_{k} \right\Vert ^{2}. \end{aligned}\nonumber \\ \end{aligned}$$\end{document}

In view of (2.12), we get that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {G}}_{\lambda ,s,k} \geqslant 0$$\end{document} for every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 2$$\end{document} . By setting

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_{0} := 2\lambda \left( c - 1 \right) ^2 \left( \alpha -1 - \frac{ 4 \left( \alpha - 1 \right) }{s \alpha }\lambda \right) ^{-1} > 0, \end{aligned}$$\end{document}

it holds for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{4\left( c - 1 \right) ^2}{\left( k+1 \right) \sqrt{k+1}} \lambda ^{2} \left\Vert z_{k+1} - z_{*} \right\Vert ^{2}&= \dfrac{C_{0}}{\left( k+1 \right) \sqrt{k+1}} \cdot 2 \lambda \left( \alpha - 1 - \dfrac{4\left( \alpha - 1 \right) }{s\alpha } \lambda \right) \\ &\quad \left\Vert z_{k+1} - z_{*} \right\Vert ^{2} \\&\leqslant \dfrac{C_{0}}{\left( k+1 \right) \sqrt{k+1}} {\mathcal {G}}_{\lambda ,s,k+1}. \end{aligned}$$\end{document}

Under these premises, we deduce from (2.17) that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant k_{3}$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \left( 1 - \dfrac{C_{0}}{\left( k+1 \right) \sqrt{k+1}} \right) {\mathcal {G}}_{\lambda , s, k+1} \leqslant&\ {\mathcal {G}}_{\lambda , s, k} - 4 \left( c - 1 \right) \lambda \gamma \left\langle z_{k+1} - z_{*}, \xi _{k+1} + F \left( z_{k+1} \right) \right\rangle \\&\ - \mu _{k} \gamma ^{2} \left\Vert v_{k+1} - v_{k} \right\Vert ^{2}\\&\ + \frac{1}{2}\left( 1 - \delta \right) \gamma ^2 \omega _{4} k \left\Vert v_{k+1} \right\Vert ^{2} + \left( 1 - \delta \right) \omega _{3} k\left\Vert z_{k+1} - z_{k} \right\Vert ^{2}. \end{aligned}\nonumber \\ \end{aligned}$$\end{document}

Choosing $[eqn]$ , we have that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ k \geqslant k_{4} $$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( 1 - \dfrac{C_{0}}{\left( k+1 \right) \sqrt{k+1}} \right) ^{-1} = \dfrac{\left( k+1 \right) \sqrt{k+1}}{\left( k+1 \right) \sqrt{k+1} - C_{0}} = 1 + \dfrac{C_{0}}{\left( k+1 \right) \sqrt{k+1} - C_{0}} > 1. \end{aligned}$$\end{document}

Hence, using the monotonicity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M+F$$\end{document} and that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega _3, \omega _4 <0$$\end{document} , (2.18) leads for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant { k_{4}}$$\end{document} to

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {G}}_{\lambda ,s, k+1} \leqslant&\ \left( 1 + \dfrac{C_{0}}{\left( k+1 \right) \sqrt{k+1} - C_{0}} \right) {\mathcal {G}}_{\lambda ,s,k} - 4 \left( c - 1 \right) \lambda \gamma \left\langle z_{k+1} - z_{*}, \xi _{k+1} + F \left( z_{k+1} \right) \right\rangle \\&\ - \mu _{k} \gamma ^{2} \left\Vert v_{k+1} - v_{k} \right\Vert ^{2} + \frac{1}{2}\left( 1 - \delta \right) \gamma ^2 \omega _{4} k \left\Vert v_{k+1} \right\Vert ^{2} + \left( 1 - \delta \right) \omega _{3} k\left\Vert z_{k+1} - z_{k} \right\Vert ^{2}. \end{aligned}$$\end{document}

Denoting

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} b_{\lambda , s, k} :=&\ 4 \left( c - 1 \right) \lambda \gamma \left\langle z_{k+1} - z_{*}, \xi _{k+1} + F \left( z_{k+1} \right) \right\rangle + \mu _{k} \gamma ^{2} \left\Vert v_{k+1} - v_{k} \right\Vert ^{2}\\&\ - \frac{1}{2}\left( 1 - \delta \right) \gamma ^2 \omega _{4} k \left\Vert v_{k+1} \right\Vert ^{2} -\left( 1 - \delta \right) \omega _{3} k\left\Vert z_{k+1} - z_{k} \right\Vert ^{2}\\ \geqslant&\ 0, \\ d_{\lambda , s, k}&:= \ \dfrac{C_{0}}{\left( k+1 \right) \sqrt{k+1} - C_{0}} > 0, \end{aligned}$$\end{document}

we see that we are in the context of Lemma A.2. From here we get the summability statements in (i) as well as the convergence of the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\mathcal {G}}_{\lambda ,s,k} \right) _{k \geqslant 2}$$\end{document} .

Since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\mathcal {G}}_{\lambda ,s,k} \right) _{k \geqslant 2}$$\end{document} converges, it is also bounded from above, thus, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 2$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\frac{1}{4s}\omega _{0} \left\Vert 4 \lambda \left( z_{k} - z_{*} \right) + 2k \left( z_{k} - z_{k-1} \right) + 2s \gamma k v_{k} \right\Vert ^{2} \\&\qquad + \frac{\omega _{0}^2}{4s}k^{2} \left\Vert z_{k} - z_{k-1} \right\Vert ^{2}+ 2 \lambda \left( \alpha - 1 -\frac{4\left( \alpha -1\right) }{s\alpha }\lambda \right) \left\Vert z_{k} - z_{*} \right\Vert ^{2}\\&\quad \leqslant \ {\mathcal {G}}_{\lambda ,s, k} \leqslant \sup _{k \geqslant 1} {\mathcal {G}}_{\lambda ,s, k} < +\infty . \end{aligned}$$\end{document}

From here we obtain that the sequences

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \left( 4 \lambda \left( z_{k} - z_{*} \right) + 2k \left( z_{k} - z_{k-1} \right) + 2s \gamma k v_{k} \right) _{k \geqslant 1}, \quad \left( k \left( z_{k} - z_{k-1} \right) \right) _{k \geqslant 1} \quad \text {and} \quad \left( z_{k} \right) _{k \geqslant 0} \end{aligned} \end{aligned}$$\end{document}

are bounded. In particular, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 2$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert 4 \lambda \left( z_{k} - z_{*} \right) + 2k \left( z_{k} - z_{k-1} \right) + 2s\gamma k v_{k} \right\Vert \leqslant&\ C_{1} := \sqrt{\dfrac{4s}{\omega _{0}} \sup _{k \geqslant 1} {\mathcal {G}}_{\lambda ,k}}, \nonumber \\ k \left\Vert z_{k} - z_{k-1} \right\Vert \leqslant&\ C_{2} := \sqrt{\dfrac{4s}{\omega _{0}^2} \sup _{k \geqslant 1} {\mathcal {G}}_{\lambda ,k}}, \nonumber \\ \left\Vert z_{k} - z_{*} \right\Vert \leqslant&\ C_{3} := \sqrt{\frac{1}{2 \lambda \left( \alpha - 1 -\frac{4\left( \alpha -1\right) }{s\alpha }\lambda \right) } \sup _{k \geqslant 1} {\mathcal {G}}_{\lambda ,k}}, \end{aligned}$$\end{document}

therefore

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert v_{k} \right\Vert \leqslant&\ \dfrac{1}{2 s \gamma k} \left\Vert 4 \lambda \left( z_{k} - z_{*} \right) + 2k \left( z_{k} - z_{k-1} \right) + 2s \gamma k v_{k} \right\Vert \nonumber \\&\ + \dfrac{1}{s \gamma } \left\Vert z_{k} - z_{k-1} \right\Vert + \dfrac{2 \lambda }{s\gamma k} \left\Vert z_{k} - z_{*} \right\Vert \leqslant \dfrac{C_{4}}{k}, \end{aligned}$$\end{document}

where

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_{4} := \dfrac{1}{2 s\gamma } \left( C_{1} + 2C_{2} + 4 {\overline{\lambda }} \left( \alpha ,c,s \right) C_{3} \right) > 0 . \end{aligned}$$\end{document}

From (i) we have

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow + \infty } k \left\Vert v_{k+1} - v_{k} \right\Vert = 0 \quad \Rightarrow \quad C_{5} := \sup _{k \geqslant 1} \left\{ k \left\Vert v_{k+1} - v_{k} \right\Vert \right\} < + \infty , \end{aligned}$$\end{document}

which, together with (2.6), implies that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \left\Vert \xi _{k+1} + F \left( z_{k+1} \right) \right\Vert&\leqslant \left\Vert \xi _{k+1} + F \left( z_{k+1} \right) - v_{k+1} \right\Vert + \left\Vert v_{k+1} \right\Vert \\&\leqslant \left\Vert v_{k+1} - v_{k} \right\Vert + \left\Vert v_{k+1} \right\Vert \leqslant \dfrac{C_{6}}{k}, \end{aligned} \end{aligned}$$\end{document}

where

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_{6} := C_{4} + C_{5} > 0 . \end{aligned}$$\end{document}

The Cauchy-Schwarz inequality and the boundedness of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} allow us to provide a similar estimate for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\langle z_{k} - z_{*}, \xi _{k} + F \left( z_{k} \right) \right\rangle $$\end{document} . This proves (ii). To complete the proof, we are going to show that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow + \infty } {\mathcal {E}}_{\lambda ,s,k} = \lim \limits _{k \rightarrow + \infty } {\mathcal {G}}_{\lambda ,s,k} \in {\mathbb {R}}. \end{aligned}$$\end{document}

Indeed, we have already seen that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow + \infty } k \left\Vert v_{k} - v_{k-1} \right\Vert = 0, \end{aligned}$$\end{document}

which, by the Cauchy-Schwarz inequality and (2.6), yields

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 0 \leqslant \lim \limits _{k \rightarrow + \infty } k^{2} \left|\left\langle z_{k} - z_{k-1}, F \left( z_{k} \right) - F \left( w_{k-1} \right) \right\rangle \right|&\leqslant C_{2} \lim \limits _{k \rightarrow + \infty } k \left\Vert F \left( z_{k} \right) - F \left( w_{k-1} \right) \right\Vert \\&\leqslant C_{2} \lim \limits _{k \rightarrow + \infty } k \left\Vert v_{k} - v_{k-1} \right\Vert = 0 . \end{aligned}$$\end{document}

It is from here that we get the statement we want. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Next we will prove the convergence of the sequence of iterates generated by Algorithm 1.

Theorem 2.8

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{Zer}(M+F) \ne \emptyset $$\end{document} . The sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} generated by Algorithm 1 converges weakly to a solution of (1.3).

Proof

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_*\in \textrm{Zer}(M+F)$$\end{document} . Further, let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s \in \left( 1+ \frac{\alpha }{4 c - \alpha }, 2 \right) $$\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \leqslant {\underline{\lambda }} \left( \alpha ,c, s \right) < {\overline{\lambda }} \left( \alpha ,c ,s \right) \leqslant \frac{s\alpha }{4}$$\end{document} be the parameters provided by Proposition 2.7(iii) with the property that, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\underline{\lambda }} \left( \alpha ,c,s \right)< \lambda < {\overline{\lambda }} \left( \alpha ,c,s \right) $$\end{document} , the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\mathcal {E}}_{\lambda ,s,k} \right) _{k \geqslant 1}$$\end{document} is convergent.

For all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document} and any \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda \in \left( {\underline{\lambda }} \left( \alpha ,c,s \right) , {\overline{\lambda }} \left( \alpha ,c,s \right) \right) $$\end{document} we have

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathcal {E}}_{\lambda ,s, k} =&\ \dfrac{1}{2} \left\Vert 2 \lambda \left( z_{k} - z_{*} \right) + 2k \left( z_{k} - z_{k-1} \right) + s \gamma k v_{k} \right\Vert ^{2} + 2 \lambda \left( \alpha - 1 - \lambda \right) \left\Vert z_{k} - z_{*} \right\Vert ^{2} \\&+ 2\lambda \gamma \left( \left( 2 - s \right) k + 2\left( \alpha - c \right) \right) \left\langle z_{k} - z_{*}, v_{k} \right\rangle + \dfrac{1}{2} \gamma ^{2} \left( \left( 2 -s \right) k + 2\left( \alpha - c \right) \right) \\&\quad \left( sk + 2c\right) \left\Vert v_{k} \right\Vert ^{2} \\ =&\ 2 \lambda \left( \alpha - 1 \right) \left\Vert z_{k} - z_{*} \right\Vert ^{2} + 4 \lambda k \left\langle z_{k} - z_{*}, z_{k} - z_{k-1} + \gamma v_{k} \right\rangle \\&+ \dfrac{k^{2}}{2} \left\Vert 2 \left( z_{k} - z_{k-1} \right) + s \gamma v_{k} \right\Vert ^{2} + 4\left( \alpha - c \right) \lambda \gamma \langle z_{k} - z_{*}, v_{k} \rangle \\&+ \frac{1}{2}\gamma ^2\big (\left( 2 - s\right) k + 2\left( \alpha - c \right) \big )\left( sk + 2c\right) \left\Vert v_{k} \right\Vert ^{2}. \end{aligned} \end{aligned}$$\end{document}

We choose \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\underline{\lambda }} \left( \alpha ,c,s \right)< \lambda _{1}< \lambda _{2} < {\overline{\lambda }} \left( \alpha ,c,s \right) $$\end{document} , and get

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&{\mathcal {E}}_{\lambda _{2},s,k} - {\mathcal {E}}_{\lambda _{1},s,k} \\&\quad = 4 \left( \lambda _{2} - \lambda _{1} \right) \bigg ( \dfrac{1}{2} \left( \alpha - 1 \right) \left\Vert z_{k} - z_{*} \right\Vert ^{2} + k \left\langle z_{k} - z_{*}, z_{k} - z_{k-1} + \gamma v_{k} \right\rangle + \gamma \left( \alpha - c \right) \langle z_k - z_{*}, v_k\rangle \bigg ) \\&\quad = \ 4 \left( \lambda _{2} - \lambda _{1} \right) \big ( p_{k} + \gamma \left( \alpha - c \right) \langle z_k - z_{*}, v_k\rangle \big ), \end{aligned}$$\end{document}

where for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p_{k} := \dfrac{1}{2} \left( \alpha - 1 \right) \left\Vert z_{k} - z_{*} \right\Vert ^{2} + k \left\langle z_{k} - z_{*}, z_{k} - z_{k-1} + \gamma v_{k} \right\rangle . \end{aligned}$$\end{document}

According to (2.19) and (2.20), we have

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 0 \leqslant \lim _{k\rightarrow +\infty } | \langle z_k-z_{*}, v_k\rangle | \leqslant C_3\lim _{k\rightarrow +\infty }\left\Vert v_{k} \right\Vert =0, \end{aligned}$$\end{document}

which, together with the fact that the limit \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } \left( {\mathcal {E}}_{\lambda _{2},s,k} - {\mathcal {E}}_{\lambda _{1},s,k} \right) \in {\mathbb {R}}$$\end{document} exists, leads to

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow + \infty } p_{k} \in {\mathbb {R}}\text { exists}. \end{aligned}$$\end{document}

We define for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} q_{k} := \dfrac{1}{2} \left\Vert z_{k} - z_{*} \right\Vert ^{2} + \gamma \sum _{i = 1}^{k} \left\langle z_{i} - z_{*}, v_{i} \right\rangle , \end{aligned}$$\end{document}

and notice that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 2$$\end{document} it holds

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} q_{k} - q_{k-1}&= \left\langle z_{k} - z_{*}, z_{k} - z_{k-1} \right\rangle - \dfrac{1}{2} \left\Vert z_{k} - z_{k-1} \right\Vert ^{2} + \gamma \left\langle z_{k} - z_{*}, v_{k} \right\rangle , \end{aligned}$$\end{document}

thus

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( \alpha - 1 \right) q_{k} + k \left( q_{k} - q_{k-1} \right) = p_{k} + \left( \alpha - 1 \right) \gamma \sum _{i = 1}^{k} \left\langle z_{i} - z_{*}, v_{i} \right\rangle - \dfrac{k}{2} \left\Vert z_{k} - z_{k-1} \right\Vert ^{2} . \end{aligned}$$\end{document}

Thanks to Proposition 2.7(i), we have \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } k \left\Vert z_{k+1} - z_{k} \right\Vert ^{2} = 0$$\end{document} . So if we can show that the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( \sum _{i = 1}^{k} \left\langle z_{i} - z_{*}, v_{i} \right\rangle \right) _{k \geqslant 1}$$\end{document} also converges, then we know that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow + \infty } \left( \alpha - 1 \right) q_{k} + k \left( q_{k} - q_{k-1} \right) \in {\mathbb {R}}\text { exists} . \end{aligned}$$\end{document}

To show that the series above converges, we first observe that for every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 2$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sum _{i = 2}^{k} \left|\left\langle z_{i} - z_{*}, F \left( w_{i-1} \right) - F \left( z_{i} \right) \right\rangle \right|&\leqslant \sum _{i = 2}^{k} \left\Vert z_{i} - z_{*} \right\Vert \left\Vert F \left( w_{i-1} \right) - F \left( z_{i} \right) \right\Vert \nonumber \\&\leqslant \frac{1}{2} \sum _{i=2}^{k} \frac{1}{i^{2}} \left\Vert z_{i} - z_{*} \right\Vert ^{2} + \frac{1}{2} {\sum _{i=2}^{k} }i^{2} \left\Vert F(w_{i-1}) - F(z_{i}) \right\Vert ^{2} \nonumber \\&\leqslant \frac{1}{2} \sum _{i \geqslant 2} \frac{1}{i^{2}} \left\Vert z_{i} - z_{*} \right\Vert ^{2} + \frac{1}{2} \sum _{i \geqslant 2} i^{2} \left\Vert F(w_{i-1}) - F(z_{i}) \right\Vert ^{2} \nonumber \\&< + \infty , \end{aligned}$$\end{document}

where the first series in (2.26) converges due to (2.19), and the second series converges due to (2.6) and Proposition 2.7(i). This proves that the series \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum _{k \geqslant 2} \left\langle z_{k} - z_{*}, F \left( w_{k-1} \right) - F \left( z_{k} \right) \right\rangle $$\end{document} is absolutely convergent, so convergent. Again using Proposition 2.7(i), we see that the limit

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned}&\lim _{k \rightarrow + \infty } \sum _{i = 1}^{k} \left\langle z_{i} - z_{*}, v_{i} \right\rangle = \lim _{k \rightarrow + \infty } \sum _{i = 1}^{k} \left\langle z_{i} - z_{*}, \xi _{i} + F \left( z_{i} \right) \right\rangle \\&\quad + \lim _{k \rightarrow + \infty } \sum _{i = 1}^{k} \left\langle z_{i} - z_{*}, F \left( w_{i-1} \right) - F \left( z_{i} \right) \right\rangle \in {\mathbb {R}}\end{aligned} \end{aligned}$$\end{document}

exists. Consequently, (2.25) holds. Therefore, we can apply Lemma A.1 to guarantee that the limit \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } q_{k} \in {\mathbb {R}}$$\end{document} also exists. The required boundedness of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} follows from Proposition 2.7(ii) and the fact that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } \sum _{i = 1}^{k} \left\langle z_{i} - z_{*}, v_{i} \right\rangle \in {\mathbb {R}}$$\end{document} exists. Given the definition of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(q_{k})_{k \geqslant 1}$$\end{document} , the latter property also guarantees that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } \left\Vert z_{k} - z_{*} \right\Vert \in {\mathbb {R}}$$\end{document} exists. The hypothesis (i) in the Opial Lemma (see Lemma A.3) is therefore fulfilled.

Let z be a weak sequential cluster point of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} , which means that there exists a subsequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k_{l}} \right) _{l \geqslant 0}$$\end{document} which converges weakly to z as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l \rightarrow +\infty $$\end{document} . It follows from Proposition 2.7(ii) that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _{k_{l}} + F \left( z_{k_{l}} \right) $$\end{document} strongly converges to 0 as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l \rightarrow +\infty $$\end{document} . Since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F :{\mathcal {H}} \rightarrow {\mathcal {H}} $$\end{document} is a single-valued monotone and L-Lipschitz continuous operator, it is a maximally monotone operator (see [5, Corollary 20.28]) with full domain. Therfore, the sum \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M+F$$\end{document} is also a maximally monotone operator, (see [5, Corollary 25.5]), given that M is maximally monotone. The fact that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _{k_{l}} \in M(z_{k_{l}})$$\end{document} for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l \geqslant 0$$\end{document} , and the maximal monotonicity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M + F$$\end{document} implies that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \in \left( M + F \right) \left( z \right) $$\end{document} , see [5, Proposition 20.33], meaning that hypothesis (ii) of Lemma A.3 is also verified. The weak convergence of the iterates to an element in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{Zer}(M+F)$$\end{document} is therefore a consequence of the Opial Lemma. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

We will conclude the convergence analysis by proving that the Fast RFB algorithm does indeed have convergence rates of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$o(\frac{1}{k})$$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} .

Theorem 2.9

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{*} \in \textrm{Zer}(M+F)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( z_{k} \right) _{k \geqslant 0}$$\end{document} be the sequence generated by Algorithm 1. The following holds as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ k \rightarrow +\infty $$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert z_{k} - z_{k-1} \right\Vert = o \left( \dfrac{1}{k} \right) , \quad \Vert \xi _k + F(z_k) \Vert = o \left( \frac{1}{k} \right) , \quad \langle \xi _{k} + F(z_{k}),z_{k} - z_{*}\rangle =&o \left( \frac{1}{k} \right) ,\\ r_{tan}(z_{k})= \textrm{dist}(0, M(z_k) + F(z_k)) = o \left( \frac{1}{k} \right) , \quad r_{fix}(z_{k})= \left\Vert z_k-J_{\gamma M}\big (z_k-\gamma F \left( z_{k} \right) \big )\right\Vert =&o \left( \dfrac{1}{k} \right) . \end{aligned}$$\end{document}

Proof

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s \in \left( 1+ \frac{\alpha }{4 c - \alpha }, 2 \right) $$\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \leqslant {\underline{\lambda }} \left( \alpha ,c, s \right) < {\overline{\lambda }} \left( \alpha ,c ,s \right) \leqslant \frac{s\alpha }{4}$$\end{document} be the parameters provided by Proposition 2.7(iii) with the property that, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\underline{\lambda }} \left( \alpha ,c,s \right)< \lambda < {\overline{\lambda }} \left( \alpha ,c,s \right) $$\end{document} the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\mathcal {E}}_{\lambda ,s,k} \right) _{k \geqslant 1}$$\end{document} , is convergent.

We choose \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda \in \left( {\underline{\lambda }} \left( \alpha ,c,s \right) , {\overline{\lambda }} \left( \alpha ,c,s \right) \right) $$\end{document} and set for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} h_{s,k} := \dfrac{k^{2}}{2} \left( \left\Vert 2 \left( z_{k} - z_{k-1} \right) + s \gamma v_{k} \right\Vert ^{2} + (2-s)s \gamma ^{2} \left\Vert v_{k} \right\Vert ^{2} \right) . \end{aligned}$$\end{document}

We are going to show that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow + \infty } h_{s,k} = 0 . \end{aligned}$$\end{document}

This assertion will immediately imply

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow +\infty } k \left\Vert 2 \left( z_{k} - z_{k-1} \right) + s \gamma v_{k} \right\Vert = \lim \limits _{k \rightarrow +\infty } k \left\Vert v_{k} \right\Vert = 0, \end{aligned}$$\end{document}

and further \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } k\left\Vert z_{k} - z_{k-1} \right\Vert = 0$$\end{document} . The fact that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim _{k \rightarrow + \infty } k \left\Vert \xi _{k} + F \left( z_{k} \right) \right\Vert = 0 \end{aligned}$$\end{document}

will follow from (2.6), (2.21) and (2.22), since

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} 0 \leqslant \lim \limits _{k \rightarrow + \infty } k \left\Vert \xi _{k} + F \left( z_{k} \right) \right\Vert \leqslant \lim \limits _{k \rightarrow + \infty } k \left\Vert v_{k} - v_{k-1} \right\Vert + \lim \limits _{k \rightarrow + \infty } k \left\Vert v_{k} \right\Vert = 0 . \end{aligned}$$\end{document}

The convergence statement above will imply that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim _{k \rightarrow + \infty } k \ \textrm{dist}\left( 0, M(z_k)+ F \left( z_{k} \right) \right) = 0, \end{aligned}$$\end{document}

and, by using the boundedness of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(z_k)_{k \geqslant 0}$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow + \infty } k \left\langle \xi _{k} + F(z_{k}),z_{k} - z_{*} \right\rangle = 0. \end{aligned}$$\end{document}

The convergence rate

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_{fix}(z_{k})= \left\Vert z_k-J_{\gamma M}\big (z_k-\gamma F \left( z_{k} \right) \big )\right\Vert = o \left( \dfrac{1}{k } \right) \ \text{ as } \ k \rightarrow +\infty \end{aligned}$$\end{document}

is a consequence of (1.8).

What remains to be shown is that (2.27) does in fact hold. In view of (2.23), we have for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} & h_{s,k} = {\mathcal {E}}_{\lambda ,s,k} - 4 \lambda p_{k} - \gamma ^2\left( \big ( \left( 2 - s\right) c + \left( \alpha - c \right) s \big )k + 2\left( \alpha - c \right) c\right) \left\Vert v_{k} \right\Vert ^{2}\\ & \quad -4\left( \alpha - c \right) \lambda \gamma \langle z_k - z_{*},v_k\rangle . \end{aligned}$$\end{document}

From Proposition 2.7(iii) and (2.24), we have \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } {\mathcal {E}}_{\lambda ,s,k} \in {\mathbb {R}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } p_{k} \in {\mathbb {R}}$$\end{document} , respectively. From (2.20) it yields

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow +\infty } k \left\Vert v_{k} \right\Vert ^{2} = 0\quad \text {and} \quad \lim \limits _{k \rightarrow +\infty } \left\Vert v_{k} \right\Vert ^{2} = 0, \end{aligned}$$\end{document}

which implies the existence of

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim \limits _{k \rightarrow + \infty } h_{s,k} \in {\mathbb {R}}. \end{aligned}$$\end{document}

To get the precise value, we recall that the summability results in Proposition 2.7(i) guarantee that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sum _{k \geqslant 1} \dfrac{1}{k} h_{s,k} \leqslant 4 \sum _{k\geqslant 1} \left\Vert z_{k} - z_{k-1} \right\Vert ^{2} + \frac{1}{2}(2+s)s \gamma ^{2} \sum _{k \geqslant 1} k \left\Vert v_{k} \right\Vert ^{2} < + \infty . \end{aligned}$$\end{document}

We must have \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lim _{k \rightarrow + \infty } h_{k} = 0$$\end{document} , and the proof is finished. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

A fast primal-dual full splitting algorithm

In this section we will apply the Fast RFB algorithm to the solution of the saddle point problem and further to a convex optimization problem with linear cone constraints. For the resulting primal-dual full splitting methods, we will formulate the convergence and convergence rates statements that follow from those proved in the general setting.

Convex-concave saddle point problems with smooth coupling term

First, we consider the application the proposed fast algorithm to the saddle point problem (1.1)

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{x\in {\mathcal {X}}}\max _{\lambda \in {\mathcal {Y}}} \Psi \left( x , \lambda \right) {:=} f(x) + \Phi \left( x , \lambda \right) - g(\lambda ), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {X}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ {\mathcal {Y}}$$\end{document} are two real Hilbert spaces, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f :{\mathcal {X}}\rightarrow {\mathbb {R}}\cup \left\{ + \infty \right\} $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g :{\mathcal {Y}}\rightarrow {\mathbb {R}}\cup \left\{ + \infty \right\} $$\end{document} are proper, convex, and lower semicontinuous functions, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi :{\mathcal {X}}\times {\mathcal {Y}}\rightarrow {\mathbb {R}}$$\end{document} is a differentiable function with Lipschitz continuous gradient, convex in x and concave in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda $$\end{document} .

Finding a solution to (1.1) reduces to the solving of the monotone inclusion (1.3), for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {H}}:= {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M : {\mathcal {H}}\rightarrow 2^{\mathcal {H}}, M(x,\lambda ) = (\partial f(x), \partial g(\lambda ))$$\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F : {\mathcal {H}}\rightarrow {\mathcal {H}}, F(x,\lambda ) = (\nabla _x \Phi (x,\lambda ), -\nabla _\lambda \Phi (x,\lambda ))$$\end{document} . Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L >0$$\end{document} be the Lipschitz constant of F. Applying the Fast RFB algorithm to this particular setting leads to the following full splitting algorithms, which have the feature that each function is evaluated separately: the non-smooth tones f and g via their proximal operators, and the smooth one via a gradient step. The following result is based on the theorems 2.8 and 2.9.

Theorem 3.1

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{*} , \lambda _{*} \right) \in {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} be a saddle point of (1.1), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k},\lambda _{k} \right) _{k \geqslant 0}$$\end{document} the sequence generated by Algorithm 2. The following statements are true:

• (i)the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k} , \lambda _{k} \right) _{k \geqslant 0}$$\end{document} converges weakly to a saddle point of (1.1);
• (ii)the following holds as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert x_{k} - x_{k-1} \right\Vert = o \left( \dfrac{1}{k} \right) \,\, \text { and }\,\, \left\Vert \lambda _{k} - \lambda _{k-1} \right\Vert&= o \left( \dfrac{1}{k} \right) \\ \left\Vert u_k + \nabla _{x} \Phi (x_{k} , \lambda _{k}) \right\Vert = o \left( \dfrac{1}{k} \right) \,\, \text { and } \,\, \left\Vert v_{k}- \nabla _{\lambda } \Phi (x_{k} , \lambda _{k}) \right\Vert&= o \left( \dfrac{1}{k} \right) \\ \left\langle x_{k} - x_{*} , u_k + \nabla _{x} \Phi (x_{k} , \lambda _{k})\right\rangle + \left\langle \lambda _{k} - \lambda _{*} , v_{k} - \nabla _{\lambda } \Phi (x_{k} , \lambda _{k})\right\rangle&= o \left( \dfrac{1}{k} \right) \\ \Psi \left( x_{k} , \lambda _{*} \right) -\Psi \left( x_{*} , \lambda _{k} \right)&= o \left( \dfrac{1}{k } \right) , \end{aligned}$$\end{document}

where, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 0$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} u_{k+1} :&= \frac{1}{\gamma }y_{1,k}-\nabla _{x} \Phi \left( w_{1,k},w_{2,k}\right) - \frac{1}{\gamma }x_{k+1} \in \partial f(x_{k+1}) \\ v_{k+1}:&= \frac{1}{\gamma }y_{2,k} + \nabla _{\lambda } \Phi \left( w_{1,k},w_{2,k}\right) -\frac{1}{\gamma }\lambda _{k+1} \in \partial g \left( \lambda _{k+1} \right) . \end{aligned}$$\end{document}

Proof

Algorithm 2 is a special instance of Algorithm 1 when applied to the monotone inclusion (1.2), for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_k:=(x_{k},\lambda _{k})$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_k:=(y_{1,k},y_{2,k})$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_k:=(w_{1,k},w_{2,k})$$\end{document} for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 0$$\end{document} . The third block Algorithm 1 is obviously equivalent for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document} to

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{aligned}&x_{k+1}=\textrm{prox}_{\gamma f}\left( y_{1,k} - \gamma \nabla _{x} \Phi \left( w_{1,k},w_{2,k}\right) \right) ,\\&\lambda _{k+1} = \textrm{prox}_{\gamma g}\left( y_{2,k} + \gamma \nabla _{\lambda } \Phi \left( w_{1,k},w_{2,k}\right) \right) . \end{aligned}\right. \end{aligned}$$\end{document}

The sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\xi _{k})_{k \geqslant 1}$$\end{document} introduced in Proposition 2.1, and defined for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document} as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \xi _{k}=\frac{1}{\gamma }\left( y_{k-1} - z_{k}\right) - F(w_{k-1}) \end{aligned}$$\end{document}

plays a crucial role in the formulation of the convergence rates. In the context Algorithm 2, we have for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _k:= (u_k,v_{k})$$\end{document} and all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{aligned} u_{k}&=\frac{1}{\gamma }y_{1,k-1} - \nabla _{x} \Phi \left( w_{1,k-1},w_{2,k-1}\right) -\frac{1}{\gamma }x_{k},\\ v_{k}&= \frac{1}{\gamma }y_{2,k-1} + \nabla _{\lambda } \Phi \left( w_{1,k-1},w_{2,k-1}\right) -\frac{1}{\gamma }\lambda _{k}. \end{aligned}\right. \end{aligned}$$\end{document}

Furthermore, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _{k} \in M(z_k)$$\end{document} becomes for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{aligned} u_{k}&\in \partial f(x_{k}),\\ v_{k}&\in \partial g(\lambda _k). \end{aligned}\right. \end{aligned}$$\end{document}

The weak convergence of the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k} , \lambda _{k} \right) _{k \geqslant 0}$$\end{document} to a saddle point of (1.1) is a direct consequence of Theorem 2.8. In addition, Theorem 2.9 yields

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert z_k - z_{k-1} \right\Vert&= o \left( \dfrac{1}{k } \right) , \quad \left\Vert \xi _k + F(z_k)\right\Vert = o \left( \dfrac{1}{k} \right) , \quad \left\langle z_k - z_{*} , \xi _k + F(z_k) \right\rangle = o \left( \dfrac{1}{k } \right) \quad \text{ as } \quad \\ &k \rightarrow +\infty , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_*:=(x_*, \lambda _*)$$\end{document} . From the first statement, we obtain

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\left\Vert x_k - x_{k-1} \right\Vert = o \left( \dfrac{1}{k } \right) \quad \text{ and } \quad \left\Vert \lambda _k-\lambda _{k-1} \right\Vert = o \left( \dfrac{1}{k } \right) \\&\quad \text{ as } \ k \rightarrow +\infty . \end{aligned}$$\end{document}

For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_k=(x_{k},\lambda _{k})$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _k= (u_k,v_k)$$\end{document} , given by (3.1), the other two statements become

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert \begin{pmatrix} u_k + \nabla _{x} \Phi (x_{k} , \lambda _{k}), v_{k} - \nabla _{\lambda } \Phi (x_{k} , \lambda _{k}) \end{pmatrix} \right\Vert = o \left( \dfrac{1}{k} \right) \ \text{ as } \ k \rightarrow +\infty , \end{aligned}$$\end{document}

and

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\langle \begin{pmatrix} x_{k} - x_{*} , \lambda _{k} - \lambda _{*} \end{pmatrix} , \begin{pmatrix} u_k + \nabla _{x} \Phi (x_{k} , \lambda _{k}), v_{k} - \nabla _{\lambda } \Phi (x_{k} , \lambda _{k}) \end{pmatrix} \right\rangle = o \left( \dfrac{1}{k } \right) \ \text{ as } \ k \rightarrow +\infty , \end{aligned}$$\end{document}

respectively. Obviously,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert u_k + \nabla _{x} \Phi (x_{k} , \lambda _{k}) \right\Vert = o \left( \dfrac{1}{k } \right) \quad \text{ and } \quad \left\Vert v_{k} - \nabla _{\lambda } \Phi (x_{k} , \lambda _{k}) \right\Vert = o \left( \dfrac{1}{k} \right) \ \text{ as } \ k \rightarrow +\infty . \end{aligned}$$\end{document}

Using the convexity of f and g, and the convexity and concavity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi $$\end{document} in its first and second variables respectively, it follows for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document} that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\left\langle \begin{pmatrix} x_{k} - x_{*} , \lambda _{k} - \lambda _{*} \end{pmatrix} , \begin{pmatrix} u_k + \nabla _{x} \Phi (x_{k} , \lambda _{k}), v_{k} - \nabla _{\lambda } \Phi (x_{k} , \lambda _{k}) \end{pmatrix} \right\rangle \\&\quad = \langle u_k + \nabla _{x} \Phi (x_{k} , \lambda _{k}),x_{k}-x_{*}\rangle +\langle v_{k}- \nabla _{\lambda } \Phi (x_{k} , \lambda _{k}) ,\lambda _{k}-\lambda _{*}\rangle \\&\quad = \langle u_k , x_{k}-x_{*}\rangle + \langle \nabla _{x} \Phi (x_{k} , \lambda _{k}), x_{k}-x_{*} \rangle +\langle v_{k},\lambda _{k}-\lambda _{*}\rangle +\langle - \nabla _{\lambda } \Phi (x_{k} , \lambda _{k}), \lambda _{k}-\lambda _{*} \rangle \\&\quad \geqslant f\left( x_{k} \right) - f(x_*) + \Phi \left( x_{k},\lambda _{*} \right) - \Phi \left( x_*,\lambda _{*} \right) + g(\lambda _k) - g \left( \lambda _{*} \right) \\ &\quad - \Phi \left( x_{*},\lambda _{k} \right) + \Phi \left( x_{*},\lambda _{*} \right) \\&\quad = \Psi \left( x_{k} , \lambda _{*} \right) -\Psi \left( x_{*} , \lambda _{k} \right) \geqslant 0. \end{aligned}$$\end{document}

This yields

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\Psi \left( x_{k} , \lambda _{*} \right) -\Psi \left( x_{*} , \lambda _{k} \right) = o \left( \dfrac{1}{k } \right) \quad \text{ as } \quad k \rightarrow +\infty . \end{aligned}$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Composite convex optimization problems

The aim of this subsection is to show that the algorithm proposed in this paper leads to a fast primal-dual full splitting algorithm which solves the composite convex optimization problem

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{x\in {\mathcal {X}}} f(x)+g(Ax)+h(x), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {X}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {Y}}$$\end{document} are two real Hilbert spaces, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f, g :{\mathcal {X}}\rightarrow {\mathbb {R}}\cup \left\{ + \infty \right\} $$\end{document} are proper, convex, and lower semicontinuous functions, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h: {\mathcal {X}}\rightarrow {\mathbb {R}}$$\end{document} is a convex and differentiable function such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla h$$\end{document} is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{\nabla h}$$\end{document} -Lipschitz continuous, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A: {\mathcal {X}}\rightarrow {\mathcal {Y}}$$\end{document} is a linear continuous operator. The corresponding Fenchel dual problem is

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \max _{\lambda \in {\mathcal {Y}}} -\big (f+h\big )^*\big (-A^*\lambda \big )-g^*(\lambda ). \end{aligned}$$\end{document}

If \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(x_*, \lambda _*) \in {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} is a primal-dual solution of (3.2)-(3.3), in other words, a solution of the KKT system

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\left\{ \begin{array}{ll} 0 \in \partial f \left( x\right) +\nabla h \left( x\right) + A^*\lambda \\ A x \in \partial g^*(\lambda ), \end{array}\right. } \end{aligned}$$\end{document}

then \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_*\in {\mathcal {X}}$$\end{document} is an optimal solution of the primal problem, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _*\in {\mathcal {Y}}$$\end{document} is an optimal solution of the dual problem, and strong duality holds. Viceversa, under suitable constraint qualification (see [5, 8]), if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_* \in {\mathcal {X}}$$\end{document} is a solution of (3.2), then there exists an optimal solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _* \in {\mathcal {Y}}$$\end{document} of (3.3) such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(x_*, \lambda _*)$$\end{document} solves (3.4). It is easy to see that the solutions of (3.4) are nothing else than the saddle points of the Lagrangian

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{{\mathcal {L}}}} \left( x,\lambda \right) := f \left( x \right) +h \left( x \right) + \left\langle \lambda , Ax \right\rangle - g^*(\lambda ), \end{aligned}$$\end{document}

which provides a compelling motivation for treating this problem as a specific instance of the framework developed in the previous subsection. Thus, the algorithm and the convergence theorem of the previous subsection lead to the following statements, respectively.

Theorem 3.2

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{*} , \lambda _{*} \right) \in {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} be a primal-dual optimal solution of (3.2)-(3.3), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k},\lambda _{k} \right) _{k \geqslant 0}$$\end{document} the sequence generated by Algorithm 3. The following statements are true:

• (i)the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k} , \lambda _{k} \right) _{k \geqslant 0}$$\end{document} converges weakly to a primal-dual optimal solution of (3.2)-(3.3);
• (ii)the following holds as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert x_{k} - x_{k-1} \right\Vert = o \left( \dfrac{1}{k} \right) \quad \text {and} \quad \left\Vert \lambda _{k} - \lambda _{k-1} \right\Vert&= o \left( \dfrac{1}{k} \right) \\ \left\Vert u_k + \nabla h\left( x_{k} \right) + A^{*}\lambda _{k}\right\Vert = o \left( \dfrac{1}{k} \right) \quad \text {and} \quad \left\Vert v_{k} -A x_{k} \right\Vert&= o \left( \dfrac{1}{k} \right) \\ {{{\mathcal {L}}}} \left( x_{k} , \lambda _{*} \right) - {{{\mathcal {L}}}}\left( x_{*} , \lambda _{k} \right)&= o \left( \dfrac{1}{k } \right) , \end{aligned}$$\end{document}

where, for every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 0$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} u_{k+1}&:= \frac{1}{\gamma }y_{1,k}- \nabla h\left( w_{1,k} \right) - A^{*}w_{2,k} - \frac{1}{\gamma }x_{k+1} \in \partial f(x_{k+1}),\\ v_{k+1}&:= \frac{1}{\gamma }y_{2,k} + A w_{1,k}-\frac{1}{\gamma }\lambda _{k+1}\in \partial g^{*}(\lambda _{k+1}). \end{aligned}$$\end{document}

Remark 3.3

To the best of our knowledge, these are the strongest convergence rate results among primal dual full-splitting algorithms for composite convex optimization problems. In the merely convex case, the algorithms proposed in the literature typically achieve an ergodic convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k}\right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the primal dual gap (see, e.g., [17]). In contrast, Algorithm 3 establishes two advances: (i) global convergence of the entire primal dual sequence to a primal dual optimal solution, and (ii) a nonergodic convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$o\left( \frac{1}{k}\right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} for the primal and dual discrete velocities, the tangent residual and the primal-dual gap.

Convex optimization problems with linear cone constraints

In this subsection, we will study the optimization problem (1.15)

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min \ &f \left( x \right) +h \left( x \right) ,\\ \text {subject to} \&Ax - b \in -{\mathcal {K}} \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {X}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {Y}}$$\end{document} are real Hilbert spaces, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {K}}$$\end{document} is a nonempty, convex and closed cone in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {Y}}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f :{\mathcal {X}}\rightarrow {\mathbb {R}}\cup \left\{ + \infty \right\} $$\end{document} is a proper, convex, and lower semicontinuous function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h :{\mathcal {X}}\rightarrow {\mathbb {R}}$$\end{document} is a convex and differentiable function such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla h$$\end{document} is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{\nabla h}$$\end{document} -Lipschitz continuous, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A :{\mathcal {X}}\rightarrow {\mathcal {Y}}$$\end{document} is a linear continuous operator.

We aim to design efficient algorithms for detecting primal-dual optimal solutions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(x_*, \lambda _*)$$\end{document} of (1.15), which correspond to solutions of the associated system of optimality conditions

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\left\{ \begin{array}{ll} 0 \in \partial f \left( x\right) +\nabla h \left( x\right) + A^*\lambda \\ A x - b \in N_{{\mathcal {K}}^{*}}(\lambda ), \end{array}\right. } \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {K}}^{*}:=\{\lambda \in {\mathcal {Y}}: \langle \lambda , \zeta \rangle \geqslant 0 \ \forall \zeta \in {\mathcal {K}}\}$$\end{document} denotes the dual cone of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {K}}$$\end{document} . Given a primal-dual optimal solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(x_*, \lambda _*)$$\end{document} of (1.15), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_*$$\end{document} is an optimal solution of (1.15) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _*$$\end{document} is an optimal solution of the Lagrange dual problem of (1.15).

Solving (3.6) is equivalent to finding the saddle points of the associated Lagrangian

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{{\mathcal {L}}}} \left( x,\lambda \right) := f \left( x \right) +h \left( x \right) + \left\langle \lambda , Ax - b \right\rangle - \delta _{{\mathcal {K}}^{*}}\left( \lambda \right) , \end{aligned}$$\end{document}

which is a specific case of (3.5). Therefore, it follows from Algorithm 3 and Theorem 3.2 that obtain the following statements.

Theorem 3.4

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{*} , \lambda _{*} \right) \in {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} be a primal-dual optimal solution of (1.15), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k},\lambda _{k} \right) _{k \geqslant 0}$$\end{document} the sequence generated by Algorithm 4. The following statements are true:

• (i)the sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{k} , \lambda _{k} \right) _{k \geqslant 0}$$\end{document} converges weakly to a primal-dual optimal solution of (1.15);
• (ii)the following holds as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\Vert x_{k} - x_{k-1} \right\Vert = o \left( \dfrac{1}{k} \right) \quad \text {and} \quad \left\Vert \lambda _{k} - \lambda _{k-1} \right\Vert&= o \left( \dfrac{1}{k} \right) \\ \left\Vert u_k + \nabla h\left( x_{k} \right) + A^{*}\lambda _{k}\right\Vert = o \left( \dfrac{1}{k} \right) \quad \text {and} \quad \left\Vert v_{k} -A x_{k}+b \right\Vert&= o \left( \dfrac{1}{k} \right) \\ {{{\mathcal {L}}}} \left( x_{k} , \lambda _{*} \right) - {{{\mathcal {L}}}}\left( x_{*} , \lambda _{k} \right)&= o \left( \dfrac{1}{k } \right) , \\ \left|\left( f+h \right) \left( x_{k} \right) - \left( f+h \right) \left( x_{*} \right) \right|= o \left( \dfrac{1}{k} \right) \quad \text {and} \quad \left| \langle \lambda _{k}, A x_{k} - b\rangle \right|&= o \left( \frac{1}{k} \right) , \end{aligned}$$\end{document}

where, for every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 0$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} u_{k+1}&:= \frac{1}{\gamma }y_{1,k}- \nabla h\left( w_{1,k} \right) - A^{*}w_{2,k} - \frac{1}{\gamma }x_{k+1} \in \partial f(x_{k+1}),\\ v_{k+1}&:= \frac{1}{\gamma }y_{2,k} + A w_{1,k}-\frac{1}{\gamma }\lambda _{k+1}-b\in N_{{\mathcal {K}}^{*}}(\lambda _{k+1}). \end{aligned}$$\end{document}

Proof

Given a primal-dual optimal solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( x_{*} , \lambda _{*} \right) \in {\mathcal {X}}\times {\mathcal {Y}}$$\end{document} of (1.15), it holds (see, for instance, [5, Proposition 27.17]) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{*}\in {\mathcal {K}}^{*}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Ax_{*} -b \in -{\mathcal {K}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\langle \lambda _{*}, Ax_{*} - b\rangle =0$$\end{document} . The weak convergence of the sequence of primal-dual iterates to a primal-dual solution of (1.15), and the statements in the first three blocks of (ii) follow directly from Theorem 3.1.

In addition, we have for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document} that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_k \in N_{{\mathcal {K}}^{*}}(\lambda _{k})$$\end{document} , which implies that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{k}\in {\mathcal {K}}^{*}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_{k} \in -{\mathcal {K}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\langle \lambda _{k}, v_{k}\rangle =0$$\end{document} . Therefore,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left| \langle \lambda _{k}, A x_{k} - b\rangle \right| =\left| \langle \lambda _{k}, A x_{k} - b - v_{k}\rangle \right| \leqslant \Vert \lambda _{k}\Vert \Vert A x_{k} - b - v_{k}\Vert . \end{aligned}$$\end{document}

Since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\lambda _k)_{k \geqslant 0}$$\end{document} is bounded and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\Vert v_{k} - A x_k +b \right\Vert = o \left( \frac{1}{k} \right) $$\end{document} , we conclude that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left| \langle \lambda _{k}, A x_{k} - b\rangle \right| = o \left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} .

Since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u_k \in \partial f(x_k)$$\end{document} , we have for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (f+h)(x_k) - (f+h)(x_*) \leqslant&\ \langle u_k + \nabla h(x_k), x_k -x_* \rangle \\ =&\ \langle u_k + \nabla h(x_k) + A^*\lambda _k, x_k -x_* \rangle + \langle \lambda _k, b- Ax_k \rangle \\ \leqslant&\ \Vert u_k + \nabla h(x_k) + A^*\lambda _k\Vert \Vert x_k - x_*\Vert + |\langle \lambda _k, b- Ax_k \rangle |. \end{aligned}$$\end{document}

On the other hand, since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-A^{*}\lambda _{*} \in \partial (f+h) \left( x_{*} \right) $$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\langle \lambda _{*}, Ax_{*} - b\rangle =0$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\langle \lambda _{*},v_k\rangle \leqslant 0$$\end{document} , we for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (f+h) \left( x_{k} \right)&\geqslant (f+h) \left( x_{*} \right) - \langle A^{*}\lambda _{*} , x_{k} - x_{*} \rangle = (f+h)(x_{*}) - \langle \lambda _{*} , Ax_{k} - b \rangle \\&= (f+h)(x_{*}) -\langle \lambda _{*} , - v_{k}+ A x_{k} - b \rangle - \langle \lambda _{*},v_k\rangle \\&\geqslant (f+h)(x_{*}) -\Vert \lambda _{*} \Vert \Vert v_{k} -A x_{k} + b \Vert . \end{aligned}$$\end{document}

Therefore, we obtain for every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \geqslant 1$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\ |(f+h) \left( x_{k} \right) - (f+h)(x_{*})| \\ \leqslant&\ \max \{\Vert u_k + \nabla h(x_k) + A^*\lambda _k\Vert \Vert x_k - x_*\Vert + |\langle \lambda _k, b- Ax_k \rangle |,\Vert \lambda _{*} \Vert \Vert v_{k} -A x_{k} + b \Vert \}. \end{aligned}$$\end{document}

Since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(x_k)_{k \geqslant 0}$$\end{document} is bounded, the right-hand side converges to zero with a convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$o \left( \frac{1}{k} \right) $$\end{document} as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \rightarrow +\infty $$\end{document} . \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Numerical experiments

In this section, we present numerical experiments to illustrate the convergence rates established for the proposed fast method and compare our algorithm with those in the existing literature.

The role of the algorithm parameters

In this subsection, we investigated the influence of the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} and c on the algorithm’s convergence behavior. Consider the convex optimization problem

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min&\left\Vert x \right\Vert _{1} + \frac{1}{2}\langle x,Hx\rangle - \langle x,h\rangle , \\ \text {such that} \,&Ax - b \in -{\mathbb {R}}_{+}^{n}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbb {R}}_{+}^{n}$$\end{document} denotes the nonnegative orthant of in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbb {R}}^{n}$$\end{document} ,

$$[eqn]$$

The associated Lagrangian is

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {L}}\left( x , \lambda \right) = \left\Vert x \right\Vert _{1} + \frac{1}{2}\langle x,Hx\rangle - \langle x,h\rangle + \left\langle \lambda , Ax - b \right\rangle - \delta _{{\mathbb {R}}_{+}^{n}} \left( \lambda \right) . \end{aligned}$$\end{document}

The numerical experiments were conducted with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=1000$$\end{document} and a maximum of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$10^4$$\end{document} iterations. Guided by theoretical insights, various choices for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} and c were explored, with the stepsize selected as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \gamma = \frac{0.99}{2\sqrt{\left( \left\Vert H \right\Vert + \left\Vert A \right\Vert \right) ^2 + \left\Vert A \right\Vert ^2}}. \end{aligned}$$\end{document}

Fig. 1. The impact of the parameter c on the convergence behavior of the discrete velocity, the tangent residual, the primal-dual gap, and the function values when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =3$$\end{document}

Figures 1,2,3,4 depict the convergence performance in terms of the discrete velocity, tangent residual, primal-dual gap, and function values, for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =3, 5, 10$$\end{document} and 20, and different values of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c \in \left( \frac{\alpha }{2}, \alpha -1 \right) $$\end{document} . All the results are plotted on a semilog scale, namely, by taking the logarithm of the measured quantities. The results demonstrate that increasing c within the allowable range enhances the convergence behavior of the proposed algorithm. Furthermore, larger values of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} significantly improve the algorithm’s convergence, with the impact of c lead to improved convergence of the algorithm and that for higher values of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} the impact on c on performance becoming more pronounced as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} increases. We also adopt a convergence rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}\left( \frac{1}{k}\right) $$\end{document} as a predefined reference standard. When compared against this baseline, our methods exhibit substantially superior performance.Fig. 2. The impact of the parameter c on the convergence behavior of the discrete velocity, the tangent residual, the primal-dual gap, and the function values when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =5$$\end{document} Fig. 3. The impact of the parameter c on the convergence behavior of the discrete velocity, the tangent residual, the primal-dual gap, and the function values when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =10$$\end{document} Fig. 4. The impact of the parameter c on the convergence behavior of the discrete velocity, the tangent residual, the primal-dual gap, and the function values when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =20$$\end{document}

Comparisons with other algorithms

In this subsection, we will compare the performance of the Fast RFB algorithm with that of other algorithms from the literature when it comes to finding saddle points of

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {L}}\left( x , \lambda \right) = \left\Vert x \right\Vert _{1} + \frac{1}{2}\langle x,Hx\rangle - \langle x,h\rangle + \left\langle \lambda , Ax - b \right\rangle , \end{aligned}$$\end{document}

which is the Lagrangian associated with the optimization problem with linear equality constraints

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min&\left\Vert x \right\Vert _{1} + \frac{1}{2}\langle x,Hx\rangle - \langle x,h\rangle .\\ \text {such that} \,&Ax =b \end{aligned}$$\end{document}

The matrices \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A, H \in {\mathbb {R}}^{n \times n}$$\end{document} and the vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h, b \in {\mathbb {R}}^n$$\end{document} are chosen as in the previous subsection. This problem amounts to solving the monotone inclusion problem (1.3) for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M(x,\lambda ) = (\partial \Vert \cdot \Vert _1(x), 0)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F(x,\lambda ) = (Hx - h + A^*\lambda , b-Ax)$$\end{document} . The Lipschitz constant L of the operator F is taken as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L = \sqrt{\left( \left\Vert H \right\Vert + \left\Vert A \right\Vert \right) ^2 + \left\Vert A \right\Vert ^2}. \end{aligned}$$\end{document}

Fig. 5A comparison of various methods in terms of discrete velocity, tangent residual, primal-dual gap, and function values for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=1000$$\end{document}

In the following, we present the algorithms used in the numerical experiments together with their corresponding parameters:

1. EG: Extragradient method (1.7) (see [1, 29, 52]) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\frac{0.99}{L}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta = 1$$\end{document} ;
2. OGDA: Optimistic Gradient Descent Ascent method (1.9) (see [47, 52]) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\frac{0.99}{2L}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta = 1$$\end{document} ;
3. FRB: Forward-Reflected-Backward method (1.12) (see [14, 40]) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\frac{0.99}{2L}$$\end{document} ;
4. RFB: Reflected Forward-Backward method (1.13) (see [15]) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\frac{0.99(\sqrt{2}-1)}{L}$$\end{document} ;
5. PEAG: Past Extra-Anchored Gradient method (see [49, 52]) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\sqrt{\frac{2}{17}}\frac{0.99}{L}$$\end{document} ;
6. ARG: Accelerated Reflected Gradient method (see [14]) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\frac{0.99}{\sqrt{24}L}$$\end{document} ;
7. Fast RFB: our Algorithm 3 (for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\mathcal {K}}}} = \{0\}$$\end{document} ) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\frac{0.99}{2L}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=\frac{\alpha + 0.1(\alpha -2)}{2}$$\end{document} , for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =5$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =10$$\end{document} . Figure 5 presents, on a semi-logarithmic scale, the convergence behavior of the discrete velocity, tangent residual, primal-dual gap, and function values generated by each of the above-mentioned algorithms for the case \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=1000$$\end{document} after \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$5\times 10^5$$\end{document} iterations per algorithm. We also included in our experiments Tseng’s Forward-Backward-Forward (FBF) method (1.10) (see [39, 52, 56]) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\frac{0.99}{L}$$\end{document} , and the Past Forward-Backward-Forward method (1.11) (see [39]) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =\frac{0.99}{2L}$$\end{document} . However, since their performance was quite similar to that of EG and OGDA, respectively, we decided not to include them in the plots. From the plots, it can be seen that, for the considered instance, Fast RFB achieves the best convergence performance in most cases among all evaluated methods.

We also analyze the asymptotic behaviour of the residual

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} & V \left( x_{k} , \lambda _{k} \right) = \begin{pmatrix} u_k + \nabla h(x_k) + A^*\lambda _k\\ b-Ax_k \end{pmatrix} = \begin{pmatrix} u_k + Hx_k -h + A^*\lambda _k\\ b-Ax_k \end{pmatrix}, \quad \\ & \text{ where } \ u_k \in \partial \Vert \cdot \Vert _1(x_k). \end{aligned}$$\end{document}

Figure 6 presents, on a semi-logarithmic scale, the convergence behaviour of the norm of the residual for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n = 200, 500, 800$$\end{document} and 1000 after \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$5\times 10^5$$\end{document} iterations per algorithm. Fast RFB achieves the best convergence performance in all four instances.Fig. 6A comparison of various methods in terms of the norm of the residual \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\Vert V \left( x_{k} , \lambda _{k} \right) \right\Vert $$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=200, 500, 800$$\end{document} and 1000Table 1Success rate of the methods in satisfying the stopping criterion \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\Vert V \left( x_{k} , \lambda _{k} \right) \right\Vert \leqslant 10^{-1}$$\end{document} , along with the corresponding runtimes and iteration countsMethodSuccessAvg. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\#$$\end{document} iter.Std. dev. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\#$$\end{document} iter.Avg. timeStd. dev. timeEG1.019501.2956.46415.1914.188OGDA1.042866.61970.93036.1336.013FRB1.042878.31969.76720.0953.252RFB1.052768.72408.20223.7184.128PEAG1.0215600.114.13090.93720.876ARG1.0365924.624.359158.85937.623Fast RFB: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =5$$\end{document} 1.032172.88.06614.7352.606Fast RFB: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =10$$\end{document} 1.021439.85.0079.6831.555Table 2Success rate of the methods in satisfying the stopping criterion \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\Vert V \left( x_{k} , \lambda _{k} \right) \right\Vert \leqslant 10^{-2}$$\end{document} , along with the corresponding runtimes and iteration countsMethodSuccessAvg. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\#$$\end{document} iter.Std. dev. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\#$$\end{document} iter.Avg. timeStd. dev. timeEG1.0434690.71034.704261.14913.192OGDA0.0NaNNaNNaNNaNFRB0.0NaNNaNNaNNaNRFB0.0NaNNaNNaNNaNPEAG0.0NaNNaNNaNNaNARG0.0NaNNaNNaNNaNFast RFB: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =5$$\end{document} 1.076644.47.38229.2664.250Fast RFB: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =10$$\end{document} 1.034052.06.97613.3931.418Table 3Success rate of the methods in satisfying the stopping criterion \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\Vert V \left( x_{k} , \lambda _{k} \right) \right\Vert \leqslant 10^{-3}$$\end{document} , along with the corresponding runtimes and iteration countsMethodSuccessAvg. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\#$$\end{document} iter.Std. dev. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\#$$\end{document} iter.Avg. timeStd. dev. timeEG1.0881605.31719.348579.38710.389OGDA0.0NaNNaNNaNNaNFRB0.0NaNNaNNaNNaNRFB0.0NaNNaNNaNNaNPEAG0.0NaNNaNNaNNaNARG0.0NaNNaNNaNNaNFast RFB: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =5$$\end{document} 1.0179003.76.99367.6371.640Fast RFB: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =10$$\end{document} 1.051009.83.49022.2420.296

Furthermore, we compare the performance of all algorithms with termination criterion \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\Vert V \left( x_{k} , \lambda _{k} \right) \right\Vert $$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\leqslant \varepsilon $$\end{document} under varying precision thresholds \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varepsilon \in \{10^{-1}, 10^{-2}, 10^{-3}\}$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=200$$\end{document} . Each algorithm is terminated after \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$10^6$$\end{document} iterations, even if the stopping criterion is not satisfied. We run the experiment with 10 different initializations and record the average number of iterations, the standard deviation of the number of iterations, the average runtime, and the standard deviation of the runtime (in seconds). The results are summarized in Tables 1,2,3.

For low-accuracy settings, such as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varepsilon = 10^{-1}$$\end{document} , EG requires the fewest iterations, whereas our method reaches the solution in less computational time. As the accuracy requirement increases, our method not only attains the stopping criterion faster than the others but also consistently requires less computational time. Moreover, our method appears to be more stable, in the sense that its performance does not vary significantly across different initializations, as indicated by the standard deviation of both the number of iterations and the runtime.