Weak, Strong and Linear Convergence of a Double-Layer Fixed Point Algorithm

TL;DR

This paper introduces a double-layer fixed point algorithm for convex feasibility problems in Hilbert spaces, demonstrating various convergence rates and accelerating convergence through a novel control strategy.

Contribution

It presents a new double-layer fixed point algorithm with convergence guarantees under different regularity conditions, unifying and extending existing projection methods.

Findings

Weak, strong, and linear convergence under various regularity assumptions

The algorithm covers many known projection methods in the literature

Numerical examples show accelerated convergence with the double-layer approach

Abstract

In this article we consider a consistent convex feasibility problem in a real Hilbert space defined by a finite family of sets $C_i$. We are interested, in particular, in the case where for each $i$, $C_i=Fix (U_i)=\{z\in \mathcal H\mid p_i(z)=0\}$, $U_i\colon\mathcal H\rightarrow \mathcal H$ is a cutter and $p_i\colon\mathcal H\rightarrow [0,\infty)$ is a proximity function. Moreover, we make the following assumption: the computation of $p_i$ is at most as difficult as the evaluation of $U_i$ and this is at most as difficult as projecting onto $C_i$. We study a double-layer fixed point algorithm which applies two types of controls in every iteration step. The first one -- the outer control -- is assumed to be almost cyclic. The second one -- the inner control -- determines the most important sets from those offered by the first one. The selection is made in terms of proximity functions. The convergence results presented in this manuscript depend on the conditions which first, bind together the sets, the operators and the proximity functions and second, connect the inner and outer controls. In particular, weak regularity (demi-closedness principle), bounded regularity and bounded linear regularity imply weak, strong and linear convergence of our algorithm, respectively. The framework presented in this paper covers many known (subgradient) projection algorithms already existing in the literature; for example, those applied with (almost) cyclic, remotest-set, maximum displacement, most-violated constraint and simultaneous controls. In addition, we provide several new examples, where the double-layer approach indeed accelerates the convergence speed as we demonstrate numerically.

Equations

$$C_i=\mathrm{Fix}{U}_i=p_i^{-1}(0).$$

$$x^{k+1}:=x^k+{\alpha }_k\left(\sum _{i\in I_k}{\omega }_i^k{U}_i{x}^k-x^k\right),$$

$${\color[rgb]{0,0,0}I=J_{k}\cup\ldots\cup J_{k+s-1}}$$

$$⟨x-y,z-y⟩\le 0$$

$$|d(x,C)-d(y,C)|\le \Vert x-y\Vert$$

$$P_{f}x:=\{\begin{array}{ll}x-\frac{f(x)}{\Vert g_f(x){\Vert }^2}{g}_f(x)& \text{ if f(x)>0,}\\ x& \text{ otherwise;}\end{array}$$

$$P_{f}x=x-\frac{f_{+}(x)}{\Vert g_f(x){\Vert }^2}{g}_f(x),$$

$$⟨x-P_{f}x,z-P_{f}x⟩\le 0.$$

$${\delta }_{K}d(x,S(g,0))\le g_{+}(x)$$

$$\Vert x^{k+1}-z\Vert \le \Vert x^k-z\Vert$$

$$\underset{k\to \infty }{\lim }d(x^k,C)=0;$$

$$\Vert x^k-x^{\infty }\Vert \le 2d(x^0,C)q^k;$$

$$\Vert x^k-x^{\infty }\Vert \le \frac{c}{(\sqrt[s]{q}{)}^{s-1}}{\left(\sqrt[s]{q}\right)}^k;$$

$$\Vert Ux-z\Vert \le \Vert x-z\Vert ;$$

$$\Vert Ux-z{\Vert }^2\le \Vert x-z{\Vert }^2-\rho \Vert Ux-x{\Vert }^2;$$

$$⟨z-Ux,x-Ux⟩\le 0.$$

$$\sqrt{\rho }\Vert Ux-x\Vert \le d(x,\mathrm{Fix}U),$$

$$\Vert Ux-z{\Vert }^2\le \Vert x-z{\Vert }^2-\sum _{i\in I}{\omega }_i{\rho }_i\Vert U_{i}x-x{\Vert }^2$$

$$\frac{1}{2R}\sum _{i\in I}{\omega }_i{\rho }_i\Vert U_{i}x-x{\Vert }^2\le \Vert Ux-x\Vert$$

$$\underset{k\to \infty }{\lim }\underset{i\in I}{\max }d(x^k,C_i)=0⟹\underset{k\to \infty }{\lim }d(x^k,C)=0;$$

$$d(x,C)\le {\kappa }_S\underset{i\in I}{\max }d(x,C_i).$$

$$c(C_1,...,C_m):=\sup \left\{\frac{1}{m-1}\frac{{\sum }_{i\ne j}⟨c_i,c_j⟩}{{\sum }_{i=1}^m\Vert c_i{\Vert }^2}\mid c_i\in C_i\cap C^{\perp }\text{ and }\sum _{i=1}^m\Vert c_i{\Vert }^2\ne 0\right\}.$$

$$\kappa (C_1,...,C_m):=\underset{x\notin C}{\sup }\frac{d(x,C)}{{\max }_{i\in I}d(x,C_i)}.$$

$$\frac{1}{\sqrt{(1-c)(m-1)}}\le \kappa \le \frac{2m}{(1-c)(m-1)}.$$

$$\left.\begin{array}[]{l}x^{k}\rightharpoonup x^{\infty}\\ Ux^{k}-x^{k}\rightarrow 0\end{array}\right\}\quad\Longrightarrow\quad x^{\infty}\in\operatorname*{Fix}U;$$

$$\underset{k\to \infty }{\lim }\Vert U{x}^k-x^k\Vert =0⟹\underset{k\to \infty }{\lim }d(x^k,\mathrm{Fix}U)=0;$$

$${\delta }_{S}d(x,\mathrm{Fix}U)\le \Vert Ux-x\Vert .$$

$$\Vert x-P_{f}x\Vert =\frac{f_{+}(x)}{\Vert g_f(x)\Vert }\ge \frac{{\delta }_K}{{\Delta }_K}d(x,S(f,0)).$$

$$C_i=\mathrm{Fix}{U}_i=p_i^{-1}(0)\subseteq \mathcal{H},$$

$$x^{0}\in\mathcal{H};\quad x^{k+1}:=x^{k}+\alpha_{k}\left(\sum_{i\in I_{k}}{\color[rgb]{0,0,0}\omega_{i}^{k}}U_{i}x^{k}-x^{k}\right)\text{ for }k=0,1,2,\ldots,$$

Full text

{AMS}

46N10, 46N40, 47H09, 47H10, 47N10, 47J25, 65F10, 65J99.

1 Introduction

In this paper we consider a consistent convex feasibility problem (CFP), which is to find $x\in C:=\bigcap_{i\in I}C_{i}\neq\emptyset$, where for each $i\in I:=\{1,\ldots,m\}$, $C_{i}$ is a closed and convex subset of a Hilbert space $\mathcal{H}$. We are interested, in particular, in the case where for every $i\in I$, there is a given cutter operator $U_{i}\colon\mathcal{H}\rightarrow\mathcal{H}$ and a proximity function $p_{i}\colon\mathcal{H}\rightarrow[0,\infty)$ such that

[TABLE]

We recall that an operator $U\colon\mathcal{H}\rightarrow\mathcal{H}$ with a nonempty fixed point set $\operatorname*{Fix}U:=\{z\in\mathcal{H}\mid Uz=z\}$ is said to be a cutter if the inequality $\langle x-Ux,z-Ux\rangle\leq 0$ holds for all $x\in\mathcal{H}$ and $z\in\operatorname*{Fix}U$. Because of the operators $U_{i}$, oftentimes this variant of CFP is referred to as the common fixed point problem (CFPP).

One possible approach to solving the CFP is to apply an iterative method involving metric projections onto each of the sets $C_{i}$, where at every step the chosen projections $P_{C_{i}}$ are utilized in a certain way. This brings us to the class of the so-called projection methods. However, in our case, the structure of the set $C_{i}$ provides additional information, which indicates that one can employ the cutters $U_{i}$ instead of $P_{C_{i}}$ whenever its computation is less difficult. This leads us to a more general class of the so-called fixed point algorithms.

To devise our fixed point algorithm in an effective way, we make the following two state-of-the-art assumptions: the computation of $p_{i}$ is at most as difficult as the evaluation of $U_{i}$ and this is at most as difficult as projecting onto $C_{i}$. These assumptions are satisfied if, for example, the set $C_{i}=\{z\in\mathcal{H}\mid f_{i}(z)\leq 0\}$ is a sublevel set of a convex functional $f_{i}$, the operator $U_{i}=P_{f_{i}}$ is a subgradient projection and the proximity $p_{i}=f^{+}_{i}$. When $p_{i}$ is unspecified then, by default, one can use the displacement $p_{i}(x)=\|U_{i}x-x\|$ and, finally, the distance $p_{i}(x)=d(x,C_{i})$.

Building on these assumptions, we propose a double-layer fixed point algorithm, where for an arbitrarily given starting point $x^{0}\in\mathcal{H}$, the sequence of consecutive approximations is defined by

[TABLE]

where $I_{k}\subseteq J_{k}\subseteq I$, for each $k=0,1,2,\ldots$. The scalar $\alpha_{k}\in(0,2)$ is called a relaxation parameter and $\omega_{i}^{k}\in(0,1]$ satisfy $\sum_{i\in I_{k}}\omega_{i}^{k}=1$. We call the subsets $I_{k}$ and $J_{k}$ an inner and outer control, respectively. The inner control $I_{k}$ determines which operators $U_{j}$ among all of $j\in J_{k}$ we want to use in order to improve the current approximation $x^{k}$. By default, these operators are considered to be the most significant at the step $k$, which we measure in terms of the proximity functions $p_{j}(\cdot)$. In particular, we may use $I_{k}:=\{\operatorname*{argmax}_{j\in J_{k}}p_{j}(x^{k})\}$ or even more generally, I_{k}:=\{t\text{ indices from J_{k} with the largest proximity }p_{j}(x^{k})\}. Following Cegielski [20, Section 5.8.4.1], one could also use $I_{k}:=\{i\in J_{k}\mid p_{i}(x^{k})\geq t\max_{j\in J_{k}}p_{j}(x^{k})\}$, where $t\in[0,1]$. In view of these examples one could expect that the most substantial outer control is $J_{k}=I$. Indeed, this case of the outer control was confirmed both analytically and numerically to be the most efficient one in terms of the convergence rate. Nonetheless, in the case of a large number of constraints $C_{i}$, the cost of making a decision with $J_{k}=I$ may turn out to be too high. Therefore we propose to restrict this process to a subset $J_{k}\subseteq I$, which is not necessarily all of $I$. We emphasize here that, rather surprisingly, using $J_{k}\subseteq I$, we may get similar results as for $J_{k}=I$; see Sections 3 and 4. In our paper we consider an $s$-intermittent outer control, that is, a control for which

[TABLE]

holds true for each $k=0,1,2,\ldots$ and some $s\in\mathbb{N}$. In particular, this extends the cyclic control $\{J_{[k]}\}_{k=0}^{\infty}$ with $[k]:=(k\text{ mod }s)+1$, while assuming that $J_{1}\cup\ldots\cup J_{s}=I$.

Although in (2) we consider a quite general framework, it is not difficult to see that it covers many projection ($U_{i}=P_{C_{i}}$), subgradient projection ($U_{i}=P_{f_{i}}$) and fixed point algorithms. For example, (2) includes various sequential methods of the form $x^{k+1}=U_{i_{k}}x^{k}$. In this case, the control $\{i_{k}\}_{k=0}^{\infty}$ can be either cyclic ($i_{k}=(k\text{ mod }m)+1$) or it can determine either the remotest set ($i_{k}=\operatorname*{argmax}_{i\in I}d(x^{k},C_{i})$) or the most violated constraint for sublevel sets ($i_{k}=\operatorname*{argmax}_{i\in I}f_{i}^{+}(x^{k})$). In addition, (2) includes various simultaneous methods ($x^{k+1}=\sum_{i\in J_{k}}\omega_{i}^{k}U_{i}x^{k}$), which for $J_{k}\neq I$ are oftentimes called block iterative methods. Notice that simultaneous methods cover the cyclic case by choosing a proper control $\{J_{k}\}_{k=0}^{\infty}$. In this paper we discuss how the concept of a double-layer control may indeed accelerate the convergence speed of some particular fixed point algorithms.

1.1 Contribution

The contribution of our paper can be summarized in three statements. All of them depend on the conditions which first, bind together the sets $C_{i}$, the operators $U_{i}$ and the proximity functions $p_{i}$, $i\in I$, and second, connect inner and outer controls.

• •

For the most general form of these conditions we establish the weak convergence of $\{x^{k}\}_{k=0}^{\infty}$ to some point $x^{\infty}\in C$, while assuming that the operators $U_{i}$ are weakly regular ($U_{i}-\operatorname*{Id}$ are demi-closed at 0), $i\in I$; see Theorem 3.1.

• •

This convergence turns out to be a strong one if, in addition, the operators $U_{i}$, $i\in I$, and the family $\mathcal{C}=\{C_{i}\mid i\in I\}$ are boundedly regular; see Theorem 3.1.

• •

Moreover, by restricting these conditions and by assuming that $\mathcal{C}$ is boundedly linearly regular, we establish a linear rate of convergence. In this case we also comment on the error bound; see Theorem 3.3. We recall that by a linear rate of convergence we mean that $\|x^{k}-x^{\infty}\|\leq cq^{k}$ for some $c\in(0,\infty)$ and $q\in(0,1)$, where $c$ and $q$ may depend on $x^{0}$.

In particular, all of the above-mentioned statements can be applied either with $U_{i}=P_{C_{i}}$ and $p_{i}(x)=d(x,C_{i})$; $U_{i}=P_{f_{i}}$ and $p_{i}(x)=f_{i}^{+}(x)$; see Sections 3.1 and 3.2, respectively. Moreover, we provide examples which easily fit the general framework.

1.2 Related work

We would like to begin with two important general observations regarding the types of convergence one could expect. The first is that in the infinite dimensional setting, in view of Hundal’s counterexample, it may happen that even for basic cyclic or parallel projection methods the convergence can only be in the weak topology; see [47] and [12]. Moreover, the result of Bauschke et al. [11, Theorem 1.4] shows that norm convergence can be far away from a linear rate. Furthermore, it can be arbitrarily slow. See also Badea et al. [6] in this connection. Thus both norm and linear convergence require some additional assumptions to which we refer in general as bounded regularity and bounded linear regularity.

The germinal norm convergence result of the alternating projection method designed for two closed subspaces in Hilbert space is due to von Neumann [58]. The cyclic projection method applied to solving linear systems goes back to the seminal work of Kaczmarz [48], while its parallel version is due to Cimmino [31]. The extension of von Neumann’s result to more than two closed subspaces was established by Halperin [45]. A weak convergence theorem for a cyclic projection method for general closed and convex sets in Hilbert space was established by Bregman [17]. The case of general closed and convex sets in $\mathbb{R}^{n}$ in the context of Cimmino’s method was considered by Auslender in [5]. Gurin et al. [44] formulated sufficient conditions for norm convergence of cyclic and remotest-set projection methods for closed and convex subsets of $\mathcal{H}$. These conditions are special cases of bounded regularity. Weak convergence of a simultaneous projection method in Hilbert space appeared in [52] by Pierra. Moreover, norm convergence was investigated under similar conditions to those which appeared in [44]. The block iterative ($J_{k}\subseteq I$) projection method for general closed and convex sets in $\mathbb{R}^{n}$ is due to Aharoni and Censor [2], while an extension can be found in [42] by Flåm and Zowe. Some practical realizations of block iterative projection methods in the case of linear systems can be found in [39, 46]. A finite dimensional subgradient projection method with the most-violated constraint control can be found in [40] by Eremin, but also in [30, Section 5.4.2] by Censor and Zenios. Cyclic and parallel subgradient projection methods in $\mathbb{R}^{n}$ were studied by Censor and Lent [29], and by Dos Santos [38]. Both weak and strong convergence results for more general variants of simultaneous and cyclic projection methods can be found, for example, in [7] by Bauschke and Borwein, and in [32, 33] by Combettes. Norm convergence follows there from bounded regularity of families of sets. A similar result, but for a fixed point algorithm can be found, for example, in [3] by Aleyner and Reich. It is worth mentioning that a very general weak convergence result for a simultaneous fixed point algorithm can be found in [25] by Cegielski and Censor.

We emphasize here that weak convergence of a special case of the double-layer projection method was established in [20, Theorem 5.8.25] by Cegielski, where $I_{k}:=\{i\in J_{k}\mid\omega_{i}^{k}d(x^{k},C_{i})\geq\delta\max_{j\in J_{k}}d(x^{k},C_{j})\}$. This is also a prototypical example of method (2).

For a more detailed overview of weak and strong convergence results we refer the reader to related monographs by Censor and Zenios [30], Byrne [19], Escalante and Raydan [41], Cegielski [20], and Popa [54]. A survey of the available literature can also be found in a recent paper by Cegielski and Censor [28].

We now concentrate on the case where the convergence becomes linear. As we have mentioned above, a linear rate may happen only under some additional assumptions. The first result of this type is due to Aronszajn [4, §12], who established a linear rate for the von Neumann’s alternating projection method assuming that the cosine of the angle between the two closed subspaces is less than one. Whereas Agmon [1, Theorem 3] showed this type of convergence for a projection method applied to a system of linear inequalities in $\mathbb{R}^{n}$ with the most-violated constraint control. Gurin et al. [44, Theorem 1] established the same type of convergence for a projection method combined either with cyclic or the remotest-set control. This method was applied to general closed and convex sets in $\mathcal{H}$, while assuming that their intersection has nonempty interior or, like Agmon, that every $C_{i}$ is a half-space. Nonempty interior guaranteed a linear rate of convergence in the result of Pierra [52, Theorem 1.1], who considered a simultaneous projection method. In this spirit, Eremin [40, Theorem, p. 142] and Polyak [53, Theorem 6] established a linear convergence rate for a subgradient projection method with the most-violated constraint control under the Slater condition ($f_{i}(x)<0$ for some $x$, $i\in I$). The Slater condition also appeared in the result of De Pierro and Iusem [34, Theorem 2] in connection with the cyclic subgradient projection. Bauschke and Borwein [7, Theorems 5.7 and 5.8] established the same rate of convergence for a general projection algorithm involving almost cyclic and the remotest-set controls. Their main assumptions were that the family of sets $\mathcal{C}=\{C_{i}\mid i\in I\}$ is boundedly linearly regular, which includes all of the above conditions, and that the algorithm is linearly focusing ($\delta d(x^{k},C_{i})\leq d(x^{k},C_{i}^{k})$ and $U_{i}$ is the metric projection onto $C_{i}^{k}\supseteq C_{i}$). A closely related result for both remotest-set and parallel ($I_{k}=J_{k}=I$) projection methods can be deduced from [15, Theorem 2.2] by Beck and Teboulle. A similar rate for a cyclic projection method, expressed in terms of bounded linear regularity, appeared in [37, Theorem 4.5] by Deutsch and Hundal. The same authors investigated convergence rates from different aspects; see [35, 36]. Recently, Bauschke et al. [13, Theorem 6.1] have established a linear rate for the simultaneous ($I_{k}=J_{k}\subseteq I$) fixed point algorithm with an almost cyclic control assuming that $\mathcal{C}=\{C_{i}\mid i\in I\}$ and $U_{i}$’s are boundedly linearly regular. Borwein, Li and Tam [16, Theorem 3.6] have also investigated the convergence rate of these algorithms, but in terms of Hölder bounded regularity of the operators $U_{i}$ and the family $\mathcal{C}$.

1.3 Organization of the paper

Our paper is organized as follows. In Section 2 we comment on Fejér monotone sequences, quasi-nonexpansive and regular operators, as well as on regular sets. In Section 3 we present two of the main results of this manuscript, namely Theorems 3.1 and 3.3. These theorems are revisited in the context of projection and subgradient projection methods in Subsections 3.1 and 3.2, whereas in Subsection 3.3 we explain how one can combine lopping and flagging with our algorithm. Finally, in Section 4 we provide numerical examples.

2 Preliminaries

Let $C\subseteq\mathcal{H}$ and $x\in\mathcal{H}$, where $\mathcal{H}$ is a Hilbert space. If there is a point $y\in C$ such that $\|y-x\|\leq\|z-x\|$ for all $z\in C$, then $y$ is called a metric projection of $x$ onto $C$ and is denoted by $P_{C}x$.

Let $C$ be nonempty, closed and convex. Then for any $x\in\mathcal{H}$, the metric projection $y:=P_{C}x$ is uniquely defined. Moreover, for every $y\in C$, we have $y=P_{C}x$ if and only if

[TABLE]

for all $z\in C$; see, for example, [20, Theorem 1.2.4]. In addition, the functional $d(\cdot,C):\mathcal{H}\rightarrow\mathbb{[}0,\infty)$, defined by $d(x,C):=\inf_{z\in C}\|x-z\|$, is 1-Lipschitz continuous, that is,

[TABLE]

for every $x,y\in\mathcal{H}$ and satisfies $d(x,C)=\|P_{C}x-x\|$.

Let $f:\mathcal{H}\rightarrow\mathbb{R}$ be a convex and continuous function with a nonempty sublevel set $S(f,0):=\{x\mid f(x)\leq 0\}$. Denote by $\partial f(x)$ its subdifferential, that is, $\partial f(x):=\{g\in\mathcal{H}\mid f(y)-f(x)\geq\langle g,y-x\rangle\text{, for all }y\in\mathcal{H}\}$. The continuity of $f$ implies that the set $\partial f(x)\neq\emptyset$ for all $x\in\mathcal{H}$ (see [10, Proposition 16.14]). For each $x\in\mathcal{H}$, let $g_{f}(x)\in\partial f(x)$ be a given subgradient. The so-called subgradient projection relative to $f$ is the operator $P_{f}:\mathcal{H}\rightarrow\mathcal{H}$ defined by

[TABLE]

see, for example, [14]. Notice that $P_{f}$ is well defined because $g_{f}(x)\neq 0$ if $f(x)>0$. To simplify notation, we sometimes write

[TABLE]

where $a_{+}:=\max\{0,a\}$.

By definition, $\operatorname*{Fix}P_{f}=S(f,0)$. Moreover, we have, $P_{f}x=P_{H}x$, where $H:=\{z\in\mathcal{H}\mid\langle g_{f}(x),z-x\rangle+f(x)\leq 0\}\supseteq S(f,0)$; see [14, Fact 2.3]. Consequently, for every $x\in\mathcal{H}$ and $z\in S(f,0)$, we have

[TABLE]

Finally, we recall a very useful inequality related to convex functions in $\mathbb{R}^{n}$.

Lemma 2.1**.**

Let $f_{i}\colon\mathbb{R}^{n}\rightarrow\mathbb{R}$ be convex, $i\in I$, $g(x):=\max_{i\in I}f_{i}(x)$ and assume that the Slater condition is satisfied, that is, $g(z)<0$ for some $z\in\mathbb{R}^{n}$. Then for every compact subset $K$, there is $\delta_{K}>0$ such that

[TABLE]

*holds for every $x\in K$. *

Proof 2.2**.**

*See, for example, [43, Lemma 3.3]. *

2.1 Fejér monotone sequences

Definition 2.3**.**

Let $C\subseteq\mathcal{H}$ be a nonempty, closed and convex set, and let $\{x^{k}\}_{k=0}^{\infty}$ be a sequence in $\mathcal{H}$. We say that $\{x^{k}\}_{k=0}^{\infty}$ is Fejér monotone with respect to $C$ if

[TABLE]

*for all $z\in C$ and every integer $k\geq 0$. *

Below we present several key properties of Fejér monotone sequences, which we apply in our convergence analysis in order to establish weak, strong and linear convergence.

Theorem 2.4**.**

Let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be Fejér monotone with respect to $C$. Then

1. (i)

$\{x^{k}\}_{k=0}^{\infty}$* converges weakly to some point $x^{\infty}\in C$ if and only if all its weak cluster points lie in $C$;* 2. (ii)

$\{x^{k}\}_{k=0}^{\infty}$* converges strongly to some point $x^{\infty}\in C$ if and only if*

[TABLE] 3. (iii)

if there is some constant $q\in(0,1)$ such that $d(x^{k+1},C)\leq qd(x^{k},C)$ holds for every $k=0,1,2,\ldots$, then $\{x^{k}\}_{k=0}^{\infty}$ converges linearly to some point $x^{\infty}\in C$ and

[TABLE] 4. (iv)

if $\{x^{ks}\}_{k=0}^{\infty}$ converges linearly to some point $x^{\infty}\in C$, that is, $\|x^{ks}-x^{\infty}\|\leq cq^{k}$ for some constants $c>0$, $q\in(0,1)$ and integer $s$, then the entire sequence $\{x^{k}\}_{k=0}^{\infty}$ converges linearly and moreover,

[TABLE] 5. (v)

if $\{x^{k}\}_{k=0}^{\infty}$ converges strongly to some point $x^{\infty}\in C$, then $\|x^{k}-x^{\infty}\|\leq 2d(x^{k},C)$ for every $k=0,1,2,\ldots$.

Proof 2.5**.**

*See, for example, [7, Theorem 2.16 and Proposition 1.6]. *

2.2 Quasi-nonexpansive operators

Definition 2.6**.**

Let $U:\mathcal{H}\rightarrow\mathcal{H}$ be an operator with a fixed point, that is, $\operatorname*{Fix}U=\{z\in\mathcal{H}\mid z=Uz\}\neq\emptyset$. We say that $U$ is

1. (i)

quasi-nonexpansive (QNE) if for all $x\in\mathcal{H}$ and all $z\in\operatorname*{Fix}U$,

[TABLE] 2. (ii)

$\rho$-strongly quasi-nonexpansive ($\rho$-SQNE), where $\rho\geq 0$, if for all $x\in\mathcal{H}$ and all $z\in\operatorname*{Fix}U$,

[TABLE] 3. (iii)

a cutter if for all $x\in\mathcal{H}$ and all $z\in\operatorname*{Fix}U$,

[TABLE]

A comprehensive review of the properties of QNE, SQNE and cutter operators can be found in [20, Chapter 2].

Example 2.7**.**

*Both the metric projection $P_{C}$ onto a nonempty, closed and convex set $C\subseteq\mathcal{H}$, and the subgradient projection $P_{f}$ associated with a continuous and convex function $f\colon\mathcal{H}\rightarrow\mathbb{R}$ with a nonempty sublevel set are cutters. *

For a given $U\colon\mathcal{H}\rightarrow\mathcal{H}$ and $\alpha\in(0,\infty)$, the operator $U_{\alpha}:=\operatorname*{Id}+\alpha(U-\operatorname*{Id})$ is called an $\alpha$-relaxation of $U$, where by $\operatorname*{Id}$ we denote the identity operator. We call $\alpha$ a relaxation parameter. It is easy to see that for every $\alpha\neq 0$, $\operatorname*{Fix}U=\operatorname*{Fix}U_{\alpha}$. Usually, in connection with iterative methods, as in (2), the relaxation parameter $\alpha$ is assumed to belong to the interval $(0,2]$.

Theorem 2.8**.**

*Let $U\colon\mathcal{H}\rightarrow\mathcal{H}$ be an operator with a fixed point and let $\alpha\in(0,2]$. Then $U$ is a cutter if and only if its relaxation $\operatorname*{Id}+\alpha(U-\operatorname*{Id})$ is $(2-\alpha)/\alpha$-strongly quasi-nonexpansive. In particular, $U$ is a cutter if and only if $U$ is $1$-strongly quasi-nonexpansive. Furthermore, $U$ is quasi-nonexpansive if and only if $\frac{1}{2}(U+\operatorname*{Id})$ is a cutter. *

Proof 2.9**.**

See, for example, [20, Theorem 2.1.39].

Remark 2.10**.**

The fixed point set of a QNE operator is closed and convex (see [9, Proposition 2.6(ii)]). Note that by definition and by substituting $z=P_{\operatorname*{Fix}U}x$, for any $\rho$-SQNE operator $U$ ($\rho>0$), we have

[TABLE]

*an inequality which for cutters holds with $\rho=1$. *

Theorem 2.11**.**

Let $U_{i}:\mathcal{H}\rightarrow\mathcal{H}$ be $\rho_{i}$-strongly quasi-nonexpansive, $\rho_{i}>0$, $i\in I$, with $\bigcap_{i\in I}\operatorname*{Fix}U_{i}\neq\emptyset$ and let $U:=\sum_{i\in I}\omega_{i}U_{i}$, where $\omega_{i}>0$, $i\in I$ and $\sum_{i\in I}\omega_{i}=1$. Then $U$ is $\rho$-strongly quasi-nonexpansive, where $\rho:=\min_{i\in I}\rho_{i}>0$ and where $\operatorname*{Fix}U=\bigcap_{i\in I}\operatorname*{Fix}U_{i}$. Moreover,

[TABLE]

for every $x\in\mathcal{H}$ and $z\in\bigcap_{i\in I}\operatorname*{Fix}U_{i}$. Consequently,

[TABLE]

*for any positive $R\geq\|x-z\|$. *

Proof 2.12**.**

*See, for example, [20, Theorem 2.1.50] for the first part and [27, Proposition 4.5] for inequalities (18) and (19). *

2.3 Regular sets

Definition 2.13**.**

Let $C_{i}\subseteq\mathcal{H}$, $i\in I$, be closed and convex with $C:=\bigcap_{i\in I}C_{i}\neq\emptyset$ and let $\mathcal{C}=\{C_{i}\mid i\in I\}$. Let $S\subseteq\mathcal{H}$ be nonempty. We say that $\mathcal{C}$ is

1. (i)

regular over $S$ if for every sequence $\{x^{k}\}_{k=0}^{\infty}\subseteq S$,

[TABLE] 2. (ii)

linearly regular over $S$ if there is $\kappa_{S}>0$ such that for every $x\in S$,

[TABLE]

*If any of the above regularity conditions holds for every subset $S\subseteq\mathcal{H}$, then we simply omit the phrase “over $S$”. If the same holds, but restricted to bounded subsets $S\subseteq\mathcal{H}$, then we precede the term with the adverb boundedly. *

Below we present some sufficient conditions for regularities of sets. Many more conditions can be found, for example, in [13, Fact 5.8].

Theorem 2.14**.**

Let $C_{i}\subseteq\mathcal{H}$, $i\in I$, be closed and convex with $C:=\bigcap_{i\in I}C_{i}\neq\emptyset$ and let $\mathcal{C}=\{C_{i}\mid i\in I\}$. Then the following statements hold:

1. (i)

if $\dim\mathcal{H}<\infty$, then $\mathcal{C}$ is boundedly regular; 2. (ii)

if $C_{j}\cap\operatorname*{int}\bigcap_{i\in I\setminus\{j\}}C_{i}\neq\emptyset$, then $\mathcal{C}$ is boundedly linearly regular; 3. (iii)

if each $C_{i}$ is a half-space, then $\mathcal{C}$ is linearly regular; 4. (iv)

if $\dim\mathcal{H}<\infty$, $C_{j}$ is a half-space, $j\in J\subseteq I$, and $\bigcap_{j\in J}C_{j}\cap\bigcap_{i\in I\setminus J}\operatorname*{ri}C_{i}\neq\emptyset$, then $\mathcal{C}$ is boundedly linearly regular.

Remark 2.15**.**

Following [6] by Badea, Grivaux and Müller, we recall the notion of the extended cosine of the Friedrichs angle corresponding to a finite family of closed subspaces $C_{i}\subseteq\mathcal{H}$, $i\in I$, for which $C:=\bigcap_{i\in I}C_{i}\neq\emptyset$, that is,

[TABLE]

In order to relate this parameter to the linear regularity constant, we focus our attention on the smallest possible $\kappa_{\mathcal{H}}$ which appears in (21), that is,

[TABLE]

The inverse of this constant is oftentimes called the inclination of the subspaces; see, for example, [55, 56] by Pustylnik, Reich and Zaslavski. By [6, Proposition 3.9], we have

[TABLE]

*In particular, $\kappa<\infty$ if and only if $c<1$. *

2.4 Regular operators

We extend the notion of (boundedly) (linearly) regular families of sets to (boundedly) (linearly) regular operators. We adopt the naming convention in [13]. To make our nomenclature consistent, we also introduce the term “weakly regular”.

Definition 2.16**.**

Let $U:\mathcal{H}\rightarrow\mathcal{H}$ be an operator with a fixed point, that is, $\operatorname*{Fix}U\neq\emptyset$ and let $S\subseteq\mathcal{H}$ be nonempty. We say that the operator $U$ is

1. (i)

weakly regular over $S$ if for any sequence $\{x^{k}\}_{k=0}^{\infty}\subseteq S$ and $x^{\infty}\in\mathcal{H}$,

[TABLE] 2. (ii)

regular over $S$ if for any sequence $\{x^{k}\}_{k=0}^{\infty}\subseteq S$,

[TABLE] 3. (iii)

linearly regular over $S$ if there is $\delta_{S}>0$ such that for every $x\in S$,

[TABLE]

If any of the above regularity conditions holds for every subset $S\subseteq\mathcal{H}$, then we simply omit the phrase “over $S$”. If the same condition holds when restricted to bounded subsets $S\subseteq\mathcal{H}$, then we precede the term with the adverb boundedly. Since there is no need to distinguish between boundedly weakly and weakly regular operators, we call both weakly regular.

Remark 2.17**.**

*The above definition, points (ii) and (iii), indeed extends Definition 2.13. To see this, let $C_{i}\subseteq\mathcal{H}$, $i\in I$, be closed and convex with $C:=\bigcap_{i\in I}C_{i}\neq\emptyset$ and let $\mathcal{C}=\{C_{i}\mid i\in I\}$. Moreover, let $U:=P_{C_{i(x)}}x$, where $i(x):=\min\{i\in I\mid d(x,C_{i})=\max_{j\in I}d(x,C_{j})\}.$ It is not difficult to see that $\operatorname*{Fix}U=C$ and $\|Ux-x\|=\max_{i\in I}d(x,C_{i})$. Therefore the family $\mathcal{C}$ is (boundedly) (linearly) regular if and only if the operator $U$ is (boundedly) (linearly) regular. *

Note that for $S=\mathcal{H}$, saying that $U$ is weakly regular is nothing else than saying that $U-\operatorname*{Id}$ is demi-closed at 0. The definition of demi-closed operators goes back to papers by Browder and Petryshyn [18], and by Opial [49]. Some authors refer to condition (25) as a demi-closedness principle; see [21, Definition 4.2] and [22, page 388]. The concept of a demi-closed operator has been extended in many directions. For example Bauschke, Chen and Wang [8] have recently considered a fixed point closed mapping (defined in [8, Lemma 2.1]), where the weak convergence to $x^{\infty}$ was replaced by a strong one. On the other hand, Cegielski [21, Definition 4.4] has considered a demi-closedness condition referring to a family of operators instead of to a single one; in this direction see also [57, Lemma 3.4].

A prototypical version of condition (26) with a continuous $U$ can be found in [51, Theorem 1.2] by Petryshyn and Williamson. To the best of our knowledge, boundedly regular operators in this form first appeared in [26] by Cegielski and Zalas, who applied them to variational inequalities under the name approximately shrinking. Independently, Bauschke, Noll and Phan [13] investigated unrestricted iterations of these operators in connection with common fixed point problems under the name boundedly regular. Many properties of these operators (under the name approximately shrinking) were presented in [27] with some extensions in [59], [57] and [23], and further applications in [21] and [24].

The name linearly boundedly regular operators was proposed by Bauschke, Noll and Phan, who applied them to establish a linear rate of convergence for a block iterative fixed point algorithm [13, Theorem 6.1]. To the best of our knowledge, the concept of this type of operator goes back to Outlaw [50, Theorem 2], and Petryshyn and Williamson [51, Corollary 2.2]. A closely related condition called a linearly focusing algorithm was studied by Bauschke and Borwein [7, Definition 4.8]. The concept of a focusing algorithm goes back to Flåm and Zowe [42, Section 2], and can also be found in [33, Definition 1.2] by Combettes. Linearly regular operators appeared in [27, Definition 3.3] by Cegielski and Zalas as linearly shrinking operators.

Proposition 2.18**.**

Let $U:\mathcal{H}\rightarrow\mathcal{H}$ be such that $\operatorname*{Fix}U$ is nonempty. Then the operator

1. (i)

$U$* is boundedly regular whenever it is linearly boundedly regular.*

Moreover, if $\operatorname*{Fix}U$ is closed and convex, for example, when $U$ is quasi-nonexpansive, then

1. (ii)

$U$* is weakly regular whenever $U$ is boundedly regular;* 2. (iii)

$U$* is boundedly regular whenever $U$ is weakly regular and $\dim\mathcal{H}<\infty$.*

Proof 2.19**.**

(i) follows directly from Definition 2.16. Parts (ii) and (iii) follow from the arguments used to prove [27, Proposition 4.1].

It was shown by Opial [49, Lemma 2] that a nonexpansive (1-Lipschitz) mapping satisfies condition (25) with $S=\mathcal{H}$. Therefore for $\mathcal{H}=\mathbb{R}^{n}$ a nonexpansive mapping is boundedly regular whenever it has a fixed point. It is worth mentioning that in a general Hilbert space the weak regularity of $U$ is only a necessary condition for implication (26) and even a firmly nonexpansive mapping may not have this property; see either [59, Example 2.9] or [57, Examples 2.17 and 2.18].

We conclude this section with the following example.

Example 2.20**.**

Let $f\colon\mathcal{H}\rightarrow\mathbb{R}$ be convex and continuous.

1. (a)

If $f$ is Lipschitz continuous on bounded sets, then $P_{f}$ is weakly regular; see, for example **[20, Theorem 4.2.7]**. We recall that $f$ is Lipschitz continuous on bounded sets $\Longleftrightarrow$ $f$ maps bounded sets onto bounded sets $\Longleftrightarrow$ $\partial f$ is uniformly bounded on bounded sets; see, for example, **[7, Proposition 7.8]**. All three conditions hold true if $\mathcal{H}=\mathbb{R}^{n}$. 2. (b)

If $\mathcal{H}=\mathbb{R}^{n}$, then by (a) and Proposition 2.18, $P_{f}$ is boundedly regular. See also **[26, Lemma 24]**. 3. (c)

If $f(z)<0$ for some $z\in\mathbb{R}^{n}$, then $P_{f}$ is boundedly linearly regular. Indeed, by (a) and by Lemma 2.1, for every compact $K\subseteq\mathbb{R}^{n}$, there are $\delta_{K},\Delta_{K}>0$ such that $\|\partial f(x)\|\leq\Delta_{K}$ and $\delta_{K}d(x,S(f,0))\leq f_{+}(x)$ for every $x\in K$. Therefore we can write

[TABLE]

3 Weak, strong and linear convergence results

Unless otherwise stated, we assume from now on that $C:=\bigcap_{i\in I}C_{i}\neq\emptyset$ and that for every $i\in I$, we have

[TABLE]

where $U_{i}\colon\mathcal{H}\rightarrow\mathcal{H}$ is a cutter and $p_{i}\colon\mathcal{H}\rightarrow[0,\infty)$, $i\in I$.

Theorem 3.1** (Weak and strong convergence).**

Let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be defined by the double-layer fixed point algorithm, that is,

[TABLE]

where $I_{k}\subseteq J_{k}\subseteq I$, $\alpha_{k}\in(0,2)$ and $\omega_{i}^{k}\in(0,1]$ are such that $\sum_{i\in I_{k}}\omega_{i}^{k}=1$. Assume that

1. (i)

$\alpha_{k}\in[\alpha^{-},\alpha^{+}]$* and $\omega^{i}_{k}\in[\omega^{-},1]$, where $\alpha^{-},\omega^{-}\in(0,1]$ and $\alpha^{+}\in[1,2)$;* 2. (ii)

$I=J_{k}\cup\ldots\cup J_{k+s-1}$* *for every $k=0,1,2,\ldots$ and some integer $s\geq 1$; 3. (iii)

the inner and outer controls satisfy

[TABLE] 4. (iv)

for every bounded sequence $\{y^{k}\}_{k=0}^{\infty}\subseteq\mathcal{H}$ and $i\in I$,

[TABLE]

*If for each $i\in I$, the operator $U_{i}$ is weakly regular, then the sequence $\{x^{k}\}_{k=0}^{\infty}$ converges weakly to some point $x^{\infty}\in C$. Moreover, if for each $i\in I$, the operator $U_{i}$ is boundedly regular and the family $\mathcal{C}:=\{\operatorname*{Fix}U_{i}\mid i\in I\}$ is boundedly regular, then the convergence to $x^{\infty}$ is in norm. *

Proof 3.2**.**

To show both weak and strong convergence, we apply Theorem 2.4 (i) and (ii), respectively.

We first show that $\{x^{k}\}_{k=0}^{\infty}$ is Fejér monotone with respect to $C$. Indeed, for every $k=0,1,2,\ldots,$ let

[TABLE]

We can write $x^{k+1}=T_{k}x^{k}$. It is not difficult to see that by Theorems 2.8 and 2.11, $T_{k}$ is $\rho_{k}$-SQNE and $C\subseteq\operatorname*{Fix}T_{k}=\bigcap_{i\in I_{k}}\operatorname*{Fix}U_{i}$. Moreover, we have

[TABLE]

In addition, for every $z\in C$, we get

[TABLE]

which immediately implies that $\{x^{k}\}_{k=0}^{\infty}$ is Fejér monotone with respect to $C$.

Next we show that

[TABLE]

Note that since $\{x^{k}\}_{k=0}^{\infty}$ is Fejér monotone, it is bounded. Moreover, since the sequence $\{\|x^{k}-z\|\}_{k=0}^{\infty}$ is decreasing and bounded, it is convergent as well and, by (34)–(35),

[TABLE]

which is equivalent to

[TABLE]

Let $z\in C$ and let $R>0$ be such that $\|x^{k}-z\|\leq R$. Using Theorem 2.11, we get

[TABLE]

which together with (37) yields

[TABLE]

Therefore, by (31) and (32), we see that (36) holds true, as asserted.

Case 1. Now assume that the operators $U_{i}$ are weakly regular, $i\in I$. Let $x$ be an arbitrary weak cluster point of $\{x^{k}\}_{k=0}^{\infty}$ and let $x^{n_{k}}\rightharpoonup x$. Let $i\in I$ be fixed and for every $k=0,1,2,\ldots$, let $r_{k}\in\{0,1,2,\ldots,s-1\}$ be the smallest number such that $i\in J_{n_{k}+r_{k}}$. By (38), $x^{n_{k}+r_{k}}\rightharpoonup x$ and by (36),

[TABLE]

Hence the weak regularity of $U_{i}$ yields that $x\in\operatorname*{Fix}U_{i}$ and the arbitrariness of $i$ implies that $x\in C$. Thus we have shown that each weak cluster point of $\{x^{k}\}_{k=0}^{\infty}$ lies in $C$, which, by Theorem 2.4 (i), means that $\{x^{k}\}_{k=0}^{\infty}$ converges weakly to some point in $C$.

Case 2. Next we assume that the operators $U_{i}$ are boundedly regular, $i\in I$, and the family $\mathcal{C}$ is boundedly regular. By (36),

[TABLE]

By Theorem 2.4 (ii) and by the assumed bounded regularity of $\mathcal{C}$, it suffices to show that

[TABLE]

Indeed, let $i\in I$ and let $r_{k}\in\{0,1,2,\ldots,s-1\}$ be the smallest integer such that $i\in J_{k+r_{k}}$. By (42),

[TABLE]

Moreover, by the triangle inequality and by the definition of the metric projection, we get

[TABLE]

which, when combined with (38) and (44), yields

[TABLE]

*The arbitrariness of $i\in I$ implies (43) which, in view of Theorem 2.4 (ii), completes the proof. *

Theorem 3.3** (Linear convergence).**

Let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be defined as in Theorem 3.1 under conditions (i) and (ii). Moreover, assume that

1. (iii)

the inner and outer controls satisfy for each $k=0,1,2,\ldots$,

[TABLE] 2. (iv)

for $r=d(x^{0},C)>0$, there are numbers $\delta_{r},\Delta_{r}>0$ such that for every $x\in B(P_{C}x^{0},r)$ and $i\in I$,

[TABLE]

If the family $\mathcal{C}:=\{\operatorname*{Fix}U_{i}\mid i\in I\}$ is $\kappa_{r}$-boundedly linearly regular over $B(P_{C}x^{0},r)$, then the sequence $\{x^{k}\}_{k=0}^{\infty}$ converges linearly to some point $x^{\infty}\in C$, that is, $\|x^{k}-x^{\infty}\|\leq c_{r}q_{r}^{k}$, where

[TABLE]

(Error bound)* Moreover, we have the following estimate:*

[TABLE]

Proof 3.4**.**

For every $k=0,1,2,\ldots$, let $T_{k}$ be defined as in the proof of Theorem 3.1, that is,

[TABLE]

As before, we can write that $x^{k+1}=T_{k}x^{k}$ knowing, in addition, that $T_{k}$ is $\rho_{k}$-SQNE with $\operatorname*{Fix}T_{k}=\bigcap_{i\in I_{k}}\operatorname*{Fix}U_{i}$. Moreover, we can estimate

[TABLE]

Furthermore, we see that $\operatorname*{Id}+\alpha_{k}(U_{i}-\operatorname*{Id})$ is $\rho_{k}$-SQNE as well. We divide the rest of the proof into three steps.

Step 1. First, we show that for every $\nu\in I$, $z\in C$, and $k=0,1,2,\ldots,$ we have

[TABLE]

Indeed, let $\nu\in I$, $z\in C$ and for every $k=0,1,2,\dots,$ let $m_{k}\in\{ks,\ldots,(k+1)s-1\}$ be the smallest integer such that $\nu\in J_{m_{k}}$. Define

[TABLE]

which, by (47), satisfies

[TABLE]

Since the function $d(\cdot,C_{\nu})$ is nonexpansive, we have

[TABLE]

and therefore

[TABLE]

where the last step follows from the triangle inequality. For $r=d(x^{0},C)$, let $\delta_{r}$ and $\Delta_{r}$ be as in (48). Note that since $\{x^{k}\}_{k=0}^{\infty}$ is Fejér monotone, $\{x^{k}\}_{k=0}^{\infty}\subseteq B(P_{C}x^{0},r)$ and consequently, for every $i\in I$ and $k=0,1,2,\ldots,$

[TABLE]

By the definition of $i_{k}$ (55) and by (58), we get

[TABLE]

Now, by combining (57), the Cauchy-Schwarz inequality for real numbers and (59),

[TABLE]

where the last inequality follows from the fact that $\Delta_{r}\geq\delta_{r}$. Indeed, since $U_{i}$ is a cutter, using (48), we get for every $x\in B(P_{C}x^{0},r)$ that $\|U_{i}x-x\|\leq d(x,C_{i})\leq\Delta_{r}/\delta_{r}\|U_{i}x-x\|$.

We now estimate the first term in (3.4). By the $\rho_{k}$-strong quasi-nonexpansivity of $T_{k}$ and (52), we get for all $n=0,1,2,\ldots,$

[TABLE]

Substituting (61) into the first term produces the following estimates:

[TABLE]

Next, we estimate the second summand in (3.4). Recall that, by definition, $x^{m_{k}+1}=T_{m_{k}}x^{m_{k}}$. Therefore, by Theorem 2.11 applied to $x=x^{m_{k}}$ and $U=T_{k}$, and by the Fejér monotonicity of $\{x^{k}\}_{k=0}^{\infty}$, we have

[TABLE]

Combining (3.4) and (3.4) with (3.4), we derive (53), as asserted.

Step 2. Now we show that the estimate

[TABLE]

holds true. Indeed, by setting $z=P_{C}x^{ks}$ in (53) and by noting that

[TABLE]

we get

[TABLE]

Since (66) was proved for arbitrary $\nu\in I$, when we combine this with the bounded linear regularity of the family $\mathcal{C}$, we get

[TABLE]

which when rearranged, leads to

[TABLE]

By applying Theorem 2.4 (iii), we see that the subsequence $\{x^{ks}\}_{k=0}^{\infty}$ converges linearly and

[TABLE]

From Theorem 2.4 (iv), it follows that the entire sequence $\{x^{k}\}_{k=0}^{\infty}$ converges linearly and, in addition, (64) holds true.

Step 3. Next we show that estimate (50) holds true. Indeed, by combining Theorem 2.4 (v) with the bounded linear regularity of the family $\mathcal{C}$ and (48), one observes that

[TABLE]

*which, when combined with (49), completes the proof. *

Remark 3.5**.**

We show that condition (30) is more general than (46). For simplicity, assume that $U_{i}=P_{C_{i}}$, $p_{i}=d(\cdot,C_{i})$, $J_{k}=I$, $I_{2k}=I$ and $I_{2k+1}$ is arbitrary for every $k=0,1,2,\ldots$. In particular, we can always choose the inner block such that $I_{2k+1}\cap\operatorname*{Argmax}_{j\in I}p_{j}(x^{2k+1})=\emptyset$. Clearly, condition (46) does not hold. Now we claim that (30) does hold in this case. Indeed, assume that $\max_{i\in I_{k}}d(x^{k},C_{i})\rightarrow_{k}0$. We have to show that $\max_{i\in I}d(x^{k},C_{i})\rightarrow_{k}0$, which by the definition of $I_{k}$ and (35) is already true for even $k$’s. We have

[TABLE]

*and the right-hand side of the above inequality converges to zero since $\|x^{k+1}-x^{k}\|\rightarrow 0$ by (37). Note that (35) and (37) do not depend on (30). *

Assume that $p_{i}(x)=\|U_{i}x-x\|$ for every $i\in I$. Clearly, in this case condition (31) is satisfied and condition (30) takes the following form:

[TABLE]

Consequently we can simplify the statement of Theorem 16 to achieve weak and strong convergence while using weakly and boundedly regular operators. Moreover, since we can take $\Delta_{r}=1$ in (47), we can replace this condition by the following inequality: $\delta_{r}d(x,\operatorname*{Fix}U_{i})\leq\|U_{i}x^{k}-x^{k}\|$. Note that the existence of $\delta_{r}$ can be guaranteed by assuming that every $U_{i}$ is boundedly linearly regular and therefore we can obtain a special case of Theorem 17. Other particular variants of Theorems 16 and 17 can be obtained by considering projection and subgradient projection methods which we discuss in more detail in Sections 3.1 and 3.2.

Note that, in general, the parameters $\delta_{r},\Delta_{r}$ and $\kappa_{r}$ are not known explicitly. Nevertheless, formula (48) still enables us to compare $q_{r}$’s related to different methods. We find this observation important since $q_{r}$ characterizes convergence speed in terms of iterations. In particular, by using (48) one can deduce which operations may reduce $q_{r}$ and as a consequence, which methods should be faster. Moreover, one can deduce a natural stopping rule from (49) in terms of maximum proximity. Note that the maximum proximity which appears in (49) is entirely computable in contrast with $c_{r}$ and $q_{r}$. In the following example we show how one can compute $q_{r}$.

Example 3.6**.**

For simplicity, let $I=J_{1}\cup\ldots\cup J_{s}$ and let the outer control $\{J_{[k]}\}_{k=0}^{\infty}$ change cyclically with the constant size of the outer block, that is, $[k]:=(k\text{ mod }s)+1$ and $|J_{[k]}|=b$ for every $k=0,1,2,\ldots$ and some integer $b$. Moreover, assume that $m=s\cdot b$. Also, let $\alpha_{k}=1$ and $\omega_{i}^{k}=1/|I_{k}|$ for every $k=0,1,2,\ldots$. We now present several fixed point algorithms (FPA) which satisfy the inner-outer control relation defined in (47) and thus also in (31). In addition, by assuming the existence of constants $\kappa_{r},\delta_{r}$ and $\Delta_{r}$, and by Theorem 3.3, we may deduce the upper bound for the coefficient $q_{r}$.

1. (a)

Cyclic FPA: $x^{k+1}:=U_{[k]}x^{k},$ where $s=m$ and

[TABLE] 2. (b)

Simultaneous FPA: $x^{k+1}:=\frac{1}{b}\sum_{i\in J_{[k]}}U_{i}x^{k}$, where

[TABLE] 3. (c)

Simultaneous FPA with active sets: $x^{k+1}:=\frac{1}{|I_{k}|}\sum_{i\in I_{k}}U_{i}x^{k},$ where $I_{k}:=\{i\in J_{[k]}\mid p_{i}(x^{k})>0\}$. Since we do not control the number of violated constraints, the upper bound for $q_{r}$ is the same as in (75). 4. (d)

Maximum proximity FPA: $x^{k+1}:=U_{i_{k}}x^{k}$, where $i_{k}=\operatorname*{argmax}_{j\in J_{[k]}}p_{j}(x^{k})$ and

[TABLE] 5. (e)

Simultaneous FPA with the inner block of a fixed size: $x^{k+1}:=\frac{1}{t}\sum_{i\in I_{k}}U_{i}x^{k},$ where $I_{k}:=\{t$ smallest indices from $J_{[k]}$ with the largest proximity $p_{j}(x^{k})\}$. We have

[TABLE] 6. (f)

Simultaneous FPA with the inner block determined by the threshold: $x^{k+1}:=\frac{1}{|I_{k}|}\sum_{i\in I_{k}}U_{i}x^{k},$ where $I_{k}:=\{i\in J_{[k]}\mid p_{i}(x^{k})\geq t\max_{j\in J_{[k]}}p_{j}(x^{k})\}$ and $t\in[0,1]$. Again, since we do not control the number of constraints for which the proximity is above the threshold, we can only estimate the upper bound for $q_{r}$ by using (77) and taking $t$ there to equal $b$. This leads to the same formula as in (75).

Notice that for all of the above-mentioned methods one could set $s=1$. Thus $b=m$ and $J_{[k]}=I$. In particular, for the maximum proximity FPA we get

[TABLE]

As a matter of fact, this is the best estimate that can be derived in our setting by using formula (49). Nevertheless, the case where $J_{[k]}\subseteq I$ seems to be more applicable. We would like to emphasize that in cases (d)–(f), the $q_{r}$ is a decreasing function of $b$. Therefore, increasing the block size $b$ should speed up the convergence in terms of the iteration $k$, but at the cost of computational time. Moreover, the $q_{r}$ from case (e) is an increasing function of $t$, which implies that by considering more information from the outer block we may, in fact, reduce the convergence speed. This, when combined with (76), suggests that the maximum proximity FPA should perform in the best way out of all variants described in (a)–(f). Another observation is that by manipulating the parameter $t$ with fixed $b$, in cases (e) and (f), we may approach either the simultaneous or maximum proximity FPA from cases (b) and (d). Finally, we conjecture at this point that the maximum proximity FPA has the same convergence properties with $b<m$ as it has in the case of $b=m$ in terms of iterations. All of this we verify numerically in the last section of our paper.

Remark 3.7**.**

By definition, our double-layer fixed point algorithm requires cutters as the input operators. Nevertheless, this does not limit our generality. For example, following Bauschke, Noll and Phan [13, Theorem 6.1], we assume we want to use averaged operators $V_{i}$ in the framework of the one-layer ($I_{k}=J_{k}$) simultaneous fixed point algorithm. That is, we let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be defined in the following way:

[TABLE]

where $I_{k}\subseteq I$ and $\lambda_{i}^{k}\in(0,1)$ are such that $\sum_{i\in I_{k}}\lambda_{i}^{k}=1$. Notice that since for every $i\in I$, the operator $V_{i}$ is averaged, it can be written as $V_{i}=\operatorname*{Id}+\eta_{i}(W_{i}-\operatorname*{Id})$, where $\eta_{i}\in(0,1)$ and $W_{i}$ is nonexpansive. Thus, by substituting

[TABLE]

into (30), we recover (79), where every $U_{i}$ is a cutter, $\alpha_{k}\in(0,2)$ and $\sum_{i\in I}\omega_{i}^{k}=1$. This also corresponds to [16, Theorem 3.6].

Remark 3.8**.**

*Let $\{x^{k}\}_{k=0}^{\infty}$ be Fejér monotone with respect to $C$. If $C$ is an affine subspace and all weak cluster points of $\{x^{k}\}_{k=0}^{\infty}$ belong to $C$, then, by [13, Fact 5.3 (ii)], $x^{k}\rightharpoonup x^{\infty}=P_{C}x^{0}$. Consequently, if every $C_{i}$ in Theorem 3.1 is an affine subspace, then the sequence defined in (30) converges weakly to $x^{\infty}=P_{C}x^{0}$. Moreover, this convergence turns out to be either strong or a linear one if $\mathcal{C}$ is either boundedly regular or boundedly linearly regular, respectively. *

3.1 Projection algorithms

Unless otherwise stated, from now on we assume that $C:=\bigcap_{i\in I}C_{i}\neq\emptyset$ and $C_{i}\subseteq\mathcal{H}$ is closed and convex for every $i\in I$.

Corollary 3.9** (Weak and strong convergence).**

Let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be defined by the double-layer projection algorithm, that is,

[TABLE]

where $I_{k}\subseteq J_{k}\subseteq I$, $\alpha_{k}\in(0,2)$ and $\omega_{i}^{k}\in(0,1]$ are such that $\sum_{i\in I_{k}}\omega_{i}^{k}=1$. Assume that

1. (i)

$\alpha_{k}\in[\alpha^{-},\alpha^{+}]$* and $\omega^{i}_{k}\in[\omega^{-},1]$, where $\alpha^{-},\omega^{-}\in(0,1]$ and $\alpha^{+}\in[1,2)$;* 2. (ii)

$I=J_{k}\cup\ldots\cup J_{k+s-1}$* *for every $k=0,1,2,\ldots$ and some integer $s\geq 1$; 3. (iii)

the inner and outer controls satisfy

[TABLE]

Then the sequence $\{x^{k}\}_{k=0}^{\infty}$ converges weakly to some point $x^{\infty}\in C$. Moreover, if the family $\mathcal{C}:=\{C_{i}\mid i\in I\}$ is boundedly regular, then the convergence to $x^{\infty}$ is in norm.

Proof 3.10**.**

*This result follows from Theorem 3.1, where it suffices to substitute $U_{i}=P_{C_{i}}$ and $p_{i}(x)=d(x,C_{i})$. *

Corollary 3.11** (Linear convergence).**

Let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be defined as in Corollary 3.9 under conditions (i) and (ii). Moreover, assume that

1. (iii)

the inner and outer controls satisfy, for each $k=0,1,2,\ldots$,

[TABLE]

If the family $\mathcal{C}:=\{C_{i}\mid i\in I\}$ is $\kappa_{r}$-boundedly linearly regular over $B(P_{C}x^{0},r)$, then the sequence $\{x^{k}\}_{k=0}^{\infty}$ converges linearly to some point $x^{\infty}\in C$, that is, $\|x^{k}-x^{\infty}\|\leq c_{r}q_{r}^{k}$, where

[TABLE]

(Error bound)* Moreover, we have the following estimate*

[TABLE]

Proof 3.12**.**

*This result follows from Theorem 3.3 by substituting $U_{i}=P_{C_{i}}$ and $p_{i}(x)=d(x,C_{i})$. Note that in this case $\delta_{r}=\Delta_{r}=1$ and $U_{i}$ is linearly regular, $i\in I$. *

Example 3.13** (Example 3.6 revisited).**

We now present several projection methods (PM) which we obtain by substituting $U_{i}=P_{C_{i}}$ and $p_{i}=d(\cdot,C_{i})$ in Example 3.6. By assuming the existence of $\kappa_{r}>0$ we can apply Corollary 3.11 to find the constant $q_{r}$.

1. (a)

Cyclic PM: $x^{k+1}:=P_{C_{[k]}}x^{k}$, where $s=m$ and

[TABLE]

A similar rate can be found, for example, in **[44, Theorem 1]** by Gurin et al., in **[7, Theorem 5.7]** by Bauschke and Borwein, and in **[37, Theorem 4.5]** by Deutsch and Hundal. 2. (b)

Simultaneous PM: $x^{k+1}:=\frac{1}{b}\sum_{i\in J_{[k]}}P_{C_{i}}x^{k}$, where

[TABLE]

A similar $q_{r}$ can be deduced from **[15, Theorem 2.2]** by Beck and Teboulle with $b=m$. We recall that a linear convergence rate for $b=m$ appeared in **[52, Theorem 1.1]** by Pierra. 3. (c)

Simultaneous PM with active sets: $x^{k+1}:=\frac{1}{|I_{k}|}\sum_{i\in I_{k}}P_{C_{i}}x^{k},$ where $I_{k}:=\{i\in J_{[k]}\mid d(x^{k},C_{i})>0\}$. By similar arguments to those used in Example 3.6, the upper bound for $q_{r}$ is the same as in (87). 4. (d)

Remotest-set PM: $x^{k+1}:=P_{C_{i_{k}}}x^{k}$, where $i_{k}=\operatorname*{argmax}_{j\in J_{[k]}}d(x^{k},C_{j})$ and

[TABLE]

A similar estimate for $b=m$ can be found, for example, in **[44, Theorem 1]** by Gurin et al. and in **[7, Theorem 5.8]** by Bauschke and Borwein. It can also be deduced from **[15, Theorem 2.2]** by Beck and Teboulle. We note here that originally, for $b\leq m$, this PM appeared in **[20, Section 5.8.4]** by Cegielski. 5. (e)

Simultaneous PM with the inner block of a fixed size: $x^{k+1}:=\frac{1}{t}\sum_{i\in I_{k}}P_{C_{i}}x^{k},$ where $I_{k}:=\{t$ smallest indices from $J_{[k]}$ with the largest distance $d(x^{k},C_{j})\}$. We have

[TABLE] 6. (f)

Simultaneous PM with the inner block determined by the threshold: $x^{k+1}:=\frac{1}{|I_{k}|}\sum_{i\in I_{k}}P_{C_{i}}x^{k},$ where $I_{k}:=\{i\in J_{[k]}\mid d(x^{k},C_{i})\geq t\max_{j\in J_{[k]}}d(x^{k},C_{j})\}$ and $t\in[0,1]$. Since we do not control the number of constraints for which the distance functional is above the threshold, we can only estimate the upper bound for $q_{r}$ by using (89) and setting there $t=b$. This leads us to formula (87). We recall here that originally, for $b\leq m$, this PM appeared in **[20, Section 5.8.4]** by Cegielski.

Remark 3.14**.**

*Notice that for the projection methods designed for solving linear systems ($Ax=b$) and systems of linear inequalities ($Ax\leq b$), we can define the proximities $p_{i}(x)=|\langle a_{i},x\rangle-b_{i}|$ and $p_{i}(x)=(\langle a_{i},x\rangle-b_{i})_{+}$, respectively. By setting $U_{i}=P_{C_{i}}$ we see that $\delta_{r}=\min_{i}\|a_{i}\|$ and $\Delta_{r}=\max_{i}\|a_{i}\|$. *

3.2 Subgradient projection algorithms

Unless otherwise stated, from now on we assume that $C:=\bigcap_{i\in I}C_{i}\neq\emptyset$ and for every $i\in I$, we have

[TABLE]

where $f_{i}\colon\mathcal{H}\rightarrow\mathbb{R}$ is convex and continuous.

Corollary 3.15** (Weak convergence in $\mathcal{H}$).**

Let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be defined by the double-layer subgradient projection algorithm, that is,

[TABLE]

where $I_{k}\subseteq J_{k}\subseteq I$, $\alpha_{k}\in(0,2)$ and $\omega_{i}^{k}\in(0,1]$ are such that $\sum_{i\in I_{k}}\omega_{i}^{k}=1$. Assume that

1. (i)

$\alpha_{k}\in[\alpha^{-},\alpha^{+}]$* and $\omega^{i}_{k}\in[\omega^{-},1]$, where $\alpha^{-},\omega^{-}\in(0,1]$ and $\alpha^{+}\in[1,2)$;* 2. (ii)

$I=J_{k}\cup\ldots\cup J_{k+s-1}$* *for every $k=0,1,2,\ldots$ and some integer $s\geq 1$; 3. (iii)

the inner and outer controls satisfy

[TABLE] 4. (iv)

for each $i\in I$, the function $f_{i}$ is Lipschitz continuous on bounded sets.

*Then, by (iv), $P_{f_{i}}$ is weakly regular, $i\in I$. Consequently, the sequence $\{x^{k}\}_{k=0}^{\infty}$ converges weakly to some point $x^{\infty}\in C$. *

Proof 3.16**.**

*The weak regularity of $P_{f_{i}}$ follows from (iv); compare with Example 2.20 (a). The rest follows from Theorem 3.1, where it suffices to substitute $U_{i}=P_{f_{i}}$ and $p_{i}(x)=\|P_{f_{i}}x-x\|$. *

Remark 3.17**.**

*We emphasize that in view of Example 2.20 (a), one can replace condition (iv) either by: $f_{i}$ maps bounded sets onto bounded sets, $i\in I$; or by: $\partial f_{i}$ is uniformly bounded on bounded sets, $i\in I$. *

Corollary 3.18** (Convergence in $\mathbb{R}^{n}$).**

Let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be defined as in Corollary 3.15 under conditions (i) and (ii). Moreover, assume that

1. (iii)

the inner control and outer control satisfy

[TABLE] 2. (iv)

$\mathcal{H}=\mathbb{R}^{n}$.

Then, by (iv), for every bounded sequence $\{y^{k}\}_{k=0}^{\infty}\subseteq\mathbb{R}^{n}$ and $i\in I$,

[TABLE]

*In particular, this holds for $\{x^{k}\}_{k=0}^{\infty}$ defined in (91). Consequently, the sequence $\{x^{k}\}_{k=0}^{\infty}$ converges to some point $x^{\infty}\in C$. *

Proof 3.19**.**

To apply Theorem 3.1 with $U_{i}=P_{f_{i}}$ and $p_{i}=f_{i}^{+}$, it suffices to show that condition (94) holds true. Indeed, let $\{y^{k}\}_{k=0}^{\infty}\subseteq S$, where $S\subseteq\mathbb{R}^{n}$ is bounded. Since $\mathcal{H}=\mathbb{R}^{n}$, for each $i\in I$, the subdifferential $\partial f_{i}$ is uniformly bounded on bounded sets; see Example 2.20 (a). In particular, there is $\Delta_{S}$ such that for every $i\in I$ and $x\in S$, $\|g_{f_{i}}(x)\|\leq\Delta_{S}$. Consequently, for every $k=0,1,2,\ldots,$ we have

[TABLE]

Therefore, by the definition of the subgradient projection, see (7),

[TABLE]

where the last inequality follows from the fact that $P_{f_{i}}$ is a cutter; see (8) and Remark 2.10.

*To complete the proof it suffices to show that $f^{+}_{i}(y^{k})\rightarrow 0$ implies that $d(y^{k},C_{i})\rightarrow 0$. Indeed, following [26, Lemma 24], we assume to the contrary that $f^{+}_{i}(y^{k})\rightarrow 0$ and $\limsup_{k}d(y^{k},S(f_{i},0))=\varepsilon>0$. Since $\{y^{k}\}_{k=0}^{\infty}$ is bounded, by eventually passing to a subsequence, without loss of generality we may assume that $y^{k}\rightarrow y$ and $\lim_{k}d(y^{k},S(f_{i},0))=\varepsilon$. By continuity of $d(\cdot,S(f_{i},0))$, we have $d(y,S(f_{i},0))=\varepsilon$. The lower semicontinuity of $f^{+}_{i}$ implies that $0=\lim_{k}f^{+}_{i}(y^{k})=\liminf_{k}f^{+}_{i}(y^{k})\geq f^{+}_{i}(y)$, which is in contradiction with $\varepsilon>0$. *

Corollary 3.20** (Linear convergence in $\mathbb{R}^{n}$).**

Let the sequence $\{x^{k}\}_{k=0}^{\infty}$ be defined as in Corollary 3.15 under conditions (i) and (ii). Moreover, assume that

1. (iii)

the inner and outer controls satisfy, for each $k=0,1,2,\ldots$,

[TABLE] 2. (iv)

$\mathcal{H}=\mathbb{R}^{n}$* and for each $i\in I$, *there is a point $z^{i}$ such that $f_{i}(z^{i})<0$.

Then, by (iv), for $r=d(x^{0},C)>0$, there are numbers $\delta_{r},\Delta_{r}>0$ such that for every $x\in B(P_{C}x^{0},r)$ and $i\in I$,

[TABLE]

In addition, if the family $\mathcal{C}:=\{S(f_{i},0)\mid i\in I\}$ is $\kappa_{r}$-boundedly linearly regular over $B(P_{C}x^{0},r)$, in particular, when $\max_{i\in I}f_{i}(z)<0$ for some $z$ (Slater condition), then the sequence $\{x^{k}\}_{k=0}^{\infty}$ converges linearly to some point $x^{\infty}\in C$, that is, $\|x^{k}-x^{\infty}\|\leq c_{r}q_{r}^{k}$, where

[TABLE]

(Error bound)* Moreover, we have the following estimate:*

[TABLE]

Proof 3.21**.**

It suffices to show that condition (98) holds which, by Theorem 3.3 with $U_{i}=P_{f_{i}}$ and $p_{i}=f_{i}^{+}$, yields the result. Indeed, since $\mathcal{H}=\mathbb{R}^{n}$, for each $i\in I$, the subdifferential $\partial f_{i}$ is uniformly bounded on bounded sets; see Example 2.20 (a). This holds, in particular, for the ball $B(P_{C}x^{0},r=d(x^{0},C))$. Therefore there is a number $\Delta_{r}$ such that for every $i\in I$ and $x\in B(P_{C}x^{0},r)$, we have $\|g_{f_{i}}(x)\|\leq\Delta_{r}$. Thus, by the definition of the subgradient projection, see (7), we get

[TABLE]

*Now, assume that for each $i\in I$, there is $z^{i}\in S(f_{i},0)$ such that $f_{i}(z^{i})<0$ (Condition (iv)). Then, by Lemma 2.1 applied to eaxch $f_{i}$ separately, for every compact set $K$ there is $\delta_{K}>0$ such that for every $x\in K$, $\delta_{K}d(x,S(f_{i},0))\leq f_{i}^{+}(x)$. This holds, in particular, for $K=B(P_{C}x^{0},r)$ with a common constant $\delta_{r}$, which completes the proof. *

Example 3.22** (Examples 3.6 and 3.13 continued).**

We now present several subgradient projection methods (SPM), which we obtain by substituting $U_{i}=P_{f_{i}}$ and $p_{i}=f_{i}^{+}$ in Example 3.6. By assuming the existence of $\kappa_{r}>0$ and condition (iv) from Corollary 3.11, we can obtain the constant $q_{r}$.

1. (a)

Cyclic SPM: $x^{k+1}:=P_{f_{[k]}}x^{k}$ with $q_{r}$ defined in (74). A similar rate can be found, for example, in **[34, Theorem 2]** by De Pierro and Iusem. 2. (b)

Simultaneous SPM: $x^{k+1}:=\frac{1}{b}\sum_{i\in J_{[k]}}P_{f_{i}}x^{k}$ with $q_{r}$ defined in (75). 3. (c)

Simultaneous SPM with active sets: $x^{k+1}:=\frac{1}{|I_{k}|}\sum_{i\in I_{k}}P_{f_{i}}x^{k},$ where $I_{k}:=\{i\in J_{[k]}\mid f_{i}^{+}(x^{k})>0\}$ where $q_{r}$ is defined in (75). 4. (d)

SPM with the most-violated constraint, that is: $x^{k+1}:=P_{f_{i_{k}}}x^{k}$, where $i_{k}=\operatorname*{argmax}_{j\in J_{[k]}}f_{j}^{+}(x^{k})$ and $q_{r}$ is defined by formula (76). We recall that for $b=m$, the linear convergence rate can be found, for example, in **[40, Theorem, p. 142]** by Eremin and in **[53, Theorem 6]** by Polyak. 5. (e)

Simultaneous SPM with the inner block of a fixed size: $x^{k+1}:=\frac{1}{t}\sum_{i\in I_{k}}P_{f_{i}}x^{k},$ where $I_{k}:=\{t$ smallest indices from $J_{[k]}$ with the largest violation $f_{j}^{+}(x^{k})\}$, where $q_{r}$ is defined in (77). 6. (f)

Simultaneous SPM with the inner block determined by the threshold: $x^{k+1}:=\frac{1}{|I_{k}|}\sum_{i\in I_{k}}P_{f_{i}}x^{k},$ where $I_{k}:=\{i\in J_{[k]}\mid f_{i}^{+}(x^{k})\geq t\max_{j\in J_{[k]}}f_{j}^{+}(x^{k})\}$ and $t\in[0,1]$. Here $q_{r}$ is defined in (75).

3.3 Lopping and flagging

Note that the simple choice of the outer control $\{J_{k}\}_{k=0}^{\infty}$ presented in Example 18 can easily be extended by applying the so-called lopping and flagging methodology [46, 39]. Both of these techniques were brought to our attention by an anonymous referee. This, in particular, can be considered a practical realization of the $s$-intermittent control; see (3). We recall that the idea of lopping presented in [46, 39] is to omit computation of algorithmic operators assigned to a block if that block has no active constraints and proceed to another block. One checks if the block has active constraints by comparing a certain violation measure with a user chosen threshold. By extending this idea one can flag the nonactive block as not available for the upcoming $N$ iterations and unflag it afterwards, so that it becomes available again. To this end, assume that $I=J_{1}^{\prime}\cup\ldots\cup J_{s}^{\prime}$ for some $s\geq 1$, $J_{k}^{\prime}=J_{[k]}^{\prime}$ for all $k\geq s+1$, where $[k]=(k\text{ mod }s)+1$. Moreover, let $N\geq 1$ and $\varepsilon\geq 0$ be user chosen parameters. We will use $\varepsilon$ for flagging blocks as well as for the stopping rule. Following [39], we propose the following algorithm:

{algorithm2}

[Double layer control with lopping and flagging]

• Initialize. Set $n=k:=0$, $J_{0}=J_{0}^{\prime}=:J_{1}^{\prime}$ and flag all the blocks $\{J_{k}^{\prime}\}_{k=0}^{\infty}$ as available. Moreover, choose $x^{0}\in\mathcal{H}$ and mark $J_{0}^{\prime}$ as the last used block.
• Step 1. For a given $k\geq 0$ and $x^{k}$, let $J_{l}^{\prime}$ be the last used block for some $l\geq k$. Take the next available block after $J_{l}^{\prime}$ and denote it by $J_{t}^{\prime}$. Set $J_{k}:=J_{t}^{\prime}$.
• Step 2. If $\max_{j\in J_{k}}p_{j}(x^{k})>\varepsilon$, then set $n:=0$, select a proper $I_{k}$ from $J_{k}$ and compute $x^{k+1}$ using formula (29). If $\max_{j\in J_{k}}p_{j}(x^{k})\leq\varepsilon$, then set $n:=n+1$, $x^{k+1}:=x^{k}$ and flag blocks $J_{t}^{\prime},J_{t+s}^{\prime},\ldots,J_{t+Ns}^{\prime}$ as not available.
• Step 3. If $n<s$ then set $k:=k+1$ and return to step 1. If $n=s$ then stop the algorithm since $\max_{i\in I}p_{i}(x^{k})\leq\varepsilon$.

It is easy to see that every basic block $J_{l}^{\prime}$, $l=1,\ldots,s$, has to appear at least once in $\{J_{k},\ldots,J_{k+Ns-1}\}$ and therefore $\{J_{k}\}$ is $(Ns)$-intermittent. Therefore, when $\varepsilon=0$, the sequence $\{x^{k}\}_{k=0}^{\infty}$ defined above may converge either weakly, strongly or even linearly by Theorems 16 and 17. When $\varepsilon>0$, then one can show that in a finite number of iterations we can reach the stopping rule $\max_{i\in I}p_{i}(x^{k})<\varepsilon$ under the assumption that the inner and outer controls satisfy (46), and that $p_{i}$ and $U_{i}$ are related as in (31). The argument is a slight modification of the proof of Theorem 16. We sketch it below for the convenience of the reader.

Indeed, let $K:=\{k\mid\max_{j\in J_{k}}p_{i}(x^{k})>\varepsilon\}$. Note that $x^{k+1}\neq x^{k}$ may happen only when $k\in K$. If $K$ is finite then the algorithm stops. Assume that $K$ is infinite. For every $k\in K$ and $z\in C$, we have

[TABLE]

where $T_{k}$ and $\rho$ are defined as in (32)–(33). Therefore, for every $k=0,1,2,\ldots$, we have $\|x^{k+1}-z\|\leq\|x^{k}-z\|$. This implies that the sequence $\{x^{k}\}_{k=0}^{\infty}$ is Fejér monotone with respect to $C$, $\{\|x^{k}-z\|\}_{k=0}^{\infty}$ converges and consequently,

[TABLE]

In view of (38), which holds for every $k\in K$ by (31) and (46), we get

[TABLE]

This implies that that there is $k_{0}$ such that for every $k\geq k_{0}$, $\max_{j\in J_{k}}p_{j}(x^{k})\leq\varepsilon$. Note that for $k\notin K$ this follows from the definition of the sequence $\{x^{k}\}_{k=0}^{\infty}$. Consequently, $K$ needs to be finite.

4 Numerical experiments

In this section we test the performance of selected algorithms on systems of linear inequalities $Ax\leq b$, where the dimensions of the matrix $A$ are $100\times 20$. The linear inequalities are randomly generated so that they form a consistent convex feasibility problem.

Since for every $i\in I$, the $i$-th inequality corresponds to a half-space $C_{i}=\{z\in\mathbb{R}^{20}\mid\langle a_{i},z\rangle\leq b_{i}\}$, following Remark 3.14, we define

[TABLE]

To design our algorithms, we repeat the setting of Example 3.6. First, we divide $I=\{1,\ldots,100\}$ into $s$ equal blocks of size $b$, that is, $I=J_{1}\cup\ldots\cup J_{s}$. For the last block we allow $|J_{s}|\leq b$. Second, we apply a cyclic outer control $\{J_{[k]}\}_{k=0}^{\infty}$, where $[k]:=(k\text{ mod }s)+1$. Moreover, we set $\alpha_{k}=1$ and $\omega_{i}^{k}=1/|I_{k}|$ for every $k=0,1,2,\ldots$.

The algorithms in which we are interested are the following projection methods:

• •

Cyclic PM: $x^{k+1}:=P_{C_{[k]}}x^{k}$, where in this case $s=m$ and thus $[k]:=(k\text{ mod }m)+1$;

• •

Simultaneous PM: $x^{k+1}:=\frac{1}{|J_{[k]}|}\sum_{i\in J_{[k]}}P_{C_{i}}x^{k}$;

• •

Maximum proximity PM: $x^{k+1}:=P_{C_{i_{k}}}x^{k}$, where $i_{k}=\operatorname*{argmax}_{j\in J_{[k]}}p_{j}(x^{k})$;

• •

Simultaneous PM with the inner block of a fixed size: $x^{k+1}:=\frac{1}{t}\sum_{i\in I_{k}}P_{C_{i}}x^{k},$ where $I_{k}:=\{t$ smallest indices from $J_{[k]}$ with the largest proximity $p_{j}(x^{k})\}$.

• •

Simultaneous PM with the inner block determined by a threshold: $x^{k+1}:=\frac{1}{|I_{k}|}\sum_{i\in I_{k}}P_{C_{i}}x^{k},$ where $I_{k}:=\{i\in J_{[k]}\mid p_{i}(x^{k})\geq t\max_{j\in J_{[k]}}p_{j}(x^{k})\}$ and $t\in[0,1]$.

Note that in view of Remark 3.14, for each one of the above-mentioned PM’s we can apply the formulae of Example 3.6 with $\delta_{r}=\min_{i}\|a_{i}\|$ and $\Delta_{r}=\max_{i}\|a_{i}\|$.

For every algorithm we perform $N=100$ simulations, while sharing the same set of test problems. We apply two types of stopping rules: the first one is $\max_{i\in I}p_{i}(x^{k})\leq 10^{-6}$, which we verify per every 100 iterations (first matrix dimension); the second one is an upper bound of 5000 iterations. After running all of the simulations, for every iterate we compute the maximum proximity. In order to compare our algorithms, we consider the quantity

[TABLE]

The bold line in Figures 1-4 indicates the median computed for (106). The ribbon plot represents the concentration of order 10, 20, 30, 40 and 50% around the median.

We present now several observations which we have noticed after running the numerical simulations. These observations confirmed some of the predictions which followed Example 3.6.

1. (a)

The maximum proximity PM with the largest possible outer block $J_{[k]}=I$ outperformed every other PM we considered, while simultaneous PM seems to be the slowest one; see Figures 1–3. 2. (b)

By reducing the block size $b$ in the maximum proximity PM, we approach the convergence rate of the cyclic PM. By increasing the size of the block, we speed up the convergence in terms of iterations. Even for the small blocks, where $b=2,3,5$, the improvement is significant in comparison with the cyclic method. Moreover, in the case of $b=25$, the performance is almost the same as in the case of $b=100$; see Figure 1. Therefore, in general, one could expect that the maximum proximity algorithms may have the same convergence properties with some $b<m$ as they have in the case of $b=m$. 3. (c)

In Figure 2 we regulate the size of the inner block by adjusting the parameter $t$. Following (b), we have fixed the size of the outer block to be 25 ($b=25$). By reducing the parameter $t$ we approach the maximum proximity PM, while by increasing $t$ we approach the simultaneous PM. Surprisingly, by taking into account more information from the outer block we reduce, in fact, the convergence speed. 4. (d)

In Figure 3 we control the size of the inner block by modifying the threshold determined by $t$. As above, we have fixed the size of the outer block to be 25. Similarly, by increasing the threshold we approach the maximum proximity PM and thus we speed up the convergence. On the other hand, by decreasing $t$ we approach the simultaneous PM which consequently slows down the convergence. As in case (c), by considering more information from the outer block once again, we reduce the convergence speed. 5. (e)

Observations (c) and (d), as well as formula (77), led us to the conclusion that the smaller the ratio between the sizes of the inner and outer blocks is, the faster convergence we can expect. We show this effect in Figure 4, where the ratios $t/b$ are $0.3=3/10=6/20=15/50$, $0.5=5/10=10/20=25/50$ and $0.7=7/10=14/20=35/50$.

Acknowledgments

We are grateful to two anonymous referees for all their comments and remarks which helped us to improve our manuscript.