On Remotest set and Random controls in Kaczmarz algorithm

TL;DR

This paper analyzes the Kaczmarz algorithm with Remotest set and Random control, providing conditions ensuring each index appears infinitely often during iterations, which enhances understanding of its convergence behavior.

Contribution

It introduces a sufficient condition for the recurrence of projection indices in the Kaczmarz algorithm with specific control strategies.

Findings

Provided a sufficient condition for index recurrence in the algorithm.

Enhanced understanding of convergence properties under Remotest set and Random control.

Contributed to the theoretical analysis of the Kaczmarz method.

Abstract

In this paper we analyse the Kaczmarz projection algorithm with Remotest set and Random control of projection indices and provide a sufficient condition such that each projection index appears infinitely many times during the iterations.

Equations

$$Ax=b$$

$$P_{H_i}(x)=x-\frac{⟨x,A_i⟩-b_i}{{\Vert A_i\Vert }^2}{A}_i.$$

$$x^{k+1}=P_{H_{i_k}}(x^k).$$

$$\{1,2,\dots ,m\}\subseteq \{i({\tau }_k),i({\tau }_k+1),\dots ,i({\tau }_{k+1}-1)\}.$$

$$\forall k\ge 1,\exists k_i\ge ks.t.i_k=i.$$

$${\tau }_1>{\tau }_0\mathrm{and}\{1,2,\dots ,m\}\subseteq \{i({\tau }_0),\dots ,i({\tau }_1-1)\}.$$

$${\tau }_{k+1}>{\tau }_k\mathrm{and}\{1,2,\dots ,m\}\subseteq \{i({\tau }_k),\dots ,i({\tau }_{k+1}-1)\},$$

$$|⟨A_{i_k},x^{k-1}⟩-b_{i_k}|=\underset{1\le i\le m}{\max }|⟨A_i,x^{k-1}⟩-b_i|.$$

$${\Delta }_m=\{x\in I{R}^m,x\ge 0,\sum _{i=1}^m=1\},$$

$$p\in {\Delta }_m,p_i=\frac{\Vert A_i{\Vert }^2}{\Vert A{\Vert }_F^2},i=1,\dots ,m,$$

$$i_k\sim p.$$

$$x_{LS}\ne x_{LS}^{(i)}.$$

$$i_k\ne i_0,\forall k\ge k_0.$$

$$\underset{0\ge k\to \infty }{\lim }{x}^k=x_{LS}=\underset{k_0\ge k\to \infty }{\lim }{x}^k=x_{LS}^{(i)},$$

$$x_{LS}=x_{LS}^{(i_0)},$$

$$m\le n\mathrm{and}rank(A)=m,$$

$$x_{LS}={\tilde{x}}_{LS}.$$

$$Q^{T}A^{T}=\left[\begin{array}[]{c}R\\ 0\\ \end{array}\right]=\left[\begin{array}[]{ccccc}r_{11}&r_{12}&\dots&r_{1,m-1}&r_{1m}\\ 0&r_{22}&\dots&r_{2,m-1}&r_{2m}\\ \dots&\dots&\dots&\dots&\dots\\ 0&0&\dots&r_{m-1,m-1}&r_{m-1,m}\\ 0&0&\dots&0&r_{mm}\\ 0&0&\dots&0&0\\ \vdots&&&\vdots&\vdots\\ 0&0&\dots&0&0\\ \end{array}\right]=$$

$$Q^{T}[\tilde{A}^{T}|A_{m}]=[Q^{T}\tilde{A}^{T}|Q^{T}A_{m}]=\left[\begin{array}[]{ccc}\tilde{R}&&c\\ 0&&r_{mm}\\ 0&\dots&0\\ \vdots&&\vdots\\ 0&\dots&0\\ \end{array}\right].$$

$$Q^{T}\tilde{A}^{T}=\left[\begin{array}[]{c}\tilde{R}\\ 0\\ \end{array}\right]$$

$$\tilde{R}=\left[\begin{array}[]{cccc}r_{11}&r_{12}&\dots&r_{1,m-1}\\ 0&r_{22}&\dots&r_{2,m-1}\\ \dots&\dots&\dots&\dots\\ 0&0&\dots&r_{m-1,m-1}\\ \end{array}\right]~{}{\rm and}~{}c=(r_{1m},r_{2m},\dots,r_{m-1,m})^{T}.$$

$$A^{+}=A^T(A{A}^T{)}^{-1},{\tilde{A}}^{+}={\tilde{A}}^T(\tilde{A}{\tilde{A}}^T{)}^{-1},$$

$$x_{LS}=A^{+}b=Q\left[\begin{array}[]{c}{R^{-T}}b\\ 0\\ \end{array}\right],~{}~{}\tilde{x}_{LS}=\tilde{A}^{+}\tilde{b}=Q\left[\begin{array}[]{c}\tilde{R}^{-T}\tilde{b}\\ 0\\ \end{array}\right].$$

$${R^{-T}}b=\left[\begin{array}[]{c}\tilde{R}^{-T}\tilde{b}\\ 0\\ \end{array}\right],~{}{\rm where}~{}R^{T}=\left[\begin{array}[]{cc}\tilde{R}^{T}&0\\ c^{T}&r_{mm}\\ \end{array}\right]:m\times m.$$

$$R^{-T}=(R^{T})^{-1}=\left[\begin{array}[]{cc}\tilde{R}^{-T}&0\\ -\frac{1}{r_{mm}}c^{T}\tilde{R}^{-T}&\frac{1}{r_{mm}}\\ \end{array}\right],$$

$$\left[\begin{array}[]{c}\tilde{R}^{-T}\tilde{b}\\ 0\\ \end{array}\right]=\left[\begin{array}[]{cc}\tilde{R}^{-T}&0\\ -\frac{1}{r_{mm}}c^{T}\tilde{R}^{-T}&\frac{1}{r_{mm}}\\ \end{array}\right]\left[\begin{array}[]{c}\tilde{b}\\ b_{m}\\ \end{array}\right]=$$

$$\left[\begin{array}[]{c}\tilde{R}^{-T}\tilde{b}\\ -\frac{1}{r_{mm}}c^{T}\tilde{R}^{-T}\tilde{b}+\frac{b_{m}}{r_{mm}}\\ \end{array}\right],$$

$$0=-\frac{1}{r_{mm}}{c}^T{\tilde{R}}^{-T}\tilde{b}+\frac{b_m}{r_{mm}}\mathrm{or}{c}^T{\tilde{R}}^{-T}\tilde{b}=b_m.$$

$$c^T{\tilde{R}}^{-T}\tilde{b}=b_m,$$

Full text

A note on Kaczmarz algorithm with remotest set control sequence

CONSTANTIN POPA

Ovidius University of Constanta, Blvd. Mamaia 124, Constanta 900527, Romania; cpopa@univ-ovidius.ro

“Gheorghe Mihoc - Caius Iacob” Institute of Statistical Mathematics and Applied Mathematics of the Romanian Academy, Calea 13 Septembrie, Nr. 13, Bucharest 050711, Romania

Abstract. In this paper we analyse the Kaczmarz projection algorithm with Remotest set and Random control of projection indices and provide a sufficient condition such that each projection index appears infinitely many times during the iterations.

Keywords: Kaczmarz algorithm; remotest set control

MSC (2010): 65F10; 65F20

1 Introduction

For an $m\times n$ (real) matrix $A$ and $b\in I\!\!R^{m}$ let

[TABLE]

be a consistent system of linear equations and denote by $S(A;b),x_{LS}$ the set of its solutions and the minimal (Euclidean) norm one ($\left\langle\cdot,\cdot\right\rangle$ and $\parallel\cdot\parallel$ will denote the Euclidean scalar product and norm on some space $I\!\!R^{q}$, respectively). Other notations used will be $A^{T},A_{i},A^{j},{\cal R}(A),{\cal N}(A)$, ${rank}(A)$ for the transpose, $i$-th row, $j$-th column, range, null space and rank of $A$. The projection onto a nonempty closed convex set $V$ will be denoted by $P_{V}$, and for $V=H_{i}=\left\{x\in I\!\!R^{n},\left\langle x,A_{i}\right\rangle=b_{i}\right\}$ (the hyperplane determined by the $i$-th equation of the system (1)) we know that

[TABLE]

The Kaczmarz’s iterative method for numerical solution of (1) has the form from below.

Algorithm K.

Initialization: $x^{0}\in I\!\!R^{n}$

Iterative step: for $k=0,1,\dots$ select $i_{k}\in\{1,2,\dots,m\}$ and compute $x^{k+1}$ as

[TABLE]

There have been defined several classes of selection procedures for the indices $i_{k}$ (see [4, 3, 5, 6] and references therein). In this paper we will consider the Maximal Residual (remotest set) control and the Random control procedures, and provide a sufficient condition such that in the case of Kaczmarz’s projection method K, they belong to the class of control selections from [5]. The paper is organized as follows: in section 2 we give an equivalent formulation of the control sequence definition from [5]. In section 3 we show that, for $x^{0}=0$ and under additional assumptions, the Maximal Residual (remotest set) selection or the Random selection is a control w.r.t. section 2.

2 Control sequences

Let $I\!\!N$ denote the set of natural numbers $\{0,1,2,\dots,\}$. In [5] the following definition concerning control sequences was introduced.

Definition 1

(D1)*

Given a monotonically increasing sequence $\{\tau_{k}\}_{k\geq 0}\subset I\!\!N$, a mapping $i:I\!\!N\rightarrow\{1,2,\dots,m\}$ is called a control with respect to the sequence $\{\tau_{k}\}_{k\geq 0}$ if it defines a sequence $\{i(t)\}_{t\geq 0}$, such that for all $k\geq 0$,*

[TABLE]

The next definiton, mentioned in [2] (see also [7]) points-out on an important aspect of control sequences.

Definition 2

(D2)*

A mapping $i:I\!\!N\rightarrow\{1,2,\dots,m\}$ is called a random mapping if any $i\in\{1,\dots,m\}$ appears infinitely many times in the set ${\cal{I}}=\{i(k),k\geq 0\}$.*

It is clear that, if the mapping $i$ is a control with respect to some sequence $\{\tau_{k}\}_{k\geq 0}$, then it is also a random mapping, according to definition (D2). Indeed, if the sequence $\{\tau_{k}\}_{k\geq 0}$ is increassing then $\tau_{k+1}>\tau_{k}$ and the sets $\Delta_{k}=\{i(\tau_{k}),i(\tau_{k}+1),\dots,i(\tau_{k+1}-1)\},k\geq 0$ form a partition of $I\!\!N$ as in (4). Next proposition tell us about the reciprocal of this property, i.e. a random mapping is a control according to the definition (D1).

Proposition 1

Let ${\rm i}:I\!\!N\rightarrow\{1,2,\dots,m\}$ be a random mapping (according to (D2)). Then it exists a monotonically increasing sequence $\{\tau_{k}\}_{k\geq 0}\subset I\!\!N$ such that ${\rm i}$ is a control w.r.t. (D1).

Proof.

We will first write (D2) in the following equivalent formulation: for any $i\in\{1,\dots,m\}$ it is true that

[TABLE]

We will now recursively define an increasing sequence $\{\tau_{k}\}_{k\geq 0}\subset I\!\!N$ as follows: for $k=0$ we set $\tau_{0}=0$; for $k=1$ let $\tau_{1}$ be the smallest natural number with the properties

[TABLE]

Such a number $\tau_{1}$ exists according to the equivalent formulation (5). In general, if we already have constructed $\tau_{k}$, then $\tau_{k+1}$ will be the smallest natural number such that

[TABLE]

which is exactly the property (4) of definition (D1), and the proof is complete. $\spadesuit$

Based on the above proposition we will consider in the rest of the paper as definition for controls the equivalent formulation from (D2). In this respect, the following two selection procedures will be analysed.

• •

Maximal Residual (remotest set) ([1]): Select $i_{k}\in\{1,2,\dots,m\}$ such that

[TABLE]

• •

Random ([9]): Let the set $\Delta_{m}\subset I\!\!R^{m}$ be defined by

[TABLE]

define the discrete probability distribution

[TABLE]

and select $i_{k}\in\{1,2,\dots,m\}$ such that

[TABLE]

The main aspect regarding the above two selection procedures is concerned with the fact that the projection indices are generated recursively, without no a priori information on them. And, at least related to author’s knowledge, there are no results saying that when the algorithm K is applied with one or the other of the above selection procedures, each projection index will appear infinitely many times.

3 The Kaczmarz algorithm

We consider in this section Kaczmarz’s projection algorithm in which the Maximal Residual (remotest set) (8) or Random (10)-(11) procedure is used for selecting the projection indices in each iteration, and with the initial approximation $x^{0}=0$. In this case, in papers [1] and [9] it is proved that the sequence $(x^{k})_{k\geq 0}$ generated by algorithm K converges to the minimal norm solution $x_{LS}$ of the system (1). We will formulate a sufficient condition such that any of the above selection procedures satisfies (D2)). For $i\in\{1,\dots,m\}$ arbitrary fixed, let ${A}^{(i)}:(m-1)\times n$, ${b}^{(i)}\in I\!\!R^{m-1}$ be the submatrix of $A$ without the $i$-th row, respectively the subvector of $b$ without the $i$-th component and ${x}^{(i)}_{LS}$ the minimal norm solution of the system ${A}^{(i)}{x}={b}^{(i)}$.

Assumption C. For any index $i\in\{1,\dots,m\}$ we have

[TABLE]

Proposition 2

If the assumption C holds, then any of the above two selection procedures within the Kaczmarz’s iteration K satisfies (D2).

Proof.

Let us suppose that the conclusion of the proposition is not true. According to (5) it exists an index $i_{0}\in\{1,\dots,m\}$ and an integer $k_{0}\geq 1$ such that, in the selection procedure of the K algorithm iterations we have

[TABLE]

Therefore, the sequence $(x^{k})_{k\geq k_{0}}$ is generated by the K algorithm applied (only !) to the subsystem ${A}^{(i_{0})}{x}={b}^{(i_{0})}$. By the theory from [1] and [9], respectively, it results that

[TABLE]

hence

[TABLE]

which contradicts (12) and completes the proof. $\spadesuit$

In order to understand what means a condition like (12) we will analyse it in the particular case

[TABLE]

for which the system (1) is consistent for any $b\in I\!\!R^{m}$. In this case for any index $i$ the matrix ${A}^{(i)}$ is also full-row rank, and the system ${A}^{(i_{0})}{x}={b}^{(i_{0})}$ also consistent. We will arbitrary fix the index $i\in\{1,\dots,m\}$ and denote by $\tilde{A},\tilde{b},\tilde{x}_{LS}$ the elements ${A}^{(i)},{b}^{(i)},{x}^{(i)}_{LS}$, respectively. Moreover, we will analyse the opposite assumption of (12), namely

[TABLE]

What does this mean in terms of the matrix $A$ and right hand side $b$ ? For simplfying the presentation we will suppose that $i=m$ (this assumption is not too restrictive because it can be obtained by a row-permutation in $A$ and $b$, which does not affect the spectral properties of $A$ and the solution set $S(A;b)$). Because $A^{T}$ is overdetermined and full-column rank, there exist an $n\times n$ orthogonal matrix $Q$ and the QR decomposition

[TABLE]

[TABLE]

Therefore

[TABLE]

will be a QR decomposition for $\tilde{A}$, where

[TABLE]

Because $m\leq n$ and $A,\tilde{A}$ have full-row rank, we know that (see e.g. [8])

[TABLE]

hence

[TABLE]

In our hypothesis (17), and by using (18) and (20) we get from (21) the equality

[TABLE]

It can be easily shown that

[TABLE]

which together with the first equality in (22) gives

[TABLE]

[TABLE]

and therefore

[TABLE]

Eventually, we proved that if (17) holds (for $i=m$), then

[TABLE]

where the elements $c,\tilde{R}$ are from (18) and $\tilde{b}$ is the right hand side of the system ${A}^{(i_{0})}{x}={b}^{(i_{0})}$. But, also the converse holds, namely: if (24) is true with the above elements, then (17) holds (for $i=m$). This is true if we assume that in the QR decomposition (18) the diagonal elements satisfy $r_{ii}>0,\forall i$, which gives us the unicity of the factor $R$ in the QR decomposition.

Remark 1

Although the assumption $x^{0}=0$ in K is essential for the proof of Proposition 2, we conjecture that this result is stil true for a larger class of initial approximations $x^{0}$. Unfortunately we do not have for the moment a theoretical proof in this respect.

Acknowledgements. We would like to thanks to Prof. Yair Censor for his very helpful comments that have much improved the initial version of the paper.