Stability of the Kaczmarz Reconstruction for Stationary Sequences

TL;DR

This paper investigates the stability of the Kaczmarz algorithm for reconstructing vectors from inner products with stationary sequences, proposing a relaxed version that mitigates noise effects and ensures stable reconstruction in various settings.

Contribution

It introduces a relaxation of the Kaczmarz algorithm, akin to Abel summation, that stabilizes reconstruction in noisy environments for stationary sequences.

Findings

Relaxed Kaczmarz algorithm stabilizes reconstruction under noise.

Full noise removal achieved for certain function spaces.

Stabilizes Fourier series reconstruction in singular measures.

Abstract

The Kaczmarz algorithm is an iterative method to reconstruct an unknown vector $f$ from inner products $\langle f , \varphi_{n} \rangle $. We consider the problem of how additive noise affects the reconstruction under the assumption that $\{ \varphi_{n} \}$ form a stationary sequence. Unlike other reconstruction methods, such as frame reconstructions, the Kaczmarz reconstruction is unstable in the presence of noise. We show, however, that the reconstruction can be stabilized by relaxing the Kaczmarz algorithm; this relaxation corresponds to Abel summation when viewed as a reconstruction on the unit disc. We show, moreover, that for certain noise profiles, such as those that lie in $H^{\infty}(\mathbb{D})$ or certain subspaces of $H^{2}(\mathbb{D})$, the relaxed version of the Kaczmarz algorithm can fully remove the corruption by noise in the inner products. Using the spectral representation of stationary sequences, we show that our relaxed version of the Kaczmarz algorithm also stabilizes the reconstruction of Fourier series expansions in $L^2(\mu)$ when $\mu$ is singular.

Equations

$${\vec{x}}_{n+1}={\vec{x}}_n+\frac{r_i}{\Vert {\phi }_i{\Vert }^2}{\phi }_i.$$

$${\vec{x}}_{n+1}={\vec{x}}_n+\left(⟨\vec{x},{\phi }_{n+1}⟩+{ϵ}_{n+1}-⟨{\vec{x}}_n,{\phi }_{n+1}⟩\right){\phi }_{n+1}.$$

$$g_0$$

$$g_n$$

$${\vec{x}}_n=\sum _{i=0}^n⟨\vec{x},g_i⟩{\phi }_i$$

$$\Vert \vec{x}{\Vert }^2$$

$$\vec{x}$$

$$\sum _{j=i}^n{\alpha }_{n,j}⟨{\phi }_j,{\phi }_i⟩={\delta }_{in}$$

$$\underset{N\to \infty }{\lim }{\vec{x}}_N$$

$$=\sum _{n=0}^{\infty }\left(\sum _{j=0}^n{\alpha }_{n,j}⟨\vec{x},{\phi }_j⟩\right){\phi }_n+\sum _{n=0}^{\infty }\left(\sum _{j=0}^n{\alpha }_{n,j}{ϵ}_j\right){\phi }_n$$

$$=:{\vec{x}}_{\infty }+E_{ϵ}$$

$$\underset{N\to \infty }{\lim }\Vert {\vec{x}}_N-\vec{x}\Vert =\Vert E_{ϵ}\Vert .$$

$$\underset{r\to 1^{-}}{\lim }\sum _{n=0}^{\infty }{r}^n\left(\sum _{j=0}^n{\alpha }_{n,j}\left(⟨\vec{x},{\phi }_j⟩+{ϵ}_j\right)\right){\phi }_n=\vec{x}.$$

$$\underset{r\to 1^{-}}{\lim }f(r{e}^{2\pi ix})=f^{\star }(x)\text{ for some }{f}^{\star }\in L^2(m).$$

$$V_{\mu }f(z)=\frac{1}{{\mu }^{+}(z)}{\int }_0^1\frac{f(x)}{1-z{e}^{-2\pi ix}}d\mu (x)\text{ for }f\in L^2(\mu )$$

$${\mu }^{+}(z)={\int }_0^1\frac{1}{1-z{e}^{-2\pi ix}}d\mu (x).$$

$$\hat{f}(j)={\int }_0^{1}f(x)e^{-2\pi ijx}d\mu (x)$$

$${\mu }^{+}(z)=\sum _{n=0}^{\infty }\hat{\mu }(n)z^n.$$

$$\text{Re}\left(\frac{1+b(z)}{1-b(z)}\right)={\int }_0^1\frac{1-|z{|}^2}{|e^{2\pi ix}-z{|}^2}d\mu (x)$$

$${\mu }^{+}(z)=\frac{1}{1-b(z)}⟺b(z)=1-\frac{1}{{\mu }^{+}(z)}.$$

$$1-b(z)=\sum _{n=0}^{\infty }{\alpha }_n{z}^n⟺b(z)=-\sum _{n=1}^{\infty }{\alpha }_n{z}^n.$$

$$f(x)=\sum _{n=0}^{\infty }\left(\sum _{j=0}^n{\alpha }_{n-j}\hat{f}(j)\right)e^{2\pi inx},$$

$${\vec{x}}_N={\vec{x}}_{N-1}+c_N{\phi }_N-⟨{\vec{x}}_{N-1},{\phi }_N⟩{\phi }_N$$

$$\vec{x}_{N}=\sum_{n=0}^{N}\bigg{(}\sum_{j=0}^{n}\alpha_{n-j}c_{j}\bigg{)}\varphi_{n}.$$

$$\begin{array}{rl}{\vec{x}}_N& ={\vec{x}}_{N-1}+c_N{\phi }_N-⟨{\vec{x}}_{N-1},{\phi }_N⟩{\phi }_N\\ & =\sum _{n=0}^N\left(\sum _{j=0}^n{\alpha }_{n-j}{c}_j\right){\phi }_n.\end{array}$$

$$\begin{split}\vec{x}_{N+1}&=\vec{x}_{N}+c_{N+1}\varphi_{N+1}-\langle\vec{x}_{N},\varphi_{N+1}\rangle\varphi_{N+1}\\ &=\vec{x}_{N}+c_{N+1}\varphi_{N+1}-\bigg{\langle}\sum_{n=0}^{N}\left(\sum_{j=0}^{n}\alpha_{n-j}c_{j}\right)\varphi_{n},\varphi_{N+1}\bigg{\rangle}\varphi_{N+1}\\ &=\vec{x}_{N}+c_{N+1}\varphi_{N+1}-\sum_{n=0}^{N}\left(\sum_{j=0}^{n}\alpha_{n-j}c_{j}\right)\langle\varphi_{n},\varphi_{N+1}\rangle\varphi_{N+1}.\end{split}$$

$${\vec{x}}_{N+1}=\sum _{n=0}^N\left(\sum _{j=0}^n{\alpha }_{n-j}{c}_j\right){\phi }_n+c_{N+1}{\phi }_{N+1}-\sum _{n=0}^N\left(\sum _{j=0}^n{\alpha }_{n-j}{c}_j\right)⟨{\phi }_n,{\phi }_{N+1}⟩{\phi }_{N+1}$$

$$-\sum _{n=0}^N\left(\sum _{j=0}^n{\alpha }_{n-j}{c}_j\right)⟨{\phi }_n,{\phi }_{N+1}⟩=\sum _{j=0}^N{\alpha }_{N-j}{c}_j$$

$${\vec{x}}_{N+1}$$

$$=\sum _{n=0}^{N+1}\left(\sum _{j=0}^n{\alpha }_{n-j}{c}_j\right){\phi }_n$$

$$\vec{x}=\sum ⟨\vec{x},g_n⟩g_n=\sum _{n=0}^{\infty }\left(\sum _{j=0}^n{\alpha }_{n-j}⟨\vec{x},{\phi }_j⟩\right)g_n.$$

$$E_{ϵ,g}=\sum _{n=0}^{\infty }\left(\sum _{j=0}^n{\alpha }_{n-j}{ϵ}_j\right)g_n$$

$$\sum _{n=0}^{\infty }\left(\sum _{j=0}^n{\alpha }_{n-j}{ϵ}_j\right)z^n=(1-b(z))\epsilon (z)\in H^2(\mathbb{D}),$$

Full text

1. Introduction

The Kaczmarz algorithm [Kac37] is an iterative procedure for solving a system of linear equations. We formulate the system of linear equations as inner products: given vectors $\{\varphi_{0},\dots,\varphi_{N-1}\}\subset\mathbb{C}^{d}$, find $\vec{x}\in\mathbb{C}^{d}$ such that $\langle\vec{x},\varphi_{j}\rangle=b_{j}$. The Kaczmarz algorithm proceeds as follows: given a solution guess $\vec{x}_{n}$ and an equation number $i$, we calculate $r_{i}=b_{i}-\langle\vec{x}_{n},\varphi_{i}\rangle$ (the residual for equation $i$), and define

[TABLE]

This makes the residual of $\vec{x}_{n+1}$ in equation $i$ equal to 0. We then iterate repeatedly through all equations (i.e. $\lim_{n\to\infty}\vec{x}_{n}$ where $n+1\equiv i\mod N$). Kaczmarz proved that if $\{\varphi_{0},\dots,\varphi_{N-1}\}$ span $\mathbb{C}^{d}$ and the equations $\langle\vec{x},\varphi_{j}\rangle=b_{j}$ are consistent, then $\vec{x}_{n}\to\vec{x}$. If the system is inconsistent, then the relaxed version of the Kaczmarz algorithm can be utilized to obtain the least-squares solution (see Subsection 2.5 for details).

The Kaczmarz algorithm can also be used to reconstruct a vector (signal) in an infinite dimensional Hilbert space $H$ from linear measurements. For a sequence of unit vectors $\{\varphi_{n}\}_{n=0}^{\infty}\subset H$ ($H$ is of any dimension, finite or infinite), the sequence is said to be effective if for every $\vec{x}\in H$, given the data $\{\langle\vec{x},\varphi_{n}\rangle\}_{n=0}^{\infty}$, the sequence of approximations $\vec{x}_{n}$ to $\vec{x}$ given by the Kaczmarz algorithm as defined in Equation (1) has the property that $\lim_{n\to\infty}\|\vec{x}-\vec{x}_{n}\|=0$.

In an infinite dimensional Hilbert space, the question of when a sequence of vectors is an effective sequence is subtle. There are two main existing results concerning the effectivity of sequences. The full characterization is in [HS05] (see also [Szw07, CT13]) and is given in terms of when a certain infinite matrix defines a partial isometry on $\ell^{2}(\mathbb{N}_{0})$. Another characterization concerns stationary sequences, which is our focus here. A sequence $\{\varphi_{n}\}_{n=0}^{\infty}$ is stationary provided for every $n,m,k\in\mathbb{N}_{0}$, $\langle\varphi_{n+k},\varphi_{m+k}\rangle=\langle\varphi_{n},\varphi_{m}\rangle$. Kwapien and Mycielski [KM01] prove the following: the stationary sequence $\{\varphi_{n}\}_{n=0}^{\infty}\subset H$ is effective if and only if it is linearly dense and the spectral measure of the sequence is either Lebesgue measure or purely singular.

We consider the problem of stability, in the presence of additive noise, of the Kaczmarz reconstruction for stationary sequences which are effective:

Problem 1**.**

Suppose $\{\varphi_{n}\}_{n=0}^{\infty}\subset H$ is a stationary, effective sequence. Given the data $\{\langle\vec{x},\varphi_{n}\rangle+\epsilon_{n}\}_{n=0}^{\infty}$, does the Kaczmarz algorithm provide a reconstruction of $\vec{x}$? Explicitly, the update becomes

[TABLE]

Does the sequence $\{\vec{x}_{n}\}$ of approximations converge? If so, what is the error in the reconstruction contributed by the noise $\{\epsilon_{n}\}_{n=0}^{\infty}$: $\lim_{n\to\infty}\|\vec{x}-\vec{x}_{n}\|$?

As we will demonstrate, Problem 1 can be reformulated in terms of analytic functions on $\mathbb{D}$. In particular, the characterization given by Kwapien and Mycielski uses the Herglotz Representation, which demonstrates that the Poisson transform induces a one-to-one correspondence between singular measures on $\mathbb{T}$ and inner functions on $\mathbb{D}$. In [HW17], the proof in [KM01] is restated in terms of single variable operator theory: Beurling’s theorem on the shift invariant subspaces of the Hardy space $H^{2}(\mathbb{D})$ [Beu48], the spectral theorem of rank-one perturbations of the shift [Cla72], and the Normalized Cauchy Transform [Pol93]. Subsequent work [HJW18] demonstrated that the Kaczmarz algorithm can be used to construct reproducing kernels within the Hardy space that have prescribed boundary behavior.

The boundary behavior arises from the spectral representation of stationary sequences (see Subsection 1.2). From this spectral representation, we are able to translate the problem onto $\mathbb{D}$. Our main results will be to show that the reconstruction with noisy measurements in Equation (2) can be stabilized by employing Abel summation, e.g. Theorems 1, 3 and 4, which corresponds to the boundary data of an analytic function on $\mathbb{D}$. We will describe how Abel summation can be given in terms of an augmented Kaczmarz algorithm in Theorem 5. We will then be able to use powerful results concerning the boundary behavior of the model subspaces of $H^{2}(\mathbb{D})$ (i.e., the subspaces which are backward shift invariant [Beu48, Cla72, Ale89, Pol93]) and the machinery of sub-Hardy Hilbert spaces [dBR66, Sar94]. We note that the augmented Kaczmarz algorithm shares some similarities to the relaxed Kaczmarz algorithm: both employ a relaxation parameter, and it is the limiting behavior of the relaxation parameter that yields the stability, in both cases, of the reconstructions in the presence of noise. However, we will demonstrate in Proposition 7 that they are not equivalent methods.

We note here that there are strong connections between the Kaczmarz algorithm and frame theory [KM01, Szw07, CT13, CP16, HJW19]. Frames [DS52, DGM86, Cas00] have good stability properties with respect to additive noise: if the noise is of finite energy (i.e. in $\ell^{2}$), then the frame reconstruction converges, and the error is proportional to the $\ell^{2}$-norm of the noise. However, requiring the vectors $\{\varphi_{n}\}$ to form a frame is a strong condition, and in general, stationary sequences do not form frames [HW17]. Moreover, the Kaczmarz reconstruction has broad interest in applications [GBH70, EHL81, SV09, NZZ15, NSW16, HKW19], and yields constructive proofs of the existence of Fourier series expansions for singular measures ([Pol93, HW17], see also Subsection 1.2).

1.1. Series Representation of the Kaczmarz Algorithm

The Kaczmarz reconstruction given in Equation (1) can be expressed as a series, provided the sequence $\{\vec{x}_{n}\}$ of approximations converges. Following [KM01], we define the auxiliary sequence $\{g_{n}\}_{n=0}^{\infty}$ by setting

[TABLE]

It was shown in [KM01] that

[TABLE]

and further that $\{\varphi_{n}\}$ is effective if and only if $\{g_{n}\}$ is a Parseval frame, meaning

[TABLE]

We may also define (as in [HS05]) the doubly-indexed sequence $\{\alpha_{n,j}:n\in\mathbb{N}_{0},\ j=0,\dots,n\}$ satisfying the property

[TABLE]

for all $n\in\mathbb{N}$ and $i\leq n$.

We have that

[TABLE]

and assuming both series on the right-hand side of Equation (5) converge. The identity in Equation (4) is a consequence of Lemma 1 below. If the sequence $\{\varphi_{n}\}$ is effective, then the first series converges, and $\vec{x}_{\infty}=\vec{x}$. Consequently:

Proposition 1**.**

Suppose the sequence $\{\varphi_{n}\}$ is an effective sequence. The sequence of approximations $\{\vec{x}_{N}\}$ (with noise) converges if and only if the series $E_{\epsilon}$ converges. Moreover, the reconstruction error is given by

[TABLE]

We will show in Subsection 2.2 that under rather restrictive assumptions on the noise $\{\epsilon_{n}\}$, we can obtain convergence of the series in Equation (5), and can bound the norm on $E_{\epsilon}$. However, in general, a fairly weak assumption on $\{\epsilon_{n}\}$ is not sufficient to guarantee convergence (Proposition 2). Then, in Subsection 2.3, we will show that be utilizing Abel summation rather than standard summation, we can obtain stability in the Kaczmarz reconstruction under much weaker assumptions. Indeed, we state a general theorem that illustrates our main results; see Theorems 3 and 4 for the precise statements and proofs.

Theorem 1**.**

Suppose $\{\varphi_{n}\}_{n=0}^{\infty}$ is a stationary sequence which is effective, with singular spectral measure. Under appropriate conditions on $\{\epsilon_{n}\}$ (e.g. $\sum\epsilon_{n}z^{n}\in H^{\infty}(\mathbb{D})$), we have

[TABLE]

In other words, by using Abel summation, the error term $E_{\epsilon}=0$.

We will also show in Subsection 2.4 that the Abel summation of the series in Equation (7) can be represented as an augmented Kaczmarz algorithm with relaxation parameter $r$.

1.2. Spectral Representation of the Kaczmarz Algorithm

When the effective sequence $\{\varphi_{n}\}$ is stationary, it possesses a spectral measure $\mu$ on $[0,1]$ as a result of Bochner’s theorem. The mapping $\varphi_{n}\mapsto e^{2\pi inx}$ then extends to an isometry from $H$ to $L^{2}(\mu)$; moreover, this isometry is a unitary when $\mu$ is singular. The Kwapien-Mycielski theorem states that if the sequence $\{\varphi_{n}\}$ is stationary and has dense span in $H$, then the following conditions are equivalent: (i) the sequence is effective; (ii) the spectral measure $\mu$ is either Lebesgue measure or purely singular.

Recent results elucidated the existence of a Fourier series for singular measures [HW17], wherein the exponentials of positive-integer frequencies form a linearly dense collection of vectors for $L^{2}(\mathbb{T},\mu)$, where $\mu$ is a singular measure and $\mathbb{T}$ is the one-dimensional unit circle of the complex plane, parameterized by $e^{2\pi ix}$. (For the rest of this paper, we use the notation $L^{2}(\mu)=L^{2}(\mathbb{T},\mu)$, with the understanding that all functions of a single real variable are defined to be periodic.)

It is well known that the Hardy space on the unit disc, $H^{2}(\mathbb{D})$ (notated as simply $H^{2}$), and $L^{2}_{+}(m)=\{f:f\in L^{2}(m)\text{ and }f(x)=\sum_{n=0}^{\infty}c_{n}e^{i2\pi nx}\text{ for some }c_{n}\in\ell^{2}\}$ (Lebesgue measure $m$) are isomorphic Hilbert spaces. This isomorphism is given by a change of variables $z\mapsto e^{2\pi ix}$ in the power series representation of $H^{2}$ functions. This isomorphism may also be thought of as the values the Hardy space function takes on the boundary. That is, for $f\in H^{2}$, and $z=re^{2\pi ix}$

[TABLE]

We call $f^{\star}$ the boundary function of $f$. With respect to Lebesgue measure, for each $f\in H^{2}$, there is a guaranteed $f^{\star}\in L^{2}(m)$, as the two spaces are isomorphic. Subtleties occur when dealing with singular measures, however.

A similar isomorphism may be given between $L^{2}(\mu)$ and some subspace of $H^{2}$ [HJW16]. This subspace is most-easily described as the range of the normalized Cauchy transform:

[TABLE]

where $\mu^{+}(z)$ is the unnormalized Cauchy transform of the constant function $1$:

[TABLE]

Since we seek the Fourier series for this singular measure, we denote by

[TABLE]

the Fourier-Stieltjes transform of $f$. We call the sequence $\{\widehat{f}(j)\}$ the Fourier moments of $f$. The Fourier moments of the constant function $\mathbf{1}(z)$ are notated as $\widehat{\mu}(j)$. These have the property that

[TABLE]

We also consider the Herglotz representation of singular measures and their corresponding inner functions:

[TABLE]

where the function $b\in H^{2}$ is unique for each singular measure $\mu$. We refer to the function $b$ as corresponding to the singular measure $\mu$; it has the properties that $b(0)=0$, $\lim_{r\to 1^{-}}|b(re^{2\pi ix})|=1$ $m-$almost everywhere, and $\lim_{r\to 1^{-}}b(re^{2\pi ix})=1$ $\mu-$almost everywhere [Ale89, Her16]. Additionally, we have

[TABLE]

For $\varphi_{n}=e^{2\pi in}$ we now recognize

[TABLE]

The following result from [HW17] regarding the sequence of Fourier moments summarizes much of the above:

Theorem A**.**

Assume $\mu$ is a singular Borel probability measure on $[0,1)$ and $\{\alpha_{j}\}$ is the corresponding sequence of coefficients given in Equation (8). Then

[TABLE]

where the convergence is in the $L^{2}(\mu)$-norm.

2. Main Results

We assume throughout this section that $\{\varphi_{n}\}_{n=0}^{\infty}$ is a linearly dense stationary sequence with singular spectral measure $\mu$. Thus, the sequence is an effective sequence. Moreover, we assume throughout that $b$ is the inner function corresponding to $\mu$ via the Herglotz representation, and thus the Normalized Cauchy Transform $V_{\mu}:L^{2}(\mu)\to H^{2}\ominus bH^{2}$ is a unitary transformation. We will also repeatedly refer to the sequence $\{\alpha_{n}\}$ as defined in Equation (8).

Let us note that if the spectral measure $\mu$ is neither singular nor Lebesgue, then the sequence is not effective [KM01] and so the question of stability is not well formulated in this case. If the spectral measure is Lebesgue measure, then the sequence $\{\varphi_{n}\}$ is an orthonormal basis, so we immediately obtain that the reconstruction error is $\|E_{\epsilon}\|^{2}=\sum_{n=0}^{\infty}|\epsilon_{n}|^{2}$.

Since the sequence $\{\varphi_{n}\}\subset H$ and $\{e^{2\pi inx}\}\subset L^{2}(\mu)$ are unitarily equivalent, we will pass between them interchangeably. For the noise sequence $\{\epsilon_{n}\}$ we define the formal series $\varepsilon(z)=\sum_{n=0}^{\infty}\epsilon_{n}z^{n}$; throughout this section we assume this series has radius of convergence at least $1$.

2.1. The Kaczmarz Algorithm from Arbitrary Inputs

The following provides the justification of the Kaczmarz reconstruction as a series representation.

Lemma 1**.**

For $\{c_{n}\}_{n=0}^{\infty}\subset\mathbb{C}$ we have by the Kaczmarz update:

[TABLE]

but also:

[TABLE]

Hence, the Kaczmarz reconstruction from an arbitrary sequence can be represented as a series.

Proof.

The proof follows by induction. The initial case is trivial. Thus suppose

[TABLE]

By Kaczmarz we have

[TABLE]

Hence,

[TABLE]

We note the identity (given by the definition of $\{\alpha_{k}\}_{k=0}^{\infty}$ in Equation (8))

[TABLE]

Thus we may write

[TABLE]

∎

2.2. Reconstruction using Standard Summation

We begin by considering the reconstruction with noise using the auxiliary sequence $\{g_{n}\}$ as in Equation (3). This does not fit within the iterative framework of the Kaczmarz algorithm, but we present it here to illustrate the distinctions between frame reconstructions and the Kaczmarz reconstruction. In the noiseless case, we have by the Parseval frame condition on the auxiliary sequence that

[TABLE]

Therefore, the error introduced by the noise is

[TABLE]

provided the series converges. If $\{\epsilon_{n}\}\in\ell^{2}$, then $\{\sum_{j=0}^{n}\alpha_{n-j}\epsilon_{j}\}\in\ell^{2}$ also, since

[TABLE]

and the norm satisfies the trivial estimate $\|(1-b(z))\varepsilon(z)\|\leq 2\|\varepsilon(z)\|$. Thus, we obtain:

Lemma 2**.**

If $\{\epsilon_{n}\}_{n=0}^{\infty}\in\ell^{2}$, then $\vec{y}=\sum_{n=0}^{\infty}\left(\sum_{j=0}^{n}\alpha_{n-j}(\langle\vec{x},\varphi_{j}\rangle+\epsilon_{j})\right)g_{n}$ converges, and

[TABLE]

The following is well-known concerning pseudocontinuable functions [Cla72, Nik86, Ale89, Pol93], though we include the proof in the Appendix.

Proposition 2**.**

For a singular measure $\mu$, there exists a sequence $\{\epsilon_{n}\}\in\ell^{2}$ such that the series $\sum\epsilon_{n}e^{2\pi inx}$ diverges in $L^{2}(\mu)$. Moreover, there exists a sequence $\{\epsilon_{n}\}\in\ell^{2}$ such that $\sum_{n=0}^{\infty}\left(\sum_{j=0}^{n}\alpha_{n-j}\epsilon_{j}\right)e^{2\pi inx}$ diverges in $L^{2}(\mu)$.

In the context of the stability of frames, the assumption on the noise is $\{\epsilon_{n}\}\in\ell^{2}$. As a consequence of Proposition 2, however, we must make alternative assumptions on the noise to obtain convergence, as well as stability, of the Kaczmarz reconstruction in Equation (5). To help determine when the series $E_{\epsilon}$ converges, we utilize known facts concerning the convergence of Fourier series in $L^{2}(\mu)$. The foundational result in this regard is: if $f\in H^{2}\ominus bH^{2}$, then its Fourier series converges in $L^{2}(\mu)$ [Pol93, HW17]. The following is a known extension of this result, though we include the proof in the Appendix.

Lemma 3**.**

If $f(z)\in H^{2}\ominus b^{K}H^{2}$ for some $K\in\mathbb{N}$, then the Fourier series for $f$ converges in $L^{2}(\mu)$, i.e. if $f(z)=\sum a_{n}z^{n}$, then $\sum a_{n}e^{2\pi inx}$ converges in $L^{2}(\mu)$.

We can now use this to establish our main result concerning convergence and approximation of the error using the standard Kaczmarz reconstruction.

Theorem 2**.**

Suppose $\{\varphi_{n}\}\subset H$ is a stationary sequence with singular spectral measure $\mu$. Suppose $\vec{x}\in H$, and suppose the noise $\{\epsilon_{n}\}$ is such that $(1-b)\varepsilon\in H^{2}\ominus b^{K}H^{2}$ for some $K\in\mathbb{N}$. Then, the reconstruction of $\vec{x}$ from the noisy data $\{\langle\vec{x},\varphi_{n}\rangle+\epsilon_{n}\}$ satisfies the following:

the series $\vec{x}_{\infty}$ and $E_{\epsilon}$ as in Equation (4) both converge; 2. 2.

[TABLE]

Proof.

Since the spectral measure of $\{\varphi_{n}\}$ is singular, it is an effective sequence, and thus $\vec{x}_{\infty}=\vec{x}$.

By identifying $\varphi_{n}$ with $z^{n}$ in Equation (5), we identify $E_{\epsilon}$ with $(1-b(z))\varepsilon(z)$ via Equation (11). Using the Wold decomposition of $H^{2}$ ([HJW18]), we write

[TABLE]

where $f_{j}\in H^{2}\ominus bH^{2}$. By Lemma 3, we have that the Fourier series of $(1-b(z))\varepsilon(z)$ converges in $L^{2}(\mu)$, so $E_{\epsilon}$ converges in $H$, and we obtain the norm estimate

[TABLE]

We also have the norm equality

[TABLE]

Combining these yields the inequality in (12). ∎

We can refine this result slightly by making extra assumptions on the $\{\epsilon_{n}\}$. For example, in the context of frames, we can quantify the error caused by the noise in terms of the frame bounds; but we can also quantify the error in terms of the cancellation in the frame expansion. Indeed, for a frame $\{h_{n}\}\in H$ with analysis operator $\Theta_{h}$, if $\{\epsilon_{n}\}$ has the property that $\Theta_{h}^{*}(\epsilon_{n})=0$, then the reconstruction error is [math]. This occurs exactly when $\{\epsilon_{n}\}$ is orthogonal to the range of $\Theta_{h}$. Reconsidering the reconstruction using the auxiliary sequence $\{g_{n}\}$, we obtain the following refinement.

Proposition 3**.**

If $\{\epsilon_{n}\}$ is such that $\varepsilon(z)\in bH^{2}$, then $E_{\epsilon,g}=0$.

Proof.

For the frame $\{g_{n}\}$, the range of its analysis operator $\Theta_{g}(H)=H^{2}\ominus bH^{2}$ ([HW17]). Thus, the reconstruction using $g_{n}$ in Equation (10) is such that the coefficients satisfy $(1-b(z))\varepsilon(z)\in bH^{2}$, and thus sums to [math]. ∎

We cannot obtain such a clean statement for reconstruction from $\{\varphi_{n}\}$, since the coefficients $\{\langle\vec{x},\varphi_{n}\rangle\}$ are not in a natural Hilbert space, and so there is not a natural analogue for the kernel of the “synthesis” operator. However, for reconstruction using the standard Kaczmarz algorithm, we can use the boundary functions to obtain a similar result. Indeed, if $E_{\epsilon}$ converges, and $\left[(1-b(z))\varepsilon(z)\right]^{\star}=0$, we immediately obtain $E_{\epsilon}=0$.

Proposition 4**.**

Suppose $\{\epsilon_{n}\}$ is such that $(1-b)\varepsilon\in H^{2}\ominus b^{K}H^{2}$ for some $K\in\mathbb{N}$. Then $E_{\epsilon}=0$ if and only if and the Wold decomposition of $(1-b)\varepsilon$ as in Equation (13) satisfies $\sum_{j}f_{j}=0$.

Proof.

This follows from the previous observation and the fact that the boundary

[TABLE]

if and only if $\sum_{j}f_{j}=0$ since $f_{j}\in H^{2}\ominus bH^{2}$. ∎

2.3. Reconstruction using Abel Summation

While the results of Theorem 2 allowed us to bound the norm of the reconstruction error using standard summation methods, in analogy with the relaxed Kaczmarz algorithm, we now utilize the stabilizing power of Abel summation.

Therefore, instead of the summation in Equation (4), we consider the Abel sum:

[TABLE]

provided the limit exists. We will see that in several different scenarios on the noise $\{\epsilon_{n}\}$, Abel summation in fact completely removes the noise from the reconstruction.

We first note that for $r\in(0,1)$, the convergence of the series in Equation (14) is guaranteed under minimal assumptions on $\{\epsilon_{n}\}$ (though not the existence of the limit).

Lemma 4**.**

Suppose $\{\varphi_{n}\}_{n=0}^{\infty}\subset H$ is a stationary sequence with singular spectral measure $\mu$. For $r\in(0,1)$, the series

[TABLE]

converges in $H$ provided $\limsup_{n\to\infty}\sqrt[n]{|\epsilon_{n}|}\leq 1$. Consequently, the series in Equation (14) converges.

Proof.

If we replace $\varphi_{n}$ with $e^{2\pi inx}$, the series in Equation (15) becomes:

[TABLE]

for $z\in\mathbb{D}$. Our assumption on $\{\epsilon_{n}\}$ implies the right-hand side of Equation (16) is analytic on $\mathbb{D}$. It now follows from uniform convergence that the series in Equation (15) converges in $L^{2}(\mu)$.

Since $\{\varphi_{n}\}$ is effective in $H$, we have

[TABLE]

∎

Recall that for a holomorphic function $f$ on the disc, we say that $f$ has a $L^{2}(\mu)$-boundary if there exists a $f^{\star}\in L^{2}(\mu)$ such that

[TABLE]

with convergence in norm. The following follows immediately from this definition.

Lemma 5**.**

Suppose $\{\varphi_{n}\}_{n=0}^{\infty}\subset H$ is a stationary sequence with singular spectral measure $\mu$. The limit in Equation (14) exists if and only if $\left[(1-b(z))\varepsilon(z)\right]^{\star}\in L^{2}(\mu)$. Moreover, the reconstruction error is

[TABLE]

We would like to have assumptions on $\{\epsilon_{n}\}$ that guarantee convergence of the Abel sum. This is easiest when the Abel sum of the noise term converges to [math]. We present several different assumptions that yield this convergence in Equation (14); in these cases the reconstruction error is [math].

Theorem 3**.**

Suppose $\{\varphi_{n}\}\subset H$ is a stationary sequence with singular spectral measure $\mu$. Suppose $\vec{x}\in H$, and suppose the noise $\{\epsilon_{n}\}$ is such that $\varepsilon^{\star}\in L^{2}(\mu)$ exists.

Then the following conditions hold:

[TABLE] 2. 2.

consequently, the limit in Equation (14) is $\vec{x}$.

Proof.

As before, we express the sum in Equation (17) in its spectral representation by replacing $\varphi_{n}$ with $e^{2\pi inx}$. In so doing, we obtain

[TABLE]

The integral goes to 0 by Lebesgue’s Theorem, since $|b(z)|<1$,

[TABLE]

in measure with respect to $\mu$, and $\varepsilon(re^{2\pi ix})\to\varepsilon^{\star}$ in the $L^{2}(\mu)$-norm. ∎

Corollary 1**.**

Suppose $\varepsilon^{\star}\in L^{p}(\mu)$ for $2\leq p\leq\infty$. Then

[TABLE]

The assumption that $\varepsilon^{\star}$ exists is a difficult assumption to check. We turn now to assumptions that are easier to verify. To do so, we define a maximal function

[TABLE]

Note that for $\varepsilon(z)\in H^{\infty}(\mathbb{D})$, $M_{\varepsilon}$ is essentially bounded with respect to $\mu$.

Lemma 6**.**

Suppose $\{\varphi_{n}\}\subset H$ is a stationary sequence with singular spectral measure $\mu$. Suppose $\vec{x}\in H$, and suppose the noise $\{\epsilon_{n}\}$ is such that $M_{\varepsilon}\in L^{2}(\mu)$. Then the following conditions hold:

[TABLE] 2. 2.

consequently, the limit in Equation (14) is $\vec{x}$.

Proof.

For $r\in(0,1)$, we have

[TABLE]

Thus, the norm goes to 0 as $r\to 1^{-}$ by Lebesgue’s Theorem. ∎

Theorem 4**.**

Suppose $\{\varphi_{n}\}\subset H$ is a stationary sequence with singular spectral measure $\mu$. Suppose $\vec{x}\in H$. If $\varepsilon(z)$ satisfies any of the following conditions, then the conclusions of Lemma 6 hold:

$\varepsilon(z)\in H^{\infty}$; 2. 2.

$M_{\varepsilon}\in L^{\infty}(\mu)$; 3. 3.

for some $q>1$,

[TABLE]

Proof.

Items 1. and 2. follow immediately from Lemma 6. We prove Item 3. as follows.

As before, we have $1-b(re^{2\pi ix})\to 0$ boundedly in measure, so

[TABLE]

for any $1<p<\infty$. Therefore by Hölder’s Inequality,

[TABLE]

∎

The results stated in Theorems 3 and 4 are our most powerful applications of Abel summation, which state that not only is the series representation of the Kaczmarz algorithm as given in Equation (4) Abel summable, but also Abel summation removes the corruption due to the noise $\{\epsilon_{n}\}$. In general, we would like weaker criteria on $\{\epsilon_{n}\}$ for which the series is Abel summable, with the reconstruction error bounded. This is more difficult for at least 2 reasons: 1) there are no known results that we are aware of that guarantee convergence of the Abel summation of Equation (4)–except that the series converges or the tautology in Lemma 5; 2) as we’ve demonstrated earlier, there can not in general be a bound on the error term based on the $\ell^{2}$-norm of $\{\epsilon_{n}\}$. However, we can bound the error in two special cases. The first is an immediate consequence of Theorem 2.

Proposition 5**.**

Suppose $(1-b)\varepsilon\in H^{2}\ominus b^{K}H^{2}$ for some $K\in\mathbb{N}$. Then for some constant $c_{K}$ dependent on $K$, we have that \|\big{(}(1-b)\varepsilon\big{)}^{\star}\|_{L^{2}(\mu)}\leq c_{K}\|(1-b)\varepsilon\|_{H^{2}}. Therefore,

[TABLE]

Proposition 6**.**

Suppose $\{\epsilon_{n}\}$ is such that

[TABLE]

Then

[TABLE]

In particular, if

[TABLE]

Then Inequality (19) holds with $2C$ on the right hand side.

Note that we are not concluding here that the series is Abel summable.

Proof.

We have the estimate

[TABLE]

with the (20) converging to [math] and (21) bounded by $B$. ∎

2.4. Abel Summation and the Augmented Kaczmarz Algorithm

The series and spectral representations of the Kaczmarz algorithm have provided powerful methods for determining the convergence and reconstruction errors in the presence of noise. Indeed, as we have seen, Abel summation eliminates the corruption of the noise in many cases. However, we would like to express the Abel summation method in the form of a Kaczmarz update, which we do so here in the form of an augmented iteration. An augmented Kaczmarz algorithm was introduced in [ACH*+*19], for the purpose of dualizing the Kaczmarz reconstruction. We similarly define the augmented Kaczmarz algorithm for expressing the Abel summation in Equation (15), Lemma 4, in terms of the Kaczmarz update. Given $\{c_{n}\}_{n=0}^{\infty}$, define the update as

[TABLE]

Theorem 5**.**

For a fixed $r\in(0,1)$, the $N^{th}$ step in the augmented Kaczmarz algorithm corresponds exactly with the partial Abel sum:

[TABLE]

Proof.

Again we proceed by induction with a trivial first case. Thus we assume

[TABLE]

Next note that by Lemma 1

[TABLE]

since $\varphi_{N}$ is assumed to be normalized. Hence we have

[TABLE]

∎

Corollary 2**.**

Let $r\in(0,1)$ be fixed, and suppose the hypotheses of Lemma 4 are met. Then the sequence $\{\vec{y}_{n}\}$ as defined in Equation (22) converges for $c_{n}=\langle f,\varphi_{n}\rangle+\epsilon_{n}$.

Proof.

This follows from Lemma 4 and Theorem 5. ∎

2.5. The Relaxed Kaczmarz Algorithm and the Augmented Kaczmarz Algorithm

We recall that the relaxed Kaczmarz algorithm utilizes a relaxation parameter $\omega$, typically with $\omega\in(0,2)$. For a periodic sequence (of unit vectors) $\{\psi_{n}\}_{n=1}^{\infty}$ with period $N$, we can construct a sequence of approximations $\vec{y}_{m}$ from the data $\{\langle\vec{x},\psi_{n}\rangle\}_{n=1}^{N}$ as follows:

[TABLE]

As in Lemma 1, we can replace $\langle\vec{x},\psi_{k}\rangle$ with a (periodic) sequence $\{c_{n}\}$ in the update (23). It can be shown that $\{\vec{y}_{m}\}$ is a convergent sequence [Tan71, Nat86]; this limit depends on $\omega$. If we denote

[TABLE]

then we consider

[TABLE]

It can also be shown that this limit exists [Tan71, Nat86], and moreover, is the least-squares solution, i.e. $\vec{z}_{0}$ minimizes the quantity

[TABLE]

We can think of the parameter $r$ in Abel summation as a relaxation parameter. Then the limiting processes $\omega\to 0$ and $r\to 1$ are analogous; however, these are not equivalent.

Proposition 7**.**

The augmented Kaczmarz algorithm is equal to the relaxed Kaczmarz algorithm if and only if $\{\omega_{n}\}=\{r^{n}\}$ for fixed $r\in(0,1)$, and $\{\varphi_{n}\}$ is an orthogonal set.

Proof.

Observe that in the augmented Kaczmarz algorithm we may represent

[TABLE]

where we note that each $\varphi_{N}$ has the augmented coefficient $r^{N}$ and no other powers of $r$. That is, $\vec{y}_{N}=k_{N}(r^{N}\varphi_{N})+\text{lower order terms}$.

We also see that in the relaxed Kaczmarz algorithm

[TABLE]

such that each $\varphi_{N}$ has the relaxed coefficient $\omega_{N}$. That is, $\vec{x}_{N}=\kappa_{N}(\omega^{N}\varphi_{N})+\text{lower order terms}$. As such, if equations (24) and (25) were equivalent, we would have that $\omega_{j}=r^{j}$ for all $j$.

Rewriting equation (25) recursively dependent on the $(N-2)^{th}$ term, however, we receive

[TABLE]

for $K_{1},K_{2}$ constants dependent on $\{c_{n}\},\{\varphi_{n}\}$. Clearly, for equation (26) to equal equation (25), then $K_{2}=0$, which would imply that

[TABLE]

since $\{c_{n}\}$ is arbitrary.

We could repeat this procedure to write $\vec{x}_{N}$ dependent on $\vec{x}_{N-3}$ in which case we would have that $\langle\varphi_{N-2},\varphi_{N}\rangle=0$, and so on. We thus recover that $\langle\varphi_{N-j},\varphi_{N}\rangle=0$ for all $j=1,...,N$.

Hence for the choice $\omega_{j}=r^{j}$, the augmented Kaczmarz algorithm is equal to the relaxed Kaczmarz algorithm if and only if $\{\varphi_{n}\}$ is an orthogonal set. ∎

Remark 1*.*

Recall that in the typical relaxed Kaczmarz algorithm, $\omega$ is fixed. Thus it would be impossible for $\{\omega_{n}\}=\{r^{n}\}$ for $r\in(0,1)$, implying that the relaxed Kaczmarz algorithm with any fixed parameter $\omega$ is never equal to the augmented Kaczmarz algorithm. In practice, $\omega$ is allowed to vary with the iteration $n$, though establishing convergence results in this case has been difficult to do.

3. Stability via Truncated Sequences

We consider here the situation that we make weak or no assumptions on the noise $\{\epsilon_{n}\}$. Our results here also apply when the sequence $\{\varphi_{n}\}$ is not stationary (though we still assume effective). Our basic idea is to truncate the sequence to $\{\varphi_{n}\}_{n=0}^{N}$, then periodize and reconstruct using the relaxed Kaczmarz algorithm as in Equation (23). The relaxed Kaczmarz algorithm provides stability in the presence of noise when the sequence $\{\varphi_{n}\}$ is periodic.

For $N\in\mathbb{N}$, define the periodic sequence $\{\varphi_{n}^{(N)}\}$ given by:

[TABLE]

Given the data $\{\langle\vec{x},\varphi_{n}\rangle\}_{n=1}^{N}$, we apply the standard periodic Kaczmarz algorithm as in Equation (1) to obtain

[TABLE]

where $\mathbb{P}_{N}$ is the projection onto the span of $\{\varphi_{1},\dots,\varphi_{N}\}$. This is a consequence of the fact that periodic sequences are effective for their span. The simple but crucial observation here is the following: $\lim_{N\to\infty}\mathbb{P}_{N}\vec{x}=\vec{x}$ since $\{\varphi_{n}\}_{n=0}^{\infty}$ has dense span in $H$. What we will see here is a trade-off between noise removal and accuracy of the reconstruction were there no noise.

3.1. Truncation of Effective Sequences in the Presence of Noise

We consider the situtation of Problem 1, though we do not require the sequence $\{\varphi_{n}\}$ to be stationary. From now on, we let $\{\tilde{x}_{n}\}_{n\in\mathbb{N}}$ be the sequence of terms in the reconstruction of $x$ using the corrupted terms $\{\langle x,\varphi_{n}^{(N)}\rangle+\epsilon_{n}\}_{n\in\mathbb{N}}$. Explicitly, this means

[TABLE]

We offer a simple estimate for the magnitude of the error in this reconstruction.

Proposition 8**.**

Let $P_{n}$ denote the orthogonal projection onto the span of $\varphi_{n}$ as above, and let $T=\prod_{j=0}^{N-1}(I-P_{N-j})$. Then for any $\vec{x}\in\mathcal{H}$,

[TABLE]

where $\|\cdot\|_{N}$ denotes the operator norm on $\mathcal{B}(\mathcal{H}_{N})$.

Proof.

Assume for simplicity, $\vec{x}\in\mathcal{H}_{N}$. By definition of $\tilde{x}_{n}$,

[TABLE]

If we start from $n=N$ and repeat this $N$ times, we obtain

[TABLE]

By the same reasoning, for any $m\geq 2$,

[TABLE]

It then follows by induction that for all $m\in\mathbb{N}$

[TABLE]

Taking norms, we have

[TABLE]

Just as in the proof of the classical Kaczmarz algorithm (see e.g. [HKW19]), since $\{\varphi_{n}^{(N)}\}_{n=1}^{N}$ spans $\mathcal{H}_{N}$, we have $\|T\|_{N}<1$. Thus if we take the limit as $m\to\infty$, we obtain the desired estimate. ∎

Notice that if we bound the norm of the product of the orthogonal projections above by 1, then the estimate above implies that

[TABLE]

We can improve this estimate if we apply the relaxed Kaczmarz algorithm to the truncated sequence.

Proposition 9**.**

For $N\in\mathbb{N}$, let $\{\varphi_{n}^{(N)}\}$ denote the periodic sequence with period $N$ as in Equation (27), and $\{\epsilon_{n}^{(N)}\}$ be periodized analogously. For $\omega\in(0,2)$, define the approximations

[TABLE]

Then, $\|\mathbb{P}_{N}\vec{x}-\vec{z}^{(N)}\|^{2}\leq\dfrac{1}{A_{N}}\sum_{n=1}^{N}|\epsilon_{n}|^{2}$ where $A_{N}$ is the lower frame bound of $\{\varphi_{n}\}_{n=1}^{N}$.

Proof.

We have that $\vec{z}^{(N)}$ is the least-square solution to the system of equations

[TABLE]

We know that the solution $\vec{z}^{(N)}$ is given by the Moore-Penrose inverse $\Theta_{N}^{\dagger}$ of the data:

[TABLE]

where $\Theta_{N}$ is the analysis operator of $\{\varphi_{n}\}_{n=1}^{N}$:

[TABLE]

If $A_{N}$ is the lower frame bound of $\{\varphi_{n}\}_{n=1}^{N}$, then $\Theta_{N}$ is bounded below by $\sqrt{A_{N}}$. It follows that $\Theta_{N}^{\dagger}$ is bounded above by $\frac{1}{\sqrt{A_{N}}}$. Since

[TABLE]

the estimate now follows. ∎

4. Appendix

Proof of Proposition 2.

The first statement follows from the second, but an alternative proof can be found in [DHW14, Theorem 3.10]. The second statement follows from Fatou’s construction [Koo98, Section II.A]. ∎

Proof of Lemma 3.

For the inner function $b^{K}$, let $\mu^{K}$ denote the corresponding singular measure. By the Herglotz Representation, $\mu<<\mu^{K}$. Since by [Pol93] every $f\in H^{2}\ominus b^{K}H^{2}$ has convergent Fourier series in $L^{2}(\mu^{K})$, it follows that the Fourier series also converges in $L^{2}(\mu)$. ∎

Acknowledgement*.*

Lee Przybylski and Eric Weber were supported in part by the National Science Foundation and National Geospatial-Intelligence Agency under award #1830254.