On Residual norms in the Rayleigh-Ritz and refined Projection methods

TL;DR

This paper derives computationally efficient bounds for residual norm ratios in refined and Rayleigh-Ritz projection methods, aiding in their comparison and selection.

Contribution

It introduces a new, less costly bound for residual norms, enhancing practical assessment of the two projection methods.

Findings

Derived a new residual norm ratio bound

Bound is computationally less expensive

Facilitates practical comparison of methods

Abstract

This paper derives bounds for the ratio of residual norms in the refined and Rayleigh- Ritz projection methods. To do this, it uses the Least squares and line search projection method proposed in [6]. The bound derived in this paper is less costly to compute. Further, it is practically useful to assess the superiority of the refined and the Rayleigh-Ritz projection methods one over the other.

Equations

$$AV{y}_i-{\theta }_{i}V{y}_i\perp \mathcal{V}\text{mbox}fori=1,2,\cdots ,k.$$

$$AV{y}_i-{\theta }_{i}V{y}_i\perp v,\forall v\in \mathcal{V},i=1,2\cdots ,k.$$

$$\Vert (A-{\theta }_{i}I)u_{R_i}\Vert =\min \{\Vert (A-{\theta }_{i}I)u\Vert :u\in \mathcal{V}\text{mbox}and\Vert u\Vert =1\},$$

$$\Vert (A-{\theta }_{i}I)V{m}_i\Vert =\min \{\Vert (A-{\theta }_{i}I)V{y}_i+(A-{\theta }_{i}I)Vz\Vert :z\perp y_i\},$$

$$\frac{\Vert (A-{\theta }_{i}I)V{s}_i{\Vert }^2}{\Vert s_i{\Vert }^2}\text{mbox}isminimumoverspan\{y_i,(I-y_i{y}_i^{\ast })z_i\}.$$

$$\Vert (A-\theta I)Vm{\Vert }^2=\Vert (A-\theta I)Vy{\Vert }^2-\Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2.$$

$$⟨(A-\theta I)V(I-y{y}^{\ast })z,(A-\theta I)Vy⟩=-\Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2.$$

$$\begin{gathered} \Vert (A-\theta I)Vm{\Vert }^2=\Vert (A-\theta I)Vy{\Vert }^2+\Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2 \\ +2Real⟨(A-\theta I)V(I-y{y}^{\ast })z,(A-\theta I)Vy⟩. \end{gathered}$$

$$\tau =-\frac{⟨(B-\alpha I)u,(I-u{u}^{\ast })t⟩}{⟨J_{u,s}(I-u{u}^{\ast })t,(I-u{u}^{\ast })t⟩},$$

$$\rho (s)=\alpha -\frac{|⟨(B-\alpha I)u,(I-u{u}^{\ast })t⟩{|}^2}{⟨J_{u,s}(I-u{u}^{\ast })t,(I-u{u}^{\ast })t⟩},$$

$$⟨J_{u,s}(u-s),u-s⟩=\rho (u)-\rho (s),$$

$$(B-\rho (s)I)s=(B-\alpha I)u+J_{u,s}(s-u).$$

$$J_{y,s}:=(I-yy^{\ast})\Big{(}V^{\ast}(A-\theta I)^{\ast}(A-\theta I)V-\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}I\Big{)}(I-yy^{\ast}).$$

$$\displaystyle\tau=-\frac{\big{\langle}\big{(}V^{\ast}(A-\theta I)^{\ast}(A-\theta I)V-\|A-\theta I)Vy\|^{2}I\big{)}y,(I-yy^{\ast})z\big{\rangle}}{\big{\langle}J_{y,s}(I-yy^{\ast})z,(I-yy^{\ast})z\big{\rangle}},$$

$$\frac{\Vert (A-\theta I)Vs{\Vert }^2}{\Vert s{\Vert }^2}=\Vert (A-\theta I)Vy{\Vert }^2-\tau \Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2.$$

$$\frac{\Vert (A-\theta I)Vs{\Vert }^2}{\Vert s{\Vert }^2}$$

$$⟨J_{y,s}(y-s),y-s⟩$$

$$\Big{(}V^{*}(A-\theta I)^{*}(A-\theta I)V-\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}I\Big{)}s\\ =\big{(}V^{*}(A-\theta I)^{*}(A-\theta I)V-\|(A-\theta I)Vy\|^{2}I\big{)}y+J_{y,s}(s-y).$$

$$\frac{\tau }{\Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2}=\frac{1}{⟨J_{y,s}(I-y{y}^{\ast })z,(I-y{y}^{\ast })z⟩},$$

$$\frac{\Vert (A-\theta I)Vs{\Vert }^2}{\Vert s{\Vert }^2}=\Vert (A-\theta I)Vy{\Vert }^2-\frac{\Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^4}{⟨J_{y,s}(I-y{y}^{\ast })z,(I-y{y}^{\ast })z⟩}.$$

$$\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}=\Big{(}\frac{\tau-1}{\tau}\Big{)}\frac{\|(A-\theta I)V(I-yy^{*})z)\|^{2}}{\|(I-yy^{*})z\|^{2}},$$

$$(\tau-1)\Big{(}\|(A-\theta I)Vy\|^{2}-\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}\Big{)}=\|(A-\theta I)Vs\|^{2}\Big{(}1-\frac{1}{\|s\|^{2}}\Big{)}.$$

$$\Vert (A-\theta I)Vs{\Vert }^2=\Vert (A-\theta I)Vy{\Vert }^2+({\tau }^2-2\tau )\Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2.$$

$$\Vert (A-\theta I)Vs{\Vert }^2=\frac{\Vert (A-\theta I)Vs{\Vert }^2}{\Vert s{\Vert }^2}.(1+{\tau }^2\Vert (I-y{y}^{\ast })z{\Vert }^2).$$

$$(\tau -1)\frac{\Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2}{\Vert (I-y{y}^{\ast })z{\Vert }^2}=\tau (\Vert (A-\theta I)Vy{\Vert }^2-\tau \Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2).$$

$$\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}=\Big{(}\frac{\tau-1}{\tau}\Big{)}\frac{\|(A-\theta I)V(I-yy^{\ast})z\|^{2}}{\|(I-yy^{\ast})z\|^{2}}.$$

$$(\tau -1)\Vert (A-\theta I)Vy{\Vert }^2=\tau \Vert (A-\theta I)Vm{\Vert }^2-\frac{\Vert (A-\theta I)Vs{\Vert }^2}{\Vert Vs{\Vert }^2},$$

$$(\tau -1)\frac{\Vert (A-\theta I)Vs{\Vert }^2}{\Vert Vs{\Vert }^2}=\tau \Vert (A-\theta I)Vm{\Vert }^2-\Vert (A-\theta I)Vs{\Vert }^2.$$

$$\frac{\Vert (A-\theta I)Vm{\Vert }^2}{\Vert Vm{\Vert }^2}\le \Vert (A-\theta I)Vm{\Vert }^2.$$

$$\Vert (A-\theta I)Vy{\Vert }^2-\tau \Vert (A-\theta I)V(I-y{y}^{\ast })z{\Vert }^2=\frac{\Vert (A-\theta I)Vs{\Vert }^2}{\Vert s{\Vert }^2}\le \frac{\Vert (A-\theta I)Vm{\Vert }^2}{\Vert m{\Vert }^2}.$$

Full text

\slugger

siscxxxxxxxx–x

1 Introduction

Projection methods are quite familiar to solve large sparse eigenvalue problems. These methods produce eigenpair approximations using either oblique or orthogonal projections onto a specifically chosen vector space. Depending on a chosen vector space these methods are classified as Krylov subspace methods and Jacobi-Davidson type methods. The Lanczos method for symmetric matrices and the Arnoldi method for non-symmetric matrices are well-known and come under the category of Krylov subspace methods. Similar to Lanczos and Arnoldi methods, Jacobi-Davidson method also starts with an arbitrarily chosen unit vector called the Initial Vector. Then at each iteration, it extends an existing vector space using the solution of a system of linear equations called the Correction Equation. The correction equation varies depending on the procedure chosen for extracting eigenpair approximations from a vector space.

The Rayleigh-Ritz projection is a well-known procedure for extracting eigenpair approximations and is inherent in these projection methods [7, 9]. The Rayleigh-Ritz projection produces good approximations to the eigenvalues in the exterior of the spectrum. To better approximate interior eigenvalues, It requires the inverse of a given matrix which is computationally more costly. This problem resolved by using the Harmonic projection [3], but, as in the Rayleigh-Ritz projection, the eigenvector approximations produced by the Harmonic projection method also may not converge to an eigenvector, even though the corresponding eigenvalue approximations do converge [2, 10]. This misconvergence problem is avoidable in the Refined projection method [4, 1], and in the Least squares and Line search technique (LLS).

The refined projection method preserves an eigenvalue approximation that obtained using Rayleigh-Ritz projection. Then, it determines corresponding eigenvector approximation such that residual norm is minimum overall unit vectors in a vector space, from which an eigenvalue approximation sought. To find such an approximate eigenvector, it solves a singular value problem of smaller size. The LLS technique procures an approximate eigenpair from Rayleigh-Ritz projection. Then, it improves an eigenvector approximation in Rayleigh-Ritz projection by using least squares heuristics and line search technique [5, 6].

It is a general belief that the residual norm in the refined projection method is too small compared to that in the Rayleigh-Ritz projection. However, it is still unknown how much smaller the former residual norm compared to the later. Although the answer for this will be quite useful to create robust and efficient eigensolvers, it is still unanswered as these two methods came from different perspectives, singular value problem, and eigenvalue problem respectively. This paper extinguishes this question by deriving bounds for the ratio of residual norms in Rayleigh-Ritz and refined projections.

The paper is organized as follows; Section 2 briefly discusses the Rayleigh-Ritz projection, Refined projection, and LLS methods. Then, Section 3 determines the upper and lower bounds for the concerned ratio of residual norms. Section 4 concludes the paper.

2 Rayleigh-Ritz, Refined Rayleigh-Ritz and LLS methods

Let $A$ be a given matrix of order $n$ and ${\cal V}$ is a $k$ dimensional vector space. Suppose that column vectors of a matrix $V=[v_{1}~{}v_{2}~{}\cdots v_{k}]$ form an orthonormal basis of ${\cal V}.$ Then to produce an approximate eigenpairs of $A,$ the Rayleigh-Ritz projection method solves an eigenvalue problem for the matrix $H:=V^{\ast}AV$ of order $k.$ In general, $k$ is small and this eigenvalue problem can be solved using classical methods such as the QR algorithm.

As column vectors of $V$ are orthonormal, $V^{\ast}V=I.$ Further, an eigenpair $(\theta_{i},y_{i})$ of $H$ satisfies the relation: $V^{\ast}(A-\theta_{i}I)Vy_{i}=0.$ That means, approximations to eigenpairs of $A$ produced by the Rayleigh-Ritz projection method satisfy the Galerkin condition:

[TABLE]

Equivalently, this can be written as follows:

[TABLE]

In general, the above equation is not a good indication on $Vy_{i}$ being an eigenvector approximation. It shows that $Vy_{i}$ is orthogonal to its corresponding residual vector $AVy_{i}-\theta_{i}Vy_{i}.$ It further shows that eigenvector approximations $Vy_{i}$ may not converge to an eigenvector of $A$, even though corresponding eigenvalue approximations converge to an eigenvalue [2].

The Refined projection method is a remedy for the mis-convergence problem of eigenvector approximations in Rayleigh- Ritz projection. An eigenvector approximation $u_{R_{i}}$ in the Refined projection method satisfies the following:

[TABLE]

where $\theta_{i}$ is an eigenvalue approximation that retained from the Rayleigh-Ritz projection. Thus, an eigenvector approximation $u_{R_{i}}$ has the least residual norm overall unit vectors in the vector space from which eigenvalue approximations sought. Hence, the refined projection method computes an eigenvector approximation $u_{R_{i}}$ by solving a singular value problem for $(A-\theta_{i}I)V$. It is a general belief that $\|(A-\theta_{i}I)u_{R_{i}}\|\ll\|(A-\theta_{i}I)Vy_{i}\|.$ In this paper, we estimate $\|(A-\theta_{i}I)u_{R_{i}}\|/\|(A-\theta_{i}I)Vy_{i}\|$ by using a residual vector in the LLS method.

The LLS method preserves an eigenvalue approximation from the Rayleigh-Ritz projection. Then, it solves the following least squares problem to find a vector $z_{i}$:

[TABLE]

Further the LLS method uses the line search technique and updates an eigenvector approximation $Vy_{i}$ in Rayleigh-Ritz projection to $Vs_{i}/\|Vs_{i}\|.$ Therefore, an eigenvector approximation $Vs_{i}/\|Vs_{i}\|$ in the LLS method has the following optimal property:

[TABLE]

Note that eigenvector approximations in the LLS method are explicitly related to those in the Rayleigh-Ritz projection, unlike eigenvector approximations in the refined projection method. Further, The LLS method avoids the mis-convergence problem of eigenvector approximations in the Rayleigh-Ritz projection. The LLS method proved its better efficiency in the Jacobi-Davidson method compared to the Jacobi-Davidson method that inherently uses refined projection [6]. The following section will first establish a few relations between residual norms in the refined projection and the LLS method.

In what follows, the Ritz value fixed as $\theta$ and its corresponding eigenvector approximations as $Vy,Vu_{R},\frac{Vm}{\|Vm\|}$ and $\frac{Vs}{\|Vs\|}$ in the Rayleigh-Ritz, refined and the LLS methods respectively. Further, a subscript notation $i$ in this section will be ignored.

3 Comparison of residual norms

The following theorem derives a relation between residual norms in the Rayleigh-Ritz and the least squares part of the LLS methods via using a matrix $(A-\theta I)V.$

Theorem 1**.**

Let the minimization problem (2) have a non-zero solution vector $(I-yy^{\ast})z.$ Then the following is true:

[TABLE]

Proof.

From the equation (2) note that a vector $(I-yy^{\ast})z$ minimizes the least squares functional $\|(A-\theta I)Vy+(A-\theta I)V(I-yy^{\ast})z\|^{2}.$ Thus, it is a solution of the following normal equations:

[TABLE]

Taking an inner product with $z$ on both sides of the equation (5) gives

[TABLE]

From the equation (2) note that $Vm=Vy+V(I-yy^{\ast})z.$ Thus,

[TABLE]

Therefore, by using the equation (6), the above equation proves the equation (4). ∎

The Theorem-1 shows that the least squares approach in the LLS method reduces residual norm to a better extent than the Rayleigh-Ritz method, provided $\|(A-\theta I)V(I-yy^{\ast})z\|^{2}$ is large. Next, the following lemma [6, Lemma-3] will be helpful in the Theorem-2 to see that the line search technique of the LLS method will bring a further reduction in the residual norms.

Lemma 1**.**

Let $u$ be a vector of unit norm and $\alpha$ be the Rayleigh quotient of $u$ with respect to a Hermitian matrix $B$. Let $s:=u+\tau(I-uu^{\ast})t$, where $\tau\neq 0$ is chosen so that the Rayleigh quotient $\rho(s)$ of $s$ is minimum over ${\rm span}\{u,~{}(I-uu^{\ast})t\}.$ Write $J_{u,s}:=(I-uu^{\ast})(B-\rho(s)I)(I-uu^{\ast}).$ Then the following relations hold:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Theorem 2**.**

Let $Vy$ be the Ritz vector corresponding to a Ritz value $\theta$ and the vector $V(I-yy^{*})z$ be a solution of the minimization problem (2). Let $\tau$ be a scalar such that $s:=y+\tau(I-yy^{\ast})z$ minimizes $\displaystyle{\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}}$ over ${\rm span}\{y,\>(I-yy^{\ast})z\}.$ Write

[TABLE]

Then the following relations hold:

[TABLE]

[TABLE]

Proof.

Conveying the equation (7) in the Lemma-1 for the matrix $B:=V^{*}(A-\theta I)^{*}(A-\theta I)V$ and the vectors $u:=y,~{}t:=z$ gives the relation in the equation (12). Observe that in the Lemma-1, $\alpha=\|(A-\theta I)Vy\|^{2}$ for $B:=V^{*}(A-\theta I)^{*}(A-\theta I)V$ and $u:=y.~{}$

Next, to prove the equation (13), carry the equations 8, 9 and 10 in the Lemma-1 for the matrix $B:=V^{*}(A-\theta I)^{*}(A-\theta I)V$ and the vectors $u:=y,~{}t:=z.$ This gives the following equations respectively:

[TABLE]

[TABLE]

[TABLE]

Since $y\perp(I-yy^{\ast})z,$ by using the equation (6), the equations (12) and (14) gives the following relations:

[TABLE]

[TABLE]

Now, the equation (13) follows from the last two equations. ∎

The equation (13) gives a relation between residual norms in the Rayleigh-Ritz projection and LLS methods. The following theorem derives a few more relations by utilizing the equation (13) .

Theorem 3**.**

Let the vector $V(I-yy^{*})z$ be a solution of the minimization problem (2). Let $\tau$ be a scalar such that $s:=y+\tau(I-yy^{\ast})z$ minimizes $\displaystyle{\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}}$ over ${\rm span}\{y,\>(I-yy^{\ast})z\}.$ Then, the following equations hold true:

[TABLE]

and

[TABLE]

Proof.

As the vector $(I-yy^{\ast})z$ is a solution of the minimization problem (2), it satisfies the equation (6). Now, on expanding the expression $\|(A-\theta I)Vs\|^{2}$ by using $s=y+\tau(I-yy^{\ast})z,$ we have

[TABLE]

As $s=y+\tau(I-yy^{\ast})z$ and $\|y\|=1,$ we have $\|s\|^{2}=1+\tau^{2}\|(I-yy^{\ast})z\|^{2}.$ This implies

[TABLE]

Now, substitute the equations (13) and (20) in the above equation. On simplification, this gives the following relation:

[TABLE]

Recall from the equation (13) that the right-hand side of the above equation is equal to $\tau\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}.$ Thus, we have

[TABLE]

Therefore, we proved the equation (18). To prove the equation (19), observe the following from the equations (4), (13) and (20):

[TABLE]

and

[TABLE]

Now, subtracting one of the above equation from the other gives the equation (19). ∎

In the Theorems-2 and 3, we have seen that the relations between norms of residuals in the LLS method involve the scalar $\tau.$ The following theorem gives a lower bound for the scalar $\tau$.

Theorem 4**.**

Let $\tau$ be a scalar the same as that in the Theorem-2. Then $1\leq\tau.$

Proof.

By noting that $\|Vm\|^{2}=\|Vy\|^{2}+\|V(I-yy^{\ast})z\|^{2}=1+\|(I-yy^{\ast})z\|^{2}\geq 1,$ we have

[TABLE]

Now, from equations (3) and (13), we have

[TABLE]

Now $\tau\geq 1$ is followed from equation (4) and the above. ∎

The previous theorem has shown that $\tau\geq 1.$ By using the equation (19), observe that if $\tau=1$ then either $\|s\|^{2}=1$ or $\|(A-\theta I)Vs\|^{2}=0.$ That means, when $\tau=1,$ either $\|(I-yy^{\ast})z\|=0$ or $(\theta,Vs)$ is an exact eigenpair of $A.$ Hence, in what follows we assumed that $\tau\neq 1.$

3.1 Comparison of line search least squares with refined projection

In the previous section, we compared the residual norms in the Rayleigh-Ritz projection and Line search Least squares(LLS) methods. In this subsection, we establish a connection between the LLS and refined projection methods.

Recall that an approximate eigenvalue $\theta$ in the refined projection method is the same as that in the Rayleigh-Ritz projection, and $u_{R}:=Vz_{R}$ is an eigenvector approximation, where $z_{R}$ is a right singular vector corresponding to the smallest non-zero singular value $\sigma^{2}$ of a matrix $(A-\theta I)V.$ Hence, a vector $z_{R}$ satisfies the following relations:

[TABLE]

Using the normal equations (5) of a least squares problem (2), observe that a vector $m=y+(I-yy^{\ast})z$ satisfies the following equation:

[TABLE]

Now, take an inner product on both sides with a vector $z_{R}$ and use the equation (23) to obtain the following:

[TABLE]

The above equation shows that the ratio of $z_{R}^{\ast}y$ to $z_{R}^{\ast}(I-yy^{\ast})z$ is real. In fact, the ratio is positive since $K=\|(A-\theta I)Vm\|^{2}\geq\frac{\|(A-\theta I)Vm\|^{2}}{\|m\|^{2}}\geq\sigma^{2}.$ The last inequality follows since the refined Ritz vector has smallest residual norm overall unit vectors in the vector space spanned by column vectors of $V.$

By using the equation (25) and $s=y+\tau(I-yy^{\ast})z$, we have

[TABLE]

Since $\tau>1$ and $K\geq\sigma^{2},$ the ratio of $z_{R}^{\ast}s$ to $z_{R}^{\ast}y$ is positive. Now, we restate this discussion in the form of a lemma for the future use.

Lemma 2**.**

Let $s,(I-yy^{\ast})z$ be the same as that in the equation (3), and $Vz_{R}$ be the refined Ritz vector corresponding to the Ritz value $\theta.$ Then, $\frac{z_{R}^{\ast}s}{z_{R}^{\ast}y}$ and $\frac{z_{R}^{\ast}(I-yy^{\ast})z}{z_{R}^{\ast}y}$ are positive.

The above lemma inherently assumed that $z_{R}^{\ast}y\neq 0,$ which means the Ritz and refined Ritz vectors are not orthogonal. In numerical experiments, this statement holds true, in general. In the next theorem, we will use the above lemma to derive a lower bound for $\frac{K-\sigma^{2}}{\sigma^{2}}.$

Theorem 5**.**

Let $\tau$ and a vector $(I-yy^{\ast})z$ be the same as that in the Theorem-2. Assume that $\|(A-\theta I)Vz_{R}\|^{2}=\sigma^{2},$ $K=\|(A-\theta I)Vm\|^{2},$ where $Vz_{R}$ is a refined Ritz vector corresponding to the Ritz value $\theta,$ and $m$ is a solution vector of a least squares problem (2). Then

[TABLE]

Proof.

Recall the equation (11) from the previous subsection:

[TABLE]

Note that $s-y=\tau(I-yy^{\ast})z.$ Then, by using the equations (6) and (23), we have

[TABLE]

By using the equations (4), (24), and (25), this gives

[TABLE]

Recall the following equation (16) from the previous subsection:

[TABLE]

Apply an inner product on both sides of the above equation with a vector $z_{R}.$ In the resulting equation substitute $z_{R}^{\ast}J_{y,s}(s-y)$ from the previous equation in the right-hand side expression. Then use the equation  (23) on the left-hand side expression of the same equation. It gives the following:

[TABLE]

Now, divide the both sides of the above equation with $z_{R}^{\ast}y$ to obtain the following:

[TABLE]

Recall from  Lemma-2 that $\frac{z_{R}^{\ast}s}{z_{R}^{\ast}y}$ and $\frac{z_{R}^{\ast}(I-yy^{\ast})z}{z_{R}^{\ast}y}$ are positive, and from the Theorem-4 that $\tau\geq 1.$ By using these, the following inequality relation follows from the equations (25) and (26).

[TABLE]

As $\sigma^{2}-\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}$ is non-positive, by using the above inequation, the equation (29) gives

[TABLE]

Now, by rearranging the terms, the above inequation can be written as follows:

[TABLE]

As $\sigma^{2}\leq\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}},$ we have \Big{(}\|(A-\theta I)Vy\|^{2}-\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}\Big{)}\leq(\|(A-\theta I)Vy\|^{2}-\sigma^{2}). Since $\tau>1,$ by using these two inequalities the above equation gives the following relation:

[TABLE]

Now, on substituting the equation (25) this inequality gives

[TABLE]

Further, by using the equation (19) the left-hand side expression in the above equation becomes equal to $\frac{\|s\|^{2}-1}{\tau}.$ Thus, we have

[TABLE]

As $s=y+\tau(I-yy^{\ast})z$ and $\|y\|=1,$ we have $\|s\|^{2}=1+\tau^{2}\|(I-yy^{\ast})z\|^{2}.$ Therefore, on substituting this in the above equation, we get the required inequality as in the equation (27). ∎

The above theorem gives a lower bound for $\frac{K-\sigma^{2}}{\sigma^{2}}.$ In order to derive an upper bound for $\frac{K-\sigma^{2}}{\sigma^{2}},$ we define the following function of a variable $\alpha:$

[TABLE]

Now, the following lemma describes the characteristics of the function $f(\alpha).$

Lemma 3**.**

*Let $f(\alpha)$ be a function of $\alpha$ defined as in the equation (30). Then the following are hold true:

a) $f(\alpha)$ is a monotonic decreasing function of $\alpha.$

b) If $f(\alpha)\leq 0$ for any $0\leq\alpha<\tau$ then $\alpha$ satisfies the inequation:*

[TABLE]

c) There exists a root between [math] and $\tau$ for the equation $f(\alpha)=0.$

Proof.

a) For the given function $f(\alpha),$ we have

[TABLE]

To see $f^{\prime}(\alpha)\leq 0,~{}\forall\alpha,$ use the optimal property of residual norms in the refined projection method and observe $\sigma^{2}\leq K.$ Therefore, $f(\alpha)$ is a monotonically decreasing function of $\alpha.$

b) The substitution of the equation (19) in the equation (30) leads to

[TABLE]

Thus, $f(\alpha)\leq 0$ implies $(1-\|s\|^{2})+(\tau-\alpha)(\frac{K-\sigma^{2}}{\sigma^{2}})$ is non-positive. Therefore, by using the equation (27), this gives the required inequality as $0\leq\alpha<\tau.$

c) We have

[TABLE]

By using $\|s\|^{2}=1+\tau^{2}\|(I-yy^{\ast})z\|^{2},$ the above equation can be written as

[TABLE]

As $\tau>1$, it is an easy to see that $f(0)>0$ by using (27) and the above equation.

Similarly, consider

[TABLE]

As $\|s\|^{2}=1+\tau^{2}\|(I-yy^{\ast})z\|^{2}>1,$ we have $f(\tau)<0.$ Therefore, we have $f(0)>0$ and $f(\tau)<0.$ Hence, there exists a root for the equation $f(\alpha)=0$ between [math] and $\tau.$ ∎

The Lemma-3 has shown that equation $f(\alpha)=0$ has a solution between [math] and $\tau.$ Let $\alpha_{3}$ is such a root. Then, from the equation (31) we have

[TABLE]

Note that the above equation turns the problem of finding an upper bound for $\frac{K-\sigma^{2}}{\sigma^{2}}$ into deriving an upper bound for $\alpha_{3}.$ To derive an upper bound for $\alpha_{3},$ which depends only on the scalar $\tau,$ we make use of the following function of $\alpha:$

[TABLE]

Note that the function $g(\alpha)$ is obtained by multiplying the difference between both sides of the inequality in the Lemma-3(b) with $\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}.$ In the following lemma, we characterize the function $g(\alpha)$ defined in the above equation and will establish its relation with the function $f(\alpha).$

Lemma 4**.**

*Let $g(\alpha)$ be a function defined as in the equation (33). Then, the functions $g(\alpha)$ and $g(\alpha)-f(\alpha)$ are monotonically increasing functions of $\alpha$ in the interval $[0,\tau).$

Proof.

Use $\|s\|^{2}=1+\tau^{2}\|(I-yy^{\ast})z\|^{2}$ in the equation (33)to observe that

[TABLE]

This shows that

[TABLE]

Thus, the proof is over as $f^{\prime}(\alpha)\leq 0,~{}\forall\alpha$ from the Lemma-3(a). ∎

In what follows, with the help of the function $g(\alpha)$ we derive an upper bound for $\alpha_{3},$ a solution of the equation $f(\alpha)=0.$ For this, the following theorem introduce $\alpha_{6},$ a root of the equation $f(\alpha)-\tau\alpha\|(I-yy^{\ast})z\|^{2}\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}=0$ and determine a relation between $\alpha_{6}$ and $\alpha_{3}.$

Theorem 6**.**

Let $\alpha_{6}\neq\tau$ is such that $f(\alpha_{6})-\tau\alpha_{6}\|(I-yy^{\ast})z\|^{2}\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}=0,$ then $2\alpha_{3}-\tau\leq\alpha_{6}<\alpha_{3}$ and

[TABLE]

where $\alpha_{3}<\tau$ is same as in the Lemma-3.

Proof.

Note that substituting the equation (32) in the equation (31) gives

[TABLE]

Further, using $\|s\|^{2}=1+\tau^{2}\|(I-yy^{\ast})z\|^{2}$ the above equation can be simplified as the following:

[TABLE]

Let $\alpha_{6}\neq\tau.$ Since $f(\alpha_{6})-\tau\alpha_{6}\|(I-yy^{\ast})z\|^{2}\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}=0$ the equation (36) would imply

[TABLE]

As $\tau\|(I-yy^{\ast})z\|^{2}\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}\neq 0$ and $\tau\neq\alpha_{6},$ on simplification, the above equation gives

[TABLE]

Using Componendo and Dividendo, the above equation proves the equation (35). ∎

Recall that our aim is to determine an upper bound for $\alpha_{3}$ in terms of $\tau.$ From the above theorem it is equivalent to identify an upper bound for $\alpha_{6}.$ For this,the following section introduces another scalar, called $\alpha_{7}$ and determine its location with respect to $\alpha_{3}$ and $\alpha_{6}$ on the real line.

4 Sectional Formulae

In this section, we introduce a scalar $\alpha_{7}<\tau,$ a root of the equation $f(\alpha)-g(\alpha)=0.$ The following lemma determines a sufficient condition for $\alpha_{6}$ to divide $\alpha_{7}$ and $\alpha_{3}$ externally on the real line.

Lemma 5**.**

Let $f(\alpha),g(\alpha)$ and $\alpha_{6}$ be the same as mentioned in the Lemma-4 and the Theorem-6. Assume that $\alpha_{7}<\tau$ is a root of the equation $f(\alpha)-g(\alpha)=0.$ Then $\alpha_{7}\leq\alpha_{3}.$ Further, If $\alpha_{7}\leq\tau-1$ then $\alpha_{6}\leq\alpha_{7}.$

Proof.

Observe from the equation (34) that

[TABLE]

and

[TABLE]

Note that $\alpha_{7}\leq\tau-1.$ Since $g(\alpha)$ is monotonically increasing function, from the Lemma–4 and using the first equation in the above, we have

[TABLE]

Using $f(\alpha_{7})=g(\alpha_{7})$ the above equation gives the following inequality:

[TABLE]

Now, $\alpha_{6}\leq\alpha_{7}$ is proved by invoke from the Lemma-3(a) that $f(\alpha)-\tau\alpha\|(I-yy^{\ast})z\|^{2}\frac{\|(A-\theta I)Vs\|^{2}}{\|s\|^{2}}$ is a monotonically decreasing function of $\alpha$ and

[TABLE]

Note that $\alpha_{7}\leq\alpha_{3}$ follows as $f(\alpha_{3})=0$ and $f(\alpha)$ is a monotonically decreasing function of $\alpha$ from the Lemma-3(a). ∎

The Lemma-5 says that if $\alpha_{7}\leq\tau-1$ then $\alpha_{6}\leq\tau-1.$ This implies $\alpha_{3}\leq\frac{2\tau(\tau-1)}{2\tau-1}$ as $\alpha_{3}=\frac{2\tau\alpha_{6}}{\tau+\alpha_{6}}$ from the Theorem-6. By using the Harmonic mean and Arithemetic mean inequality this gives the following lemma:

Lemma 6**.**

Let $\alpha_{7}$ and $\alpha_{3}$ be scalars the same as in the Lemma-42. If $\alpha_{7}\leq\tau-1$ then $\alpha_{3}\leq\tau-\frac{1}{2}.$

The above lemma had given an upper bound for $\alpha_{3}$ when $\alpha_{6}$ externally divides $\alpha_{7}$ and $\alpha_{3}$ on the real line. In the following, we derive an upper bound for $\alpha_{3}$ when $\alpha_{6}$ lies in between $\alpha_{7}$ and $\alpha_{3}$ on the real line. Further, In what follows we use the notation

[TABLE]

for the convenience.

Lemma 7**.**

Let $\alpha_{7}\leq\alpha_{6}$ and the point $C(\alpha_{6},\alpha_{6}z)$ divides the points $A(\alpha_{7},g(\alpha_{7}))$ and $B(\alpha_{3},0)$ internally in the ratio $m:n.$ Then, note that the following relations hold true:

[TABLE]

Proof.

Note that $A,B,$ and $C$ are lies on the same straight line as $f(\alpha_{3})=0,$ $\alpha_{6}z=f(\alpha_{6})$ and $f(\alpha_{7})=g(\alpha_{7})$ from the Theorem-6 and the Lemma-5. Further, as $C$ divides $A$ and $B$ internally in the ratio $m:n,$ we have

[TABLE]

The above equation yields the following relations:

[TABLE]

The two relations in the above equation give:

[TABLE]

Hence the lemma proved. ∎

The Lemma-7 has found the ratio at which the point $C$ divides $A$ and $B.$ By using this the next theorem finds a relation between $\alpha_{6},\alpha_{7},$ and $\tau.$

Theorem 7**.**

Let $\alpha_{3},\alpha_{6},$ and $\alpha_{7}$ be the same as in the Lemma-7. Let $\tau\neq 1$ and $z=\tau\|(I-yy^{\ast})z\|^{2}\frac{\|(A-\theta I)Vs\|^{2}}{\|Vs\|^{2}}\neq 0.$ Then

[TABLE]

Proof.

From the equation (38) note that

[TABLE]

Thus,

[TABLE]

Recall from the equations (35) and (36) that $f(2\alpha_{6}-\tau)=(\tau+2\alpha_{6})z.$ Thus, as $f(\alpha_{7})=g(\alpha_{7})$ the above equation gives the relation:

[TABLE]

Therefore,

[TABLE]

Now, by using the facts $(\tau+2\alpha_{6})z=f(2\alpha_{6}-\tau)$ and $\alpha_{3}=2\tau\alpha_{6}/(\tau+\alpha_{6})$ observe that

[TABLE]

Substituting this in the equation (42) gives the following relation:

[TABLE]

This can be written as follows:

[TABLE]

As $\alpha_{3}$ is harmonic mean of $\alpha_{6}$ and $\tau$ we have $\frac{\alpha_{3}-\alpha_{6}}{\alpha_{6}}=\frac{\tau-\alpha_{3}}{\tau}.$ Substituting this in the above equation gives

[TABLE]

Now, we prove that if $2\alpha_{6}-\tau\neq\alpha_{7}$ then $\alpha_{6}=\alpha_{3}=\tau.$ Recall from the equation (36) that $f(\alpha)$ is a straightline in the variable $\alpha$ and its slope is $\frac{-\tau z}{\tau-\alpha_{3}}.$ Thus, $\frac{f(x_{1})-f(x_{2})}{x_{1}-x_{2}}$ is constant for any $x_{1}$ and $x_{2}.$ Using this observe that $\alpha_{6}=\alpha_{3}$ from the above equation. But this implies $\alpha_{3}=\tau$ as $\alpha_{3}=\frac{2\tau\alpha_{6}}{\tau+\alpha_{6}}.$ A contradiction to the assumption that $\alpha_{3}\neq\tau.$ Note that if $\alpha_{3}=\tau$ then either $\tau=1$ or $(\theta,Vy)$ is an exact eigenpair of $A.$ Therefore $\alpha_{7}=2\alpha_{6}-\tau.$ ∎

By using the above theorem the following lemma gives the values of $\alpha_{6}$ and $\alpha_{3}$ in terms of $\tau,$ when $\alpha_{6}$ stays in between $\alpha_{7}$ and $\alpha_{3}$ as mentioned in the Lemma-7.

Lemma 8**.**

Let $g(\alpha)$ and $f(\alpha)$ be functions of $\alpha$ defined as in the equations (34) and (36) respectively. Assume that scalars $\alpha_{3},$ $\alpha_{6},$ and $\alpha_{7}$ are the same as in the Lemma-7. Then

[TABLE]

if the point $(\alpha_{6},f(\alpha_{6}))$ internally divides $(\alpha_{7},f(\alpha_{7}))$ and $(\alpha_{3},0).$

Proof.

By using the equation (34) we have

[TABLE]

From the proof of the previous theorem note that $f(2\alpha_{6}-\tau)=(\tau+2\alpha_{6})z.$ Since $\alpha_{7}=2\alpha_{6}-\tau$ and $f(\alpha_{7})=g(\alpha_{7})$ this implies

[TABLE]

As $z\neq 0,$ on simplifying this gives the following quadratic equation in $\alpha_{6}:$

[TABLE]

As $\alpha_{6}\leq\tau,$ the above equation gives

[TABLE]

Thus, by using $\alpha_{3}=\frac{2\tau\alpha_{6}}{\tau+\alpha_{6}}$ we have

[TABLE]

As $\tau>1,$ the right-hand side expression in the above equation is less than $\tau-\frac{1}{14}.$ This can be seen by plotting the graph by using any software such as MATLAB or DESMOS online grapher etc. ∎

From the Lemmas-6 and 8 observe that the inequality $\alpha_{3}\leq\tau-\frac{1}{14}$ holds true, irrespective of the position of $\alpha_{6}$ with respect to the scalars $\alpha_{3}$ and $\alpha_{7}.$ We use this in the next section to find a bound for the ratio of residual norms in the refined and Rayleigh-Ritz projection methods.

5 Main results

Recall from the equation (32) that $\frac{K-\sigma^{2}}{\sigma^{2}}=\frac{\tau^{2}\|(I-yy^{\ast})z\|^{2}}{\tau-\alpha_{3}}.$ Here, we used the fact that $\|s\|^{2}=1+\tau^{2}\|(I-yy^{\ast})z\|^{2}.$ Then using $\alpha_{3}\leq\tau-1/14$ we have

[TABLE]

This together with the Theorem-5 gives the result that we state in the form of a lemma here.

Lemma 9**.**

Let $\tau$ and a vector $(I-yy^{\ast})z$ be the same as in the Theorem-2. Also assume that $Vz_{R}$ is a refined Ritz vector corresponding to the Ritz value $\theta.$ Then

[TABLE]

where $\sigma^{2}=\|(A-\theta I)Vz_{R}\|^{2},$ $K=\|(A-\theta I)Vm\|^{2},$ and $m$ is a solution vector of a least squares problem (2).

The Lemma-9 has established the relation between residual norms in LLS and refined projection methods. It has shown that the residual norms in both the methods converge to zero together. Now, recall from the equations (4) and (21) that

[TABLE]

By using this equation, the Lemma-9 gives the following main result which relates residual norms in Rayliegh-Ritz and refined projection methods.

Theorem 8**.**

Let $\theta$ be a Ritz value, and $Vy,~{}Vz_{R}$ be the corresponding eigenvector approximations in the Rayleigh-Ritz and refined projection methods respectively. Then

[TABLE]

The above theorem relates residual norms in the Rayleigh-Ritz and refined projection methods. It helps us to predict the range of $\sigma^{2},$ square of a residual norm in the refined projection method without computing a refined Ritz vector, a right singular vector of $(A-\theta I)V.$ It just requires computing $\tau$ and $(I-yy^{\ast})z.$ Note that $(I-yy^{\ast})z$ is obtained by solving normal equations for the least squares problem in equation (2). $\tau,$ a solution of the problem considered in the equation (3) is obtained by solving an eigenvalue problem of the following matrix of order $2.$

[TABLE]

The above theorem may helps to create an efficient algorithm that use a combination of refined projection and Rayleigh-Ritz projection methods for solving sparse linear eigenvalue problems.

6 Numerical experiments

In this section, we demonstrate the theory developed so far. This section been divided into two subsections. The first part discusses a method to compute $\tau$ and $\|(I-yy^{\ast})z\|.$ The second part reports numerical results.

6.1 Implementation details

In this section, we discuss how the LLS method obtains eigenvector approximations as the LLS method is the most recent one. The following theorem will be helpful to compute an eigenvector approximation in the Least squares and line search(LLS) method.

Theorem 9**.**

Let $(\theta,Vy)$ be a Ritz pair but not an exact eigenpair of $A$. Let the vector $x$ satisfies equation

[TABLE]

Then

[TABLE]

is the corresponding eigenvector approximation in the least squares method, where

[TABLE]

For the proof of the theorem; See Theorem-4 in [5]. Using the above theorem the LLS method computes a vector $\frac{(I-yy^{\ast})z}{\|(I-yy^{\ast})z\|}$ without computing $(I-yy^{\ast})z$ explicitly.

The LLS technique further improves an eigenvector approximation in the Theorem-9 by using the Line-Search technique introduced in the Theorem-2. As mentioned in the previous section, LLS obtains it by solving the following eigenvalue problem of a matrix of order $2.$

[TABLE]

From the Lemma-9 note that $\tau$ and $\|(I-yy^{\ast})z\|^{2}$ are required to compare residual norms in the Rayleigh-Ritz projection, LLS, and refined projection methods. Observe that $\tau$ can be computed from the equation (18) since $\frac{\|(A-\theta I)Vs\|^{2}}{\|Vs\|^{2}}$ and $\frac{(I-yy^{\ast})z}{\|(I-yy^{\ast})z\|}$ are known. The following lemma describes a procedure to compute $\|(I-yy^{\ast})z\|^{2}.$

Lemma 10**.**

Let $s$ be an eigenvector of the matrix $B^{\ast}V^{\ast}(A-\theta I)^{\ast}(A-\theta I)VB,$ where B=\Big{[}y~{}~{}\frac{(I-yy^{\ast})z}{\|(I-yy^{\ast})z\|}\Big{]}. If $\|s\|=1$ and $s=\frac{y+\tau(I-yy^{\ast})z}{\sqrt{1+\tau^{2}\|(I-yy^{\ast})z\|^{2}}}$ then

[TABLE]

So far, In this section we discussed how to compute $\tau$ and $\|(I-yy^{\ast})z\|.$ In the following subsection, we report the numerical results.

6.2 Numerical results

The numerical experiments have been conducted on many benchmark matrices from the Matrix Market Website. Here we report only two examples as all the experiments validated the theory in the previous sections. All the experiments have been conducted on Intel core i7 processor using MATLAB-R2016(b) with $eps=2.2204e-16.$

Example 1**.**

In this example we used the Jacobi-Davidson method without restarting to compute right most eigenvalues of the matrix $OLM5000.$ For details of the matrix; See Matrix Market Website. The initial vector has all its entries equal to $\frac{1}{\sqrt{n}},$ where $n$ is the order of the matrix. An eigenvector approximation in the LLS method is used in the correction equation. It solved approximately by using 20 iterations of un-restarted GMRES method. In GMRES, we took the zero vector as an initial approximation to the solution of the correction equation.

At each iteration of the Jacobi-Davidson method, we compute refined Ritz vector also and compare its residual norm with those in the Rayliegh-Ritz and LLS methods in accordance with the Lemma-9.

In this example we fixed the size of a search subspace in the Jacobi-Davidson method to $200.$ It is well known that $\tau=1$ in the first iteration as the search subspace contains only initial vector.

The Figure-1 depicts the curves of $\|(I-yy^{\ast})z\|^{2}$ against the iteration number. It is clear from the figure that as the iteration number grows, $\|(I-yy^{\ast})z\|^{2}$ decreases. We found that from the $150^{th}$ iteration onwards its value is below $\mathcal{O}(10^{-5}).$ Thus, the Figure-1 confirms the well known fact that near the convergence, eigenvector approximations in the refined method and Rayleigh-Ritz projection method almost coincide.

The Figure-2 shows $K/\sigma^{2}$ and its bounds in the Lemma-9 against iteration number. Recall that $\sigma$ is residual norm in the refined projection method which is minimum over all unit vectors in the entire search subspace. From the figure it is easy to see that $K/\sigma^{2}$ lies in the interval $(1,1.2),$ that means $\sigma^{2}>(0.8)K.$ Here, $K=\|(A-\theta I)V(y+(I-yy^{\ast})z\|^{2},$ a residual norm of non-normalized vector $Vy+V(I-yy^{\ast})z.$ Therefore, normalized residual norm of this vector will be much closer to residual norm in the refined projection.

The Figure-3 shows values of $\tau$ against iteration number. Observe from the figure that $\tau$ is in the interval $(1,1.013).$ However, we do not have any theoretical evidence on a real number upper bound of $\tau.$ Thus, from this figure and the equations (4) and (13) it is evident that the line search technique brings only a marginal reduction in the residual norm obtained only with the least squares heuristics.

Example 2**.**

In this example the matrix is $DW2048$ from the Matrix Market. We use the Arnoldi method with LLS to compute the eigenvalue with largest real part. The initial vector chosen as $ones(2048,1).$ Since search subspace updataion in the Arnoldi method doesn’t require eigenvector approximation like the Jacobi-Davidson method, we tested our theoretical results with explicitly restarting Arnoldi method. In restarting Arnoldi method the size of a Krylov subspace is fixed to $10$ for this example. However, the same scenario that we present here is observed with subspaces of larger size. At the end of each restart, an initial vector updated by an eigenvector approximation at hand obtained using the LLS method.

The Figures-4, 5, and 6 shows the curves for $\|(I-yy^{\ast})z\|^{2},$ $K/\sigma^{2},$ and $\tau$ against restart number respectively. Observe from the Figure-4 that norm of $(I-yy^{\ast})z$ recedes near to zero as the restart number grows. Thus, by using the Lemma-9 note that upper and lower bounds for $K/\sigma^{2}$ nearly coincide when the restart number is larger as $\tau$ is finite. The Figure-5 demonstrate this fact. As in the previous example, It has been observed from the Figure-6 that $\tau<1.025.$ However, a theoretical result that gives a real number upper bound for $\tau$ has yet to be found.

7 Conclusions

In this paper, bounds for a ratio of residual norms in the refined and Rayleigh-Ritz projections have been derived. These bounds are in terms of $\|(I-yy^{\ast})z\|^{2}.$ and$\tau,$ a scalar in the line search and least sqaures method; See equation (3). Here, $Vy$ is an eigenvector approximation in the Rayleigh-Ritz projection method and $z$ is a solution vector of a least squares problem in the equation (2).

Moreover, the bounds that are derived in this paper are different from the relationships between the above mentioned residuals which have already been studied by Z. Jia; see Section 4 in Z. Jia “Some theoretical comparisons of refined Ritz vectors and Ritz vectors”, Science in China Ser. A Mathematics 2004 Vol.47 Supp. 222-233. In this reference, the relationships between the above-mentioned residuals are in terms of the angle between refined Ritz vector and Ritz vector and the second smallest singular value of a singular value problem in the refined method. Thus, computing those bounds practically requires the computation of a Ritz vector, refined Ritz vector, the angle between them and the second smallest singular value. It is very costly to compute all these quantities. Thus, these relations are useful only theoretically since once refined Ritz vector and Ritz vectors are computed, practically there is no requirement of computing the second smallest singular value to compare the residual norms in both the methods.

The bounds derived in this article for the ratio of residual norms in the Rayleigh-Ritz and the refined projection methods are practically useful. These bounds predicts how much smaller the residual norm in refined projection method compared to residual norm in the Rayleigh-Ritz method, without computing the refined Ritz vector.