Conjugate Direction Methods Under Inconsistent Systems

TL;DR

This paper explores the behavior of conjugate direction methods, specifically CG and CR, when applied to inconsistent linear systems, revealing their stability issues and modifications for pseudo-inverse solutions.

Contribution

It provides a theoretical and empirical analysis of CG and CR methods under inconsistent systems, including modifications for pseudo-inverse solutions and their stability properties.

Findings

CR is equivalent to the minimum residual method in this context

CG can exhibit significant numerical instability in inconsistent systems

Small modifications enable pseudo-inverse solutions

Abstract

Since the development of the conjugate gradient (CG) method in 1952 by Hestenes and Stiefel, CG, has become an indispensable tool in computational mathematics for solving positive definite linear systems. On the other hand, the conjugate residual (CR) method, closely related CG and introduced by Stiefel in 1955 for the same settings, remains relatively less known outside the numerical linear algebra community. Since their inception, these methods -- henceforth collectively referred to as conjugate direction methods -- have been extended beyond positive definite to indefinite, albeit consistent, settings. Going one step further, in this paper, we investigate the theoretical and empirical properties of these methods under inconsistent systems. Among other things, we show that small modifications to the original algorithms allow for the pseudo-inverse solution. Furthermore, we show that CR is essentially equivalent to the minimum residual method, proposed by Paige and Saunders in 1975, in such contexts. Lastly, we conduct a series of numerical experiments to shed lights on their numerical stability (or lack thereof) and their performance for inconsistent systems. Surprisingly, we will demonstrate that, unlike CR and contrary to popular belief, CG can exhibit significant numerical instability, bordering on catastrophe in some instances.