On the properties of the linear conjugate gradient method

TL;DR

This paper proves that the gradients generated by the linear conjugate gradient method are conjugate with respect to the matrix A, and introduces a new derivation based on conjugacy of search directions and gradient orthogonality.

Contribution

It establishes the conjugacy of gradients in the linear conjugate gradient method and presents a novel derivation approach based on search direction conjugacy and gradient orthogonality.

Findings

Gradients g_k are conjugate with respect to A.

New derivation of the method based on conjugacy and orthogonality.

Enhanced understanding of the properties of the conjugate gradient method.

Abstract

The linear conjugate gradient method is an efficient iterative method for the convex quadratic minimization problems $ \mathop {\min }\limits_{x \in { \mathbb R^n}} f(x) =\dfrac{1}{2}x^TAx+b^Tx $, where $ A \in R^{n \times n} $ is symmetric and positive definite and $ b \in R^n $. It is generally agreed that the gradients $ g_k $ are not conjugate with respective to $ A $ in the linear conjugate gradient method (see page 111 in Numerical optimization (2nd, Springer, 2006) by Nocedal and Wright). In the paper we prove the conjugacy of the gradients $ g_k $ generated by the linear conjugate gradient method, namely, $$g_k^TAg_i=0, \; i=0,1,\cdots, k-2.$$ In addition,a new way is exploited to derive the linear conjugate gradient method based on the conjugacy of the search directions and the orthogonality of the gradients, rather than the conjugacy of the search directions and the exact stepsize.