An optimization derivation of the method of conjugate gradients

TL;DR

This paper provides a new derivation of the conjugate gradients method by showing it minimizes a convex quadratic on the span of previous gradients, linking search directions to orthogonal gradients.

Contribution

It introduces a novel derivation of conjugate gradients based on convex quadratic minimization and orthogonal gradients, offering clearer insight into the method's geometric properties.

Findings

Conjugate gradients iterates minimize a convex quadratic on previous gradient span.

Search directions are orthogonal gradients scaled by a negative scalar.

The method relates to the Euclidean norm minimization on affine spans.

Abstract

We give a derivation of the method of conjugate gradients based on the requirement that each iterate minimizes a strictly convex quadratic on the space spanned by the previously observed gradients. Rather than verifying that the search direction has the correct properties, we show that generation of such iterates is equivalent to generation of orthogonal gradients which gives the description of the direction and the step length. Our approach gives a straightforward way to see that the search direction of the method of conjugate gradients is a negative scalar times the gradient of minimum Euclidean norm evaluated on the affine span of the iterates generated so far.