A mechanism of three-dimensional quadratic termination for the gradient method with applications

TL;DR

This paper introduces a three-dimensional quadratic termination mechanism for gradient methods, enhancing their performance in quadratic and unconstrained optimization by using a novel stepsize that leverages previous gradient information.

Contribution

It develops a new three-dimensional quadratic termination mechanism and a corresponding stepsize, improving gradient method efficiency without requiring line searches or Hessians.

Findings

The new stepsize improves the convergence of the BB method.

Numerical results show significant performance gains over existing methods.

The method extends effectively to general unconstrained optimization.

Abstract

Recent studies show that the two-dimensional quadratic termination property has great potential in improving performance of the gradient method. However, it is not clear whether higher-dimensional quadratic termination leads further benefits. In this paper, we provide an affirmative answer by introducing a mechanism of three-dimensional quadratic termination for the gradient method. A novel stepsize is derived from the mechanism such that a family of delayed gradient methods equipping with the novel stepsize have the three-dimensional quadratic termination property. When applied to the Barzilai--Borwein (BB) method, the novel stepsize does not require the use of any exact line search or the Hessian, and can be computed by stepsizes and gradient norms in previous iterations. Using long BB steps and some short steps associated with the novel stepsize in an adaptive manner, we develop an efficient gradient method for quadratic optimization and further extend it to general unconstrained optimization. Numerical experiments show that the three-dimensional quadratic termination property can significantly improve performance of the BB method, and the proposed method outperforms gradient methods that use stepsizes with the two-dimensional quadratic termination property.