Triangle Steepest Descent: A Geometry-Based Gradient Algorithm with Guaranteed R-Linear Convergence

TL;DR

The paper introduces Triangle Steepest Descent (TSD), a geometry-based gradient algorithm with proven R-linear convergence and superlinear empirical performance, improving upon existing methods for quadratic optimization.

Contribution

It presents the TSD method, the first geometry-based gradient scheme since 1959, with theoretical convergence guarantees and superior empirical results.

Findings

TSD is at least R-linearly convergent for strongly convex quadratics.

TSD exhibits superlinear behavior in numerical experiments.

TSD outperforms Barzilai-Borwein and Dai-Yuan methods in quadratic problems.

Abstract

Gradient methods are among the simplest yet most widely used algorithms for unconstrained optimization. Motivated by a geometric property of the steepest descent (SD) method that can alleviate the zigzag behavior in quadratic problems, we develop a new gradient variant called the Triangle Steepest Descent (TSD) method. The TSD method introduces a cycle parameter $j$ that governs the periodic combination of past search directions, providing a geometry-driven mechanism to enhance convergence. To the best of our knowledge, TSD is the first formally established geometry-based gradient scheme since Akaike (1959). We prove that TSD is at least R-linearly convergent for strongly convex quadratic problems and demonstrate through extensive numerical experiments that it exhibits superlinear behavior, outperforming the Barzilai-Borwein (BB) method and monotone Dai-Yuan gradient method (DY) in quadratic cases. These results suggest that incorporating geometric information into gradient directions offers a promising avenue for developing efficient optimization algorithms.