A harmonic framework for stepsize selection in gradient methods

TL;DR

This paper introduces a harmonic framework for selecting stepsizes in gradient methods, extending existing schemes and proposing new flexible, tunable stepsize families with convergence analysis and experimental validation.

Contribution

It develops a harmonic framework for stepsize selection, extending the adaptive Barzilai-Borwein method with new families of stepsizes and provides convergence analysis and experiments.

Findings

Extended the Barzilai-Borwein method with negative and positive target-based stepsizes.

Provided convergence analysis for quadratic problems.

Demonstrated potential of new stepsize families through experiments.

Abstract

We study the use of inverse harmonic Rayleigh quotients with target for the stepsize selection in gradient methods for nonlinear unconstrained optimization problems. This provides not only an elegant and flexible framework to parametrize and reinterpret existing stepsize schemes, but also gives inspiration for new flexible and tunable families of steplengths. In particular, we analyze and extend the adaptive Barzilai-Borwein method to a new family of stepsizes. While this family exploits negative values for the target, we also consider positive targets. We present a convergence analysis for quadratic problems extending results by Dai and Liao (2002), and carry out experiments outlining the potential of the approaches.