Kahan's Automatic Step-Size Control for Unconstrained Optimization

TL;DR

This paper introduces a new adaptive step-size method for unconstrained optimization based on Kahan's approach, demonstrating theoretical convergence guarantees and superior practical performance compared to existing methods.

Contribution

It derives a short version of Kahan's step-size strategy, proves convergence properties, and develops an adaptive framework that enhances optimization efficiency.

Findings

The method converges at least R-linearly for strongly convex quadratic models.

The adaptive framework achieves global convergence and local R-linear convergence.

Numerical experiments show improved performance over existing gradient methods.

Abstract

The Barzilai and Borwein (BB) gradient method is one of the most widely-used line-search gradient methods. It computes the step-size for the current iterate by using the information carried in the previous iteration. Recently, William Kahan [Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019] proposed new Gradient Descent (KGD) step-size strategies which iterate the step-size itself by effectively utilizing the information in the previous iteration. In the quadratic model, such a new step-size is shown to be mathematically equivalent to the long BB step, but no rigorous mathematical proof of its efficiency and effectiveness for the general unconstrained minimization is available. In this paper, by this equivalence with the long BB step, we first derive a short version of KGD step-size and show that, for the strongly convex quadratic model with a Hessian matrix $H$, both the long and short KGD step-size (and hence BB step-sizes) gradient methods converge at least R-linearly with a rate $1-\frac{1}{{\rm cond}(H)}$. For the general unconstrained minimization, we further propose an adaptive framework to effectively use the KGD step-sizes; global convergence and local R-linear convergence rate are proved. Numerical experiments are conducted on the CUTEst collection as well as the practical logistic regression problems, and we compare the performance of the proposed methods with various BB step-size approaches and other recently proposed adaptive gradient methods to demonstrate the efficiency and robustness.