Quasi-Newton Methods: Superlinear Convergence Without Line Searches for Self-Concordant Functions

TL;DR

This paper introduces a curvature-adaptive step size for quasi-Newton methods, enabling superlinear convergence on self-concordant functions without line searches, and demonstrates its effectiveness through numerical experiments.

Contribution

It extends Nesterov's curvature-adaptive step size to quasi-Newton methods, achieving superlinear convergence without line searches on self-concordant functions.

Findings

Superlinear convergence achieved with BFGS using adaptive step size

Numerical experiments show improved performance over traditional methods

Adaptive step size simplifies implementation by removing line searches

Abstract

We consider the use of a curvature-adaptive step size in gradient-based iterative methods, including quasi-Newton methods, for minimizing self-concordant functions, extending an approach first proposed for Newton's method by Nesterov. This step size has a simple expression that can be computed analytically; hence, line searches are not needed. We show that using this step size in the BFGS method (and quasi-Newton methods in the Broyden convex class other than the DFP method) results in superlinear convergence for strongly convex self-concordant functions. We present numerical experiments comparing gradient descent and BFGS methods using the curvature-adaptive step size to traditional methods on deterministic logistic regression problems, and to versions of stochastic gradient descent on stochastic optimization problems.