Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

TL;DR

This paper establishes explicit superlinear convergence rates for both greedy and random quasi-Newton methods, including BFGS and SR1, extending previous results and improving convergence guarantees for these optimization algorithms.

Contribution

It extends convergence rate results to random quasi-Newton methods and provides improved superlinear convergence guarantees for BFGS and SR1 methods.

Findings

Random quasi-Newton methods have explicit superlinear convergence rates.

Improved convergence rates for BFGS and SR1 methods.

Analysis applies to strongly convex, smooth, and self-concordant functions.

Abstract

Optimization is important in machine learning problems, and quasi-Newton methods have a reputation as the most efficient numerical schemes for smooth unconstrained optimization. In this paper, we consider the explicit superlinear convergence rates of quasi-Newton methods and address two open problems mentioned by Rodomanov and Nesterov. First, we extend Rodomanov and Nesterov's results to random quasi-Newton methods, which include common DFP, BFGS, SR1 methods. Such random methods adopt a random direction for updating the approximate Hessian matrix in each iteration. Second, we focus on the specific quasi-Newton methods: SR1 and BFGS methods. We provide improved versions of greedy and random methods with provable better explicit (local) superlinear convergence rates. Our analysis is closely related to the approximation of a given Hessian matrix, unconstrained quadratic objective, as well as the general strongly convex, smooth, and strongly self-concordant functions.