Non-asymptotic Global Convergence Analysis of BFGS with the Armijo-Wolfe Line Search

TL;DR

This paper provides the first explicit non-asymptotic global convergence rates for BFGS with Armijo-Wolfe line search, including linear and superlinear rates, applicable from any starting point and initial Hessian approximation.

Contribution

It establishes the first global complexity bounds for BFGS with Armijo-Wolfe line search, detailing convergence rates and step size selection mechanisms.

Findings

BFGS achieves a linear convergence rate of (1 - 1/κ)^t for strongly convex functions.

BFGS with Armijo-Wolfe line search attains a convergence rate depending only on line search parameters.

A global superlinear convergence rate of O((1/t)^t) is established.

Abstract

In this paper, we present the first explicit and non-asymptotic global convergence rates of the BFGS method when implemented with an inexact line search scheme satisfying the Armijo-Wolfe conditions. We show that BFGS achieves a global linear convergence rate of $(1 - \frac{1}{\kappa})^t$ for $\mu$-strongly convex functions with $L$-Lipschitz gradients, where $\kappa = \frac{L}{\mu}$ represents the condition number. Additionally, if the objective function's Hessian is Lipschitz, BFGS with the Armijo-Wolfe line search achieves a linear convergence rate that depends solely on the line search parameters, independent of the condition number. We also establish a global superlinear convergence rate of $\mathcal{O}((\frac{1}{t})^t)$. These global bounds are all valid for any starting point $x_0$ and any symmetric positive definite initial Hessian approximation matrix $B_0$, though the choice of $B_0$ impacts the number of iterations needed to achieve these rates. By synthesizing these results, we outline the first global complexity characterization of BFGS with the Armijo-Wolfe line search. Additionally, we clearly define a mechanism for selecting the step size to satisfy the Armijo-Wolfe conditions and characterize its overall complexity.