Block BFGS Methods

TL;DR

This paper introduces Block BFGS, a quasi-Newton optimization method with block updates, demonstrating its global and superlinear convergence properties for convex functions and convergence to stationary points for non-convex functions, supported by numerical experiments.

Contribution

The paper presents a novel Block BFGS method with convergence guarantees for both convex and non-convex functions, extending the applicability of quasi-Newton methods.

Findings

Block BFGS converges globally and superlinearly for convex functions.

Block BFGS converges to stationary points for non-convex functions with bounded Hessian.

Numerical experiments show competitive performance with BFGS and gradient descent.

Abstract

We introduce a quasi-Newton method with block updates called Block BFGS. We show that this method, performed with inexact Armijo-Wolfe line searches, converges globally and superlinearly under the same convexity assumptions as BFGS. We also show that Block BFGS is globally convergent to a stationary point when applied to non-convex functions with bounded Hessian, and discuss other modifications for non-convex minimization. Numerical experiments comparing Block BFGS, BFGS and gradient descent are presented.