Symmetric Rank-$k$ Methods

TL;DR

This paper introduces symmetric rank-$k$ (SR-$k$) block quasi-Newton methods for convex optimization, demonstrating their superlinear convergence and superiority over traditional methods through theoretical analysis.

Contribution

The paper presents the first explicit superlinear convergence rate for block quasi-Newton methods and applies this to analyze and improve block BFGS and DFP methods.

Findings

SR-$k$ methods achieve superlinear convergence rate of $ig(1-k/dig)^{t(t-1)/2}$.

SR-$k$ methods outperform ordinary quasi-Newton methods in practice.

Theoretical analysis explains the faster convergence of block quasi-Newton methods.

Abstract

This paper proposes a novel class of block quasi-Newton methods for convex optimization which we call symmetric rank-$k$ (SR-$k$) methods. Each iteration of SR-$k$ incorporates the curvature information with~$k$ Hessian-vector products achieved from the greedy or random strategy. We prove that SR-$k$ methods have the local superlinear convergence rate of $\mathcal{O}\big((1-k/d)^{t(t-1)/2}\big)$ for minimizing smooth and strongly convex function, where $d$ is the problem dimension and $t$ is the iteration counter. This is the first explicit superlinear convergence rate for block quasi-Newton methods, and it successfully explains why block quasi-Newton methods converge faster than ordinary quasi-Newton methods in practice. We also leverage the idea of SR-$k$ methods to study the block BFGS and block DFP methods, showing their superior convergence rates.