Quasi-Newton Methods for Saddle Point Problems and Beyond

TL;DR

This paper introduces novel quasi-Newton methods with superlinear convergence for solving saddle point problems, extending to nonlinear equations, and providing explicit convergence rates based on Hessian estimates.

Contribution

It proposes greedy and random Broyden family updates for saddle point problems with explicit superlinear convergence analysis, a novel approach in this context.

Findings

Proposed algorithms achieve superlinear convergence rates.

BFGS-type and SR1-type updates improve convergence speed.

Algorithms extend to general nonlinear equations with similar convergence guarantees.

Abstract

This paper studies quasi-Newton methods for solving strongly-convex-strongly-concave saddle point problems (SPP). We propose greedy and random Broyden family updates for SPP, which have explicit local superlinear convergence rate of ${\mathcal O}\big(\big(1-\frac{1}{n\kappa^2}\big)^{k(k-1)/2}\big)$, where $n$ is dimensions of the problem, $\kappa$ is the condition number and $k$ is the number of iterations. The design and analysis of proposed algorithm are based on estimating the square of indefinite Hessian matrix, which is different from classical quasi-Newton methods in convex optimization. We also present two specific Broyden family algorithms with BFGS-type and SR1-type updates, which enjoy the faster local convergence rate of $\mathcal O\big(\big(1-\frac{1}{n}\big)^{k(k-1)/2}\big)$. Additionally, we extend our algorithms to solve general nonlinear equations and prove it enjoys the similar convergence rate.