Decentralized Quasi-Newton Methods

TL;DR

This paper presents D-BFGS, a fully distributed decentralized quasi-Newton method that approximates curvature information for solving ill-conditioned optimization problems, with proven convergence and demonstrated performance benefits.

Contribution

The paper introduces D-BFGS, a novel decentralized quasi-Newton algorithm that works in asynchronous settings and converges formally, addressing limitations of first-order methods.

Findings

D-BFGS converges in both synchronous and asynchronous settings.

D-BFGS outperforms first-order methods in numerical experiments.

The method effectively handles ill-conditioned problems without second-order information.

Abstract

We introduce the decentralized Broyden-Fletcher-Goldfarb-Shanno (D-BFGS) method as a variation of the BFGS quasi-Newton method for solving decentralized optimization problems. The D-BFGS method is of interest in problems that are not well conditioned, making first order decentralized methods ineffective, and in which second order information is not readily available, making second order decentralized methods impossible. D-BFGS is a fully distributed algorithm in which nodes approximate curvature information of themselves and their neighbors through the satisfaction of a secant condition. We additionally provide a formulation of the algorithm in asynchronous settings. Convergence of D-BFGS is established formally in both the synchronous and asynchronous settings and strong performance advantages relative to first order methods are shown numerically.