A Decentralized Second-Order Method with Exact Linear Convergence Rate for Consensus Optimization

TL;DR

This paper introduces ESOM, a decentralized second-order optimization method with exact linear convergence, suitable for consensus problems, outperforming existing methods in efficiency and convergence speed.

Contribution

The paper presents ESOM, a novel second-order method for decentralized consensus optimization that achieves exact linear convergence using a quadratic approximation and Taylor series.

Findings

ESOM converges linearly under strong convexity and Lipschitz gradients.

Numerical results show ESOM's advantages over existing decentralized methods.

ESOM effectively solves least squares and logistic regression problems.

Abstract

This paper considers decentralized consensus optimization problems where different summands of a global objective function are available at nodes of a network that can communicate with neighbors only. The proximal method of multipliers is considered as a powerful tool that relies on proximal primal descent and dual ascent updates on a suitably defined augmented Lagrangian. The structure of the augmented Lagrangian makes this problem non-decomposable, which precludes distributed implementations. This problem is regularly addressed by the use of the alternating direction method of multipliers. The exact second order method (ESOM) is introduced here as an alternative that relies on: (i) The use of a separable quadratic approximation of the augmented Lagrangian. (ii) A truncated Taylor's series to estimate the solution of the first order condition imposed on the minimization of the quadratic approximation of the augmented Lagrangian. The sequences of primal and dual variables generated by ESOM are shown to converge linearly to their optimal arguments when the aggregate cost function is strongly convex and its gradients are Lipschitz continuous. Numerical results demonstrate advantages of ESOM relative to decentralized alternatives in solving least squares and logistic regression problems.