A Hessian inversion-free exact second order method for distributed consensus optimization

TL;DR

This paper introduces INDO, a Hessian inversion-free second-order distributed optimization method that achieves linear convergence and significantly reduces computational costs for large-scale problems.

Contribution

The paper develops INDO, a novel distributed optimization algorithm that avoids Hessian inverse calculations by approximating the Newton direction, improving efficiency for high-dimensional problems.

Findings

INDO achieves exact global linear convergence.

INDO reduces computational costs by at least an order of magnitude for large problems.

Numerical experiments show INDO's competitive iteration speed and communication efficiency.

Abstract

We consider a standard distributed consensus optimization problem where a set of agents connected over an undirected network minimize the sum of their individual local strongly convex costs. Alternating Direction Method of Multipliers ADMM and Proximal Method of Multipliers PMM have been proved to be effective frameworks for design of exact distributed second order methods involving calculation of local cost Hessians. However, existing methods involve explicit calculation of local Hessian inverses at each iteration that may be very costly when the dimension of the optimization variable is large. In this paper we develop a novel method termed INDO Inexact Newton method for Distributed Optimization that alleviates the need for Hessian inverse calculation. INDO follows the PMM framework but unlike existing work approximates the Newton direction through a generic fixed point method, e.g., Jacobi Overrelaxation, that does not involve Hessian inverses. We prove exact global linear convergence of INDO and provide analytical studies on how the degree of inexactness in the Newton direction calculation affects the overall methods convergence factor. Numerical experiments on several real data sets demonstrate that INDOs speed is on par or better as state of the art methods iterationwise hence having a comparable communication cost. At the same time, for sufficiently large optimization problem dimensions n (even at n on the order of couple of hundreds), INDO achieves savings in computational cost by at least an order of magnitude.