A Communication Efficient Quasi-Newton Method for Large-scale Distributed Multi-agent Optimization

TL;DR

This paper introduces a communication-efficient quasi-Newton algorithm for large-scale distributed multi-agent convex optimization, reducing communication and computational costs while ensuring linear convergence.

Contribution

It presents a novel quasi-Newton method with consensus variables that minimizes communication and computational overhead in distributed multi-agent optimization.

Findings

Achieves global linear convergence rate in expectation.

Reduces computational complexity from O(d^3) to O(cd).

Demonstrates effectiveness with real datasets.

Abstract

We propose a communication efficient quasi-Newton method for large-scale multi-agent convex composite optimization. We assume the setting of a network of agents that cooperatively solve a global minimization problem with strongly convex local cost functions augmented with a non-smooth convex regularizer. By introducing consensus variables, we obtain a block-diagonal Hessian and thus eliminate the need for additional communication when approximating the objective curvature information. Moreover, we reduce computational costs of existing primal-dual quasi-Newton methods from $\mathcal{O}(d^3)$ to $\mathcal{O}(cd)$ by storing $c$ pairs of vectors of dimension $d$. An asynchronous implementation is presented that removes the need for coordination. Global linear convergence rate in expectation is established, and we demonstrate the merit of our algorithm numerically with real datasets.