Communication Efficient Distributed Optimization using an Approximate Newton-type Method

TL;DR

This paper introduces a new Newton-type method for distributed optimization that converges quickly and scales well with data size, especially effective for stochastic learning problems.

Contribution

It proposes a novel Newton-type algorithm tailored for distributed stochastic optimization with proven linear convergence that improves with data size.

Findings

Achieves linear convergence rate for quadratic objectives

Requires an essentially constant number of iterations as data size grows

Outperforms one-shot averaging and ADMM in experiments

Abstract

We present a novel Newton-type method for distributed optimization, which is particularly well suited for stochastic optimization and learning problems. For quadratic objectives, the method enjoys a linear rate of convergence which provably \emph{improves} with the data size, requiring an essentially constant number of iterations under reasonable assumptions. We provide theoretical and empirical evidence of the advantages of our method compared to other approaches, such as one-shot parameter averaging and ADMM.