A Unification and Generalization of Exact Distributed First Order Methods

TL;DR

This paper unifies and generalizes existing exact distributed first order methods, providing a new algorithm with improved convergence speed and theoretical guarantees for strongly convex problems.

Contribution

It offers a novel unifying analysis of existing methods, introduces a new weighted gradient approach, and proves global R-linear convergence for strongly convex functions.

Findings

Unified analysis of existing methods

Proposed a new weighted gradient method

Established R-linear convergence rate

Abstract

Recently, there has been significant progress in the development of distributed first order methods. (At least) two different types of methods, designed from very different perspectives, have been proposed that achieve both exact and linear convergence when a constant step size is used -- a favorable feature that was not achievable by most prior methods. In this paper, we unify, generalize, and improve convergence speed of these exact distributed first order methods. We first carry out a novel unifying analysis that sheds light on how the different existing methods compare. The analysis reveals that a major difference between the methods is on how a past dual gradient of an associated augmented Lagrangian dual function is weighted. We then capitalize on the insights from the analysis to derive a novel method -- with a tuned past gradient weighting -- that improves upon the existing methods. We establish for the proposed generalized method global R-linear convergence rate under strongly convex costs with Lipschitz continuous gradients.