Accelerating Distributed Optimization: A Primal-Dual Perspective on Local Steps

TL;DR

This paper introduces a primal-dual method for distributed optimization that inherently incorporates local updates, achieving linear convergence and nearly optimal communication complexity without large minibatches across various convexity settings.

Contribution

It demonstrates that a primal-dual approach with local updates can achieve optimal communication complexity and linear convergence in distributed optimization, even for non-strongly convex objectives.

Findings

Achieves linear convergence in strongly convex settings.

Requires no inter-agent communication during local primal updates.

Attains nearly optimal communication complexity across different convexity regimes.

Abstract

In distributed machine learning, efficient training across multiple agents with different data distributions poses significant challenges. Even with a centralized coordinator, current algorithms that achieve optimal communication complexity typically require either large minibatches or compromise on gradient complexity. In this work, we tackle both centralized and decentralized settings across strongly convex, convex, and nonconvex objectives. We first demonstrate that a basic primal-dual method, (Accelerated) Gradient Ascent Multiple Stochastic Gradient Descent (GA-MSGD), applied to the Lagrangian of distributed optimization inherently incorporates local updates, because the inner loops of running Stochastic Gradient Descent on the primal variable require no inter-agent communication. Notably, for strongly convex objectives, (Accelerated) GA-MSGD achieves linear convergence in communication rounds despite the Lagrangian being only linear in the dual variables. This is due to a structural property where the dual variable is confined to the span of the coupling matrix, rendering the dual problem strongly concave. When integrated with the Catalyst framework, our approach achieves nearly optimal communication complexity across various settings without the need for minibatches.