Optimal Algorithms for Distributed Optimization

TL;DR

This paper establishes optimal convergence rates for distributed convex optimization under various conditions, demonstrating that Nesterov's accelerated gradient can be effectively adapted for distributed settings with near-centralized efficiency.

Contribution

It provides the first comprehensive analysis of optimal convergence bounds for distributed convex optimization across multiple problem classes, extending Nesterov's acceleration to distributed algorithms.

Findings

Distributed Nesterov acceleration achieves near-centralized optimal rates.

Optimal complexity bounds are derived for strongly convex, smooth, and convex functions.

The spectral gap influences the additional cost in distributed optimization.

Abstract

In this paper, we study the optimal convergence rate for distributed convex optimization problems in networks. We model the communication restrictions imposed by the network as a set of affine constraints and provide optimal complexity bounds for four different setups, namely: the function $F(\xb) \triangleq \sum_{i=1}^{m}f_i(\xb)$ is strongly convex and smooth, either strongly convex or smooth or just convex. Our results show that Nesterov's accelerated gradient descent on the dual problem can be executed in a distributed manner and obtains the same optimal rates as in the centralized version of the problem (up to constant or logarithmic factors) with an additional cost related to the spectral gap of the interaction matrix. Finally, we discuss some extensions to the proposed setup such as proximal friendly functions, time-varying graphs, improvement of the condition numbers.