Optimal Algorithms for Non-Smooth Distributed Optimization in Networks

TL;DR

This paper introduces optimal algorithms for distributed non-smooth convex optimization, demonstrating fast convergence rates and minimal communication impact, under different regularity assumptions.

Contribution

It presents the first optimal first-order decentralized algorithm (MSPD) for local regularity and a new distributed smoothing method (DRS) for global regularity, both with proven optimal or near-optimal rates.

Findings

MSPD achieves optimal convergence rate with communication network impact only in second-order term.

DRS is within a $d^{1/4}$ factor of the optimal rate for global regularity.

Communication effects diminish rapidly even for non-strongly convex functions.

Abstract

In this work, we consider the distributed optimization of non-smooth convex functions using a network of computing units. We investigate this problem under two regularity assumptions: (1) the Lipschitz continuity of the global objective function, and (2) the Lipschitz continuity of local individual functions. Under the local regularity assumption, we provide the first optimal first-order decentralized algorithm called multi-step primal-dual (MSPD) and its corresponding optimal convergence rate. A notable aspect of this result is that, for non-smooth functions, while the dominant term of the error is in $O(1/\sqrt{t})$, the structure of the communication network only impacts a second-order term in $O(1/t)$, where $t$ is time. In other words, the error due to limits in communication resources decreases at a fast rate even in the case of non-strongly-convex objective functions. Under the global regularity assumption, we provide a simple yet efficient algorithm called distributed randomized smoothing (DRS) based on a local smoothing of the objective function, and show that DRS is within a $d^{1/4}$ multiplicative factor of the optimal convergence rate, where $d$ is the underlying dimension.