Optimality in Decentralized Optimization under Bandwidth Constraints

TL;DR

This paper establishes optimal time complexities for decentralized stochastic optimization under bandwidth constraints, introducing new algorithms and analytical tools for both homogeneous and heterogeneous settings.

Contribution

It introduces Grace SGD and Leon SGD algorithms with bounds based on min-cut/max-flow, Gomory-Hu trees, and Steiner Tree Packing, improving upon prior work.

Findings

Derived optimal time complexities for decentralized optimization.

Developed Grace SGD and Leon SGD algorithms.

Bounded complexities using graph-theoretic tools.

Abstract

We consider a realistic decentralized setup with bandwidth-constrained communication and derive optimal time complexities for non-convex stochastic parallel and asynchronous optimization (up to logarithmic factors). We develop the corresponding methods, Grace SGD and Leon SGD, for both homogeneous and heterogeneous settings. Unlike previous work, our optimal bounds are characterized in terms of min-cut/max-flow quantities and rely on tools from Gomory-Hu trees and Steiner Tree Packing problems, providing tighter and more practical complexities.