Communication-efficient Decentralized Local SGD over Undirected Networks

TL;DR

This paper analyzes a decentralized learning algorithm that balances local computation and communication over arbitrary networks, achieving optimal convergence rates with minimal communication rounds.

Contribution

It generalizes previous analyses to arbitrary network topologies, providing bounds on the optimality gap that depend on network spectral properties.

Findings

Achieves an error of O(1/(nT)) with only Ω(n) communication rounds.

Convergence bounds depend on the spectral gap of the network.

Numerical experiments validate theoretical results.

Abstract

We consider the distributed learning problem where a network of $n$ agents seeks to minimize a global function $F$. Agents have access to $F$ through noisy gradients, and they can locally communicate with their neighbors a network. We study the Decentralized Local SDG method, where agents perform a number of local gradient steps and occasionally exchange information with their neighbors. Previous algorithmic analysis efforts have focused on the specific network topology (star topology) where a leader node aggregates all agents' information. We generalize that setting to an arbitrary network by analyzing the trade-off between the number of communication rounds and the computational effort of each agent. We bound the expected optimality gap in terms of the number of iterates $T$, the number of workers $n$, and the spectral gap of the underlying network. Our main results show that by using only $R=\Omega(n)$ communication rounds, one can achieve an error that scales as $O({1}/{nT})$, where the number of communication rounds is independent of $T$ and only depends on the number of agents. Finally, we provide numerical evidence of our theoretical results through experiments on real and synthetic data.