Communication-efficient Decentralized Local SGD over Undirected Networks
Tiancheng Qin, S. Rasoul Etesami, C\'esar A. Uribe

TL;DR
This paper analyzes a decentralized learning algorithm that balances local computation and communication over arbitrary networks, achieving optimal convergence rates with minimal communication rounds.
Contribution
It generalizes previous analyses to arbitrary network topologies, providing bounds on the optimality gap that depend on network spectral properties.
Findings
Achieves an error of O(1/(nT)) with only Ω(n) communication rounds.
Convergence bounds depend on the spectral gap of the network.
Numerical experiments validate theoretical results.
Abstract
We consider the distributed learning problem where a network of agents seeks to minimize a global function . Agents have access to through noisy gradients, and they can locally communicate with their neighbors a network. We study the Decentralized Local SDG method, where agents perform a number of local gradient steps and occasionally exchange information with their neighbors. Previous algorithmic analysis efforts have focused on the specific network topology (star topology) where a leader node aggregates all agents' information. We generalize that setting to an arbitrary network by analyzing the trade-off between the number of communication rounds and the computational effort of each agent. We bound the expected optimality gap in terms of the number of iterates , the number of workers , and the spectral gap of the underlying network. Our main results show that by using…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStochastic Gradient Optimization Techniques · Distributed Control Multi-Agent Systems · Cooperative Communication and Network Coding
