TL;DR
Push-SAGA is a novel decentralized stochastic optimization algorithm that combines variance reduction, gradient tracking, and push-sum consensus to achieve linear convergence over directed graphs, outperforming previous methods.
Contribution
It introduces Push-SAGA, the first linearly-convergent stochastic algorithm for strongly convex problems over arbitrary directed graphs.
Findings
Achieves linear convergence to the exact solution.
Attains linear speed-up over centralized algorithms.
Converges at a network-independent rate.
Abstract
In this paper, we propose Push-SAGA, a decentralized stochastic first-order method for finite-sum minimization over a directed network of nodes. Push-SAGA combines node-level variance reduction to remove the uncertainty caused by stochastic gradients, network-level gradient tracking to address the distributed nature of the data, and push-sum consensus to tackle the challenge of directed communication links. We show that Push-SAGA achieves linear convergence to the exact solution for smooth and strongly convex problems and is thus the first linearly-convergent stochastic algorithm over arbitrary strongly connected directed graphs. We also characterize the regimes in which Push-SAGA achieves a linear speed-up compared to its centralized counterpart and achieves a network-independent convergence rate. We illustrate the behavior and convergence properties of Push-SAGA with the help of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
