Achieving Linear Speedup and Near-Optimal Complexity for Decentralized Optimization over Row-stochastic Networks

TL;DR

This paper develops new algorithms and theoretical bounds for decentralized optimization over row-stochastic networks, achieving linear speedup and near-optimal complexity, filling a significant gap in existing research.

Contribution

It introduces the first convergence lower bound for row-stochastic networks and proposes algorithms that attain this bound, achieving linear speedup and near-optimal complexity.

Findings

Pull-Diag-GT achieves linear speedup in decentralized optimization.

Incorporating multi-step gossip protocol attains near-optimal complexity.

Established the first convergence lower bound for row-stochastic networks.

Abstract

A key challenge in decentralized optimization is determining the optimal convergence rate and designing algorithms to achieve it. While this problem has been extensively addressed for doubly-stochastic and column-stochastic mixing matrices, the row-stochastic scenario remains unexplored. This paper bridges this gap by introducing effective metrics to capture the influence of row-stochastic mixing matrices and establishing the first convergence lower bound for decentralized learning over row-stochastic networks. However, existing algorithms fail to attain this lower bound due to two key issues: deviation in the descent direction caused by the adapted gradient tracking (GT) and instability introduced by the Pull-Diag protocol. To address descent deviation, we propose a novel analysis framework demonstrating that Pull-Diag-GT achieves linear speedup, the first such result for row-stochastic decentralized optimization. Moreover, by incorporating a multi-step gossip (MG) protocol, we resolve the instability issue and attain the lower bound, achieving near-optimal complexity for decentralized optimization over row-stochastic networks.