Variance reduced stochastic optimization over directed graphs with row and column stochastic weights

TL;DR

This paper introduces AB-SAGA, a distributed stochastic optimization algorithm that achieves linear convergence over directed graphs using variance reduction and linear consensus updates with row and column stochastic weights.

Contribution

AB-SAGA is the first method combining variance reduction and linear consensus updates with row and column stochastic weights for directed graphs.

Findings

AB-SAGA converges linearly with a constant step-size.

The method achieves linear speed-up over centralized algorithms under certain conditions.

Numerical experiments confirm convergence for both convex and nonconvex problems.

Abstract

This paper proposes AB-SAGA, a first-order distributed stochastic optimization method to minimize a finite-sum of smooth and strongly convex functions distributed over an arbitrary directed graph. AB-SAGA removes the uncertainty caused by the stochastic gradients using a node-level variance reduction and subsequently employs network-level gradient tracking to address the data dissimilarity across the nodes. Unlike existing methods that use the nonlinear push-sum correction to cancel the imbalance caused by the directed communication, the consensus updates in AB-SAGA are linear and uses both row and column stochastic weights. We show that for a constant step-size, AB-SAGA converges linearly to the global optimal. We quantify the directed nature of the underlying graph using an explicit directivity constant and characterize the regimes in which AB-SAGA achieves a linear speed-up over its centralized counterpart. Numerical experiments illustrate the convergence of AB-SAGA for strongly convex and nonconvex problems.