Variance-Reduced Decentralized Stochastic Optimization with Gradient Tracking -- Part II: GT-SVRG

TL;DR

This paper introduces GT-SVRG, a decentralized gradient tracking algorithm using variance reduction, which matches the convergence rate of GT-SAGA for smooth, strongly-convex functions, enhancing large-scale empirical risk minimization.

Contribution

It develops GT-SVRG, a novel decentralized variance-reduction method based on SVRG, extending the previous GT-SAGA approach and analyzing its convergence properties.

Findings

GT-SVRG achieves convergence rates comparable to GT-SAGA.

The algorithm outperforms existing methods in certain regimes.

Trade-offs between GT-SVRG and GT-SAGA depend on problem settings.

Abstract

Decentralized stochastic optimization has recently benefited from gradient tracking methods \cite{DSGT_Pu,DSGT_Xin} providing efficient solutions for large-scale empirical risk minimization problems. In Part I \cite{GT_SAGA} of this work, we develop \textbf{\texttt{GT-SAGA}} that is based on a decentralized implementation of SAGA \cite{SAGA} using gradient tracking and discuss regimes of practical interest where \textbf{\texttt{GT-SAGA}} outperforms existing decentralized approaches in terms of the total number of local gradient computations. In this paper, we describe \textbf{\texttt{GT-SVRG}} that develops a decentralized gradient tracking based implementation of SVRG \cite{SVRG}, another well-known variance-reduction technique. We show that the convergence rate of \textbf{\texttt{GT-SVRG}} matches that of \textbf{\texttt{GT-SAGA}} for smooth and strongly-convex functions and highlight different trade-offs between the two algorithms in various settings.