Decentralized Local Updates with Dual-Slow Estimation and Momentum-based Variance-Reduction for Non-Convex Optimization

TL;DR

This paper introduces DSE-MVR, a decentralized learning algorithm that employs dual-slow estimation and momentum-based variance reduction to improve convergence in non-convex optimization under data heterogeneity and stochastic noise.

Contribution

It proposes a novel dual-slow estimation strategy combined with variance reduction, achieving optimal convergence for non-convex problems in heterogeneous data settings.

Findings

Achieves convergence rates independent of stochastic noise for large batches.

Outperforms state-of-the-art methods in experiments.

Validates the effectiveness of dual-slow estimation in heterogeneous data environments.

Abstract

Decentralized learning (DL) has recently employed local updates to reduce the communication cost for general non-convex optimization problems. Specifically, local updates require each node to perform multiple update steps on the parameters of the local model before communicating with others. However, most existing methods could be highly sensitive to data heterogeneity (i.e., non-iid data distribution) and adversely affected by the stochastic gradient noise. In this paper, we propose DSE-MVR to address these problems.Specifically, DSE-MVR introduces a dual-slow estimation strategy that utilizes the gradient tracking technique to estimate the global accumulated update direction for handling the data heterogeneity problem; also for stochastic noise, the method uses the mini-batch momentum-based variance-reduction technique.We theoretically prove that DSE-MVR can achieve optimal convergence results for general non-convex optimization in both iid and non-iid data distribution settings. In particular, the leading terms in the convergence rates derived by DSE-MVR are independent of the stochastic noise for large-batches or large partial average intervals (i.e., the number of local update steps). Further, we put forward DSE-SGD and theoretically justify the importance of the dual-slow estimation strategy in the data heterogeneity setting. Finally, we conduct extensive experiments to show the superiority of DSE-MVR against other state-of-the-art approaches.