Distributed Stochastic Momentum Tracking with Local Updates: Achieving Optimal Communication and Iteration Complexities

TL;DR

This paper introduces Local Momentum Tracking (LMT), a distributed stochastic gradient method that combines local updates, momentum tracking, and acceleration techniques to optimize communication and iteration complexities in networked optimization.

Contribution

LMT is the first distributed stochastic gradient method with local updates that achieves optimal communication and iteration complexities under various conditions.

Findings

LMT achieves linear speedup with respect to local updates and agents.

LMT attains optimal communication complexity with enough local updates.

LMT achieves optimal iteration complexity with moderate local updates.

Abstract

We propose Local Momentum Tracking (LMT), a novel distributed stochastic gradient method for solving distributed optimization problems over networks. To reduce communication overhead, LMT enables each agent to perform multiple local updates between consecutive communication rounds. Specifically, LMT integrates local updates with the momentum tracking strategy and the Loopless Chebyshev Acceleration (LCA) technique. We demonstrate that LMT achieves linear speedup with respect to the number of local updates as well as the number of agents for minimizing smooth objective functions with and without the Polyak-{\L}ojasiewicz (PL) condition. Notably, with sufficiently many local updates $Q\geq Q^*$, LMT attains the optimal communication complexity. For a moderate number of local updates $Q\in[1,Q^*]$, LMT achieves the optimal iteration complexity. To our knowledge, LMT is the first distributed stochastic gradient method with local updates that enjoys such properties.