STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learning

TL;DR

This paper introduces STEM, a stochastic two-sided momentum algorithm for federated learning, achieving near-optimal sample and communication complexities, and provides insights into trade-offs among key algorithm parameters.

Contribution

The work presents the first federated learning algorithm with near-optimal sample and communication complexities using stochastic momentum estimators, and analyzes trade-offs among update frequency, directions, and minibatch sizes.

Findings

STEM achieves $ ilde{O}( ext{}\epsilon^{-3/2})$ sample complexity.

STEM requires $ ilde{O}( ext{}\epsilon^{-1})$ communication rounds.

Trade-off curves exist between local update frequency and minibatch size.

Abstract

Federated Learning (FL) refers to the paradigm where multiple worker nodes (WNs) build a joint model by using local data. Despite extensive research, for a generic non-convex FL problem, it is not clear, how to choose the WNs' and the server's update directions, the minibatch sizes, and the local update frequency, so that the WNs use the minimum number of samples and communication rounds to achieve the desired solution. This work addresses the above question and considers a class of stochastic algorithms where the WNs perform a few local updates before communication. We show that when both the WN's and the server's directions are chosen based on a stochastic momentum estimator, the algorithm requires $\tilde{\mathcal{O}}(\epsilon^{-3/2})$ samples and $\tilde{\mathcal{O}}(\epsilon^{-1})$ communication rounds to compute an $\epsilon$-stationary solution. To the best of our knowledge, this is the first FL algorithm that achieves such {\it near-optimal} sample and communication complexities simultaneously. Further, we show that there is a trade-off curve between local update frequencies and local minibatch sizes, on which the above sample and communication complexities can be maintained. Finally, we show that for the classical FedAvg (a.k.a. Local SGD, which is a momentum-less special case of the STEM), a similar trade-off curve exists, albeit with worse sample and communication complexities. Our insights on this trade-off provides guidelines for choosing the four important design elements for FL algorithms, the update frequency, directions, and minibatch sizes to achieve the best performance.