On the Role of Server Momentum in Federated Learning

TL;DR

This paper introduces a comprehensive framework for server momentum in federated learning, addressing convergence issues caused by heterogeneity and enabling flexible hyperparameter scheduling and asynchronous local updates.

Contribution

It proposes a general, flexible server momentum framework with hyperparameter scheduling and supports heterogeneity, backed by rigorous convergence analysis and extensive experiments.

Findings

Improved convergence under heterogeneity

Effective hyperparameter scheduling enhances performance

Framework outperforms existing methods in experiments

Abstract

Federated Averaging (FedAvg) is known to experience convergence issues when encountering significant clients system heterogeneity and data heterogeneity. Server momentum has been proposed as an effective mitigation. However, existing server momentum works are restrictive in the momentum formulation, do not properly schedule hyperparameters and focus only on system homogeneous settings, which leaves the role of server momentum still an under-explored problem. In this paper, we propose a general framework for server momentum, that (a) covers a large class of momentum schemes that are unexplored in federated learning (FL), (b) enables a popular stagewise hyperparameter scheduler, (c) allows heterogeneous and asynchronous local computing. We provide rigorous convergence analysis for the proposed framework. To our best knowledge, this is the first work that thoroughly analyzes the performances of server momentum with a hyperparameter scheduler and system heterogeneity. Extensive experiments validate the effectiveness of our proposed framework.