Local Adaptivity in Federated Learning: Convergence and Consistency

TL;DR

This paper investigates the use of adaptive optimization methods for local updates in federated learning, revealing their benefits and drawbacks, and proposes correction techniques to improve convergence and accuracy.

Contribution

It introduces correction techniques for local adaptive methods in federated learning, addressing convergence bias and enhancing training efficiency.

Findings

Local adaptive methods can accelerate convergence.

Proposed corrections mitigate solution bias.

Algorithms achieve faster convergence and higher accuracy.

Abstract

The federated learning (FL) framework trains a machine learning model using decentralized data stored at edge client devices by periodically aggregating locally trained models. Popular optimization algorithms of FL use vanilla (stochastic) gradient descent for both local updates at clients and global updates at the aggregating server. Recently, adaptive optimization methods such as AdaGrad have been studied for server updates. However, the effect of using adaptive optimization methods for local updates at clients is not yet understood. We show in both theory and practice that while local adaptive methods can accelerate convergence, they can cause a non-vanishing solution bias, where the final converged solution may be different from the stationary point of the global objective function. We propose correction techniques to overcome this inconsistency and complement the local adaptive methods for FL. Extensive experiments on realistic federated training tasks show that the proposed algorithms can achieve faster convergence and higher test accuracy than the baselines without local adaptivity.