Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

TL;DR

This paper introduces WKAFL, a novel asynchronous federated learning method that effectively handles unbounded stale gradients and non-IID data, improving training speed, accuracy, and stability.

Contribution

WKAFL employs adaptive gradient selection and learning rate adjustment to mitigate staleness and data heterogeneity in asynchronous FL, with proven convergence analysis.

Findings

WKAFL outperforms existing algorithms in training speed and accuracy.

The method maintains stability even with unbounded staleness.

Experimental results validate the effectiveness of WKAFL on benchmark datasets.

Abstract

Federated learning (FL) is an emerging privacy-preserving paradigm that enables multiple participants collaboratively to train a global model without uploading raw data. Considering heterogeneous computing and communication capabilities of different participants, asynchronous FL can avoid the stragglers effect in synchronous FL and adapts to scenarios with vast participants. Both staleness and non-IID data in asynchronous FL would reduce the model utility. However, there exists an inherent contradiction between the solutions to the two problems. That is, mitigating the staleness requires to select less but consistent gradients while coping with non-IID data demands more comprehensive gradients. To address the dilemma, this paper proposes a two-stage weighted $K$ asynchronous FL with adaptive learning rate (WKAFL). By selecting consistent gradients and adjusting learning rate adaptively, WKAFL utilizes stale gradients and mitigates the impact of non-IID data, which can achieve multifaceted enhancement in training speed, prediction accuracy and training stability. We also present the convergence analysis for WKAFL under the assumption of unbounded staleness to understand the impact of staleness and non-IID data. Experiments implemented on both benchmark and synthetic FL datasets show that WKAFL has better overall performance compared to existing algorithms.