Communication-Efficient Federated Learning With Data and Client Heterogeneity

TL;DR

This paper introduces a new federated learning algorithm that handles data heterogeneity, client asynchrony, and communication constraints, achieving fast convergence comparable to FedAvg in large-scale, practical settings.

Contribution

It presents the first federated averaging variant supporting data heterogeneity, partial asynchrony, and communication compression with rigorous convergence analysis.

Findings

Ensures fast convergence in large-scale federated tasks.

Outperforms prior quantized and asynchronous methods.

Effective in setups with up to 300 nodes.

Abstract

Federated Learning (FL) enables large-scale distributed training of machine learning models, while still allowing individual nodes to maintain data locally. However, executing FL at scale comes with inherent practical challenges: 1) heterogeneity of the local node data distributions, 2) heterogeneity of node computational speeds (asynchrony), but also 3) constraints in the amount of communication between the clients and the server. In this work, we present the first variant of the classic federated averaging (FedAvg) algorithm which, at the same time, supports data heterogeneity, partial client asynchrony, and communication compression. Our algorithm comes with a novel, rigorous analysis showing that, in spite of these system relaxations, it can provide similar convergence to FedAvg in interesting parameter regimes. Experimental results in the rigorous LEAF benchmark on setups of up to 300 nodes show that our algorithm ensures fast convergence for standard federated tasks, improving upon prior quantized and asynchronous approaches.