Heterogeneity-Aware Client Sampling for Optimal and Efficient Federated Learning

TL;DR

This paper analyzes how communication and computation heterogeneity in federated learning cause optimization inconsistency, and introduces FedACS, a client sampling method that ensures convergence to the correct optimum while reducing costs.

Contribution

It provides the first unified theoretical analysis of heterogeneous FL and proposes FedACS, a method to eliminate objective inconsistency under diverse client capabilities.

Findings

FedACS guarantees convergence to the correct optimum at a rate of O(1/√R).

FedACS outperforms existing methods by 4.3%-36% in accuracy.

It reduces communication costs by up to 89% and computation loads by up to 105%.

Abstract

Federated learning (FL) commonly involves clients with diverse communication and computational capabilities. Such heterogeneity can significantly distort the optimization dynamics and lead to objective inconsistency, where the global model converges to an incorrect stationary point potentially far from the pursued optimum. Despite its critical impact, the joint effect of communication and computation heterogeneity has remained largely unexplored, due to the intrinsic complexity of their interaction. In this paper, we reveal the fundamentally distinct mechanisms through which heterogeneous communication and computation drive inconsistency in FL. To the best of our knowledge, this is the first unified theoretical analysis of general heterogeneous FL, offering a principled understanding of how these two forms of heterogeneity jointly distort the optimization trajectory under arbitrary choices of local solvers. Motivated by these insights, we propose Federated Heterogeneity-Aware Client Sampling, FedACS, a universal method to eliminate all types of objective inconsistency. We theoretically prove that FedACS converges to the correct optimum at a rate of $O(1/\sqrt{R})$, even in dynamic heterogeneous environments. Extensive experiments across multiple datasets show that FedACS outperforms state-of-the-art and category-specific baselines by 4.3%-36%, while reducing communication costs by 22%-89% and computation loads by 14%-105%, respectively.