Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning

TL;DR

This paper introduces HiCS-FL, a novel client selection method for federated learning that uses hierarchical clustering based on data heterogeneity, leading to faster convergence and reduced computation in non-IID data scenarios.

Contribution

The paper proposes HiCS-FL, a new client selection approach that estimates data heterogeneity via client updates and employs hierarchical clustering, improving efficiency and convergence in federated learning.

Findings

HiCS-FL achieves faster convergence than existing methods in non-IID settings.

It significantly reduces computational overhead compared to current client selection schemes.

The method adapts well to various heterogeneity scenarios.

Abstract

Statistical heterogeneity of data present at client devices in a federated learning (FL) system renders the training of a global model in such systems difficult. Particularly challenging are the settings where due to communication resource constraints only a small fraction of clients can participate in any given round of FL. Recent approaches to training a global model in FL systems with non-IID data have focused on developing client selection methods that aim to sample clients with more informative updates of the model. However, existing client selection techniques either introduce significant computation overhead or perform well only in the scenarios where clients have data with similar heterogeneity profiles. In this paper, we propose HiCS-FL (Federated Learning via Hierarchical Clustered Sampling), a novel client selection method in which the server estimates statistical heterogeneity of a client's data using the client's update of the network's output layer and relies on this information to cluster and sample the clients. We analyze the ability of the proposed techniques to compare heterogeneity of different datasets, and characterize convergence of the training process that deploys the introduced client selection method. Extensive experimental results demonstrate that in non-IID settings HiCS-FL achieves faster convergence than state-of-the-art FL client selection schemes. Notably, HiCS-FL drastically reduces computation cost compared to existing selection schemes and is adaptable to different heterogeneity scenarios.