Client2Vec: Improving Federated Learning by Distribution Shifts Aware Client Indexing

TL;DR

Client2Vec introduces a novel client indexing method to address data heterogeneity in federated learning, improving training efficiency and model performance across various scenarios.

Contribution

The paper proposes Client2Vec, a pre-training client indexing technique that enhances federated learning by mitigating distribution shifts before training begins.

Findings

Client2Vec improves client sampling, model aggregation, and local training.

Experiments show consistent performance gains across datasets and models.

The method effectively reduces the impact of data heterogeneity.

Abstract

Federated Learning (FL) is a privacy-preserving distributed machine learning paradigm. Nonetheless, the substantial distribution shifts among clients pose a considerable challenge to the performance of current FL algorithms. To mitigate this challenge, various methods have been proposed to enhance the FL training process. This paper endeavors to tackle the issue of data heterogeneity from another perspective -- by improving FL algorithms prior to the actual training stage. Specifically, we introduce the Client2Vec mechanism, which generates a unique client index for each client before the commencement of FL training. Subsequently, we leverage the generated client index to enhance the subsequent FL training process. To demonstrate the effectiveness of the proposed Client2Vec method, we conduct three case studies that assess the impact of the client index on the FL training process. These case studies encompass enhanced client sampling, model aggregation, and local training. Extensive experiments conducted on diverse datasets and model architectures show the efficacy of Client2Vec across all three case studies. Our code is avaliable at \url{https://github.com/LINs-lab/client2vec}.