FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning

TL;DR

FedSiKD introduces a similarity-based federated learning framework with knowledge distillation to effectively handle non-i.i.d. data and device constraints, achieving faster convergence and higher accuracy than existing methods.

Contribution

It presents a novel federated learning approach that uses client data similarity and knowledge distillation to improve efficiency and performance under non-i.i.d. data and device constraints.

Findings

Achieves 25 ext{ and }18\% higher accuracy on HAR and MNIST datasets.

Faster convergence with 17 ext{ and }20\% accuracy increase in early rounds.

Outperforms state-of-the-art algorithms in accuracy and convergence speed.

Abstract

In recent years, federated learning (FL) has emerged as a promising technique for training machine learning models in a decentralized manner while also preserving data privacy. The non-independent and identically distributed (non-i.i.d.) nature of client data, coupled with constraints on client or edge devices, presents significant challenges in FL. Furthermore, learning across a high number of communication rounds can be risky and potentially unsafe for model exploitation. Traditional FL approaches may suffer from these challenges. Therefore, we introduce FedSiKD, which incorporates knowledge distillation (KD) within a similarity-based federated learning framework. As clients join the system, they securely share relevant statistics about their data distribution, promoting intra-cluster homogeneity. This enhances optimization efficiency and accelerates the learning process, effectively transferring knowledge between teacher and student models and addressing device constraints. FedSiKD outperforms state-of-the-art algorithms by achieving higher accuracy, exceeding by 25\% and 18\% for highly skewed data at $\alpha = {0.1,0.5}$ on the HAR and MNIST datasets, respectively. Its faster convergence is illustrated by a 17\% and 20\% increase in accuracy within the first five rounds on the HAR and MNIST datasets, respectively, highlighting its early-stage learning proficiency. Code is publicly available and hosted on GitHub (https://github.com/SimuEnv/FedSiKD)