FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure

TL;DR

FedGroup introduces a novel clustered federated learning framework that groups clients based on optimization similarities and a data-driven distance measure, significantly improving accuracy and scalability in non-IID settings.

Contribution

The paper proposes FedGroup, a new CFL framework with client clustering based on optimization directions and a data-driven measure, enhancing efficiency and performance.

Findings

FedGroup improves test accuracy by +14.1% on FEMNIST.

FedGroup achieves +3.4% accuracy on Sentiment140.

FedGroup outperforms existing CFL methods on multiple datasets.

Abstract

Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally. Unlike the centralized training setting, the non-IID and imbalanced (statistical heterogeneity) training data of FL is distributed in the federated network, which will increase the divergences between the local models and global model, further degrading performance. In this paper, we propose a novel clustered federated learning (CFL) framework FedGroup, in which we 1) group the training of clients based on the similarities between the clients' optimization directions for high training performance; 2) construct a new data-driven distance measure to improve the efficiency of the client clustering procedure. 3) implement a newcomer device cold start mechanism based on the auxiliary global model for framework scalability and practicality. FedGroup can achieve improvements by dividing joint optimization into groups of sub-optimization and can be combined with FL optimizer FedProx. The convergence and complexity are analyzed to demonstrate the efficiency of our proposed framework. We also evaluate FedGroup and FedGrouProx (combined with FedProx) on several open datasets and made comparisons with related CFL frameworks. The results show that FedGroup can significantly improve absolute test accuracy by +14.1% on FEMNIST compared to FedAvg. +3.4% on Sentiment140 compared to FedProx, +6.9% on MNIST compared to FeSEM.