Comparative Evaluation of Clustered Federated Learning Methods

TL;DR

This paper systematically evaluates two state-of-the-art Clustered Federated Learning algorithms across various data heterogeneity scenarios, providing insights into their performance and the impact of data distribution differences.

Contribution

It introduces a taxonomy for data heterogeneity in federated learning and assesses CFL algorithms' performance across these scenarios using multiple datasets.

Findings

CFL algorithms perform variably depending on heterogeneity type

The proposed taxonomy clarifies heterogeneity scenarios in FL

Performance metrics reveal strengths and limitations of CFL methods

Abstract

Over recent years, Federated Learning (FL) has proven to be one of the most promising methods of distributed learning which preserves data privacy. As the method evolved and was confronted to various real-world scenarios, new challenges have emerged. One such challenge is the presence of highly heterogeneous (often referred as non-IID) data distributions among participants of the FL protocol. A popular solution to this hurdle is Clustered Federated Learning (CFL), which aims to partition clients into groups where the distribution are homogeneous. In the literature, state-of-the-art CFL algorithms are often tested using a few cases of data heterogeneities, without systematically justifying the choices. Further, the taxonomy used for differentiating the different heterogeneity scenarios is not always straightforward. In this paper, we explore the performance of two state-of-theart CFL algorithms with respect to a proposed taxonomy of data heterogeneities in federated learning (FL). We work with three image classification datasets and analyze the resulting clusters against the heterogeneity classes using extrinsic clustering metrics. Our objective is to provide a clearer understanding of the relationship between CFL performances and data heterogeneity scenarios.