Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning

TL;DR

This paper introduces a comprehensive benchmarking framework for evaluating data heterogeneity measures in personalized federated learning, enabling fair comparison and guiding the selection of appropriate approaches across different non-IID data scenarios.

Contribution

It provides the first unified benchmark with six approaches and extensive experiments, offering insights into their effectiveness in various federated learning settings.

Findings

Certain approaches perform better in specific non-IID scenarios

The framework guides the choice of heterogeneity measures for FL applications

Insights help improve fairness and personalization in federated learning

Abstract

There is growing research interest in measuring the statistical heterogeneity of clients' local datasets. Such measurements are used to estimate the suitability for collaborative training of personalized federated learning (PFL) models. Currently, these research endeavors are taking place in silos and there is a lack of a unified benchmark to provide a fair and convenient comparison among various approaches in common settings. We aim to bridge this important gap in this paper. The proposed benchmarking framework currently includes six representative approaches. Extensive experiments have been conducted to compare these approaches under five standard non-IID FL settings, providing much needed insights into which approaches are advantageous under which settings. The proposed framework offers useful guidance on the suitability of various data divergence measures in FL systems. It is beneficial for keeping related research activities on the right track in terms of: (1) designing PFL schemes, (2) selecting appropriate data heterogeneity evaluation approaches for specific FL application scenarios, and (3) addressing fairness issues in collaborative model training. The code is available at https://github.com/Xiaoni-61/DH-Benchmark.