An Empirical Study of Personalized Federated Learning

TL;DR

This paper benchmarks various personalized federated learning methods through comprehensive experiments, revealing no clear best method, the impact of data heterogeneity, and the surprising effectiveness of standard methods with fine-tuning.

Contribution

It provides the first extensive benchmark comparing personalized federated learning methods under consistent experimental settings.

Findings

No single method consistently outperforms others.

High data heterogeneity can improve prediction accuracy.

Standard federated learning with fine-tuning often surpasses personalized methods.

Abstract

Federated learning is a distributed machine learning approach in which a single server and multiple clients collaboratively build machine learning models without sharing datasets on clients. A challenging issue of federated learning is data heterogeneity (i.e., data distributions may differ across clients). To cope with this issue, numerous federated learning methods aim at personalized federated learning and build optimized models for clients. Whereas existing studies empirically evaluated their own methods, the experimental settings (e.g., comparison methods, datasets, and client setting) in these studies differ from each other, and it is unclear which personalized federate learning method achieves the best performance and how much progress can be made by using these methods instead of standard (i.e., non-personalized) federated learning. In this paper, we benchmark the performance of existing personalized federated learning through comprehensive experiments to evaluate the characteristics of each method. Our experimental study shows that (1) there are no champion methods, (2) large data heterogeneity often leads to high accurate predictions, and (3) standard federated learning methods (e.g. FedAvg) with fine-tuning often outperform personalized federated learning methods. We open our benchmark tool FedBench for researchers to conduct experimental studies with various experimental settings.