Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity

TL;DR

This paper introduces a partially personalized federated learning framework that balances global shared parameters with local private parameters, effectively addressing data heterogeneity issues across clients.

Contribution

It proposes a novel partial personalization approach with a simple algorithm that mitigates data heterogeneity challenges in federated learning.

Findings

Breaks the curse of data heterogeneity in various training settings

Allows clients to fit their data perfectly with overpersonalized global parameters

Enhances robustness in asynchronous and Byzantine-robust training scenarios

Abstract

We present a partially personalized formulation of Federated Learning (FL) that strikes a balance between the flexibility of personalization and cooperativeness of global training. In our framework, we split the variables into global parameters, which are shared across all clients, and individual local parameters, which are kept private. We prove that under the right split of parameters, it is possible to find global parameters that allow each client to fit their data perfectly, and refer to the obtained problem as overpersonalized. For instance, the shared global parameters can be used to learn good data representations, whereas the personalized layers are fine-tuned for a specific client. Moreover, we present a simple algorithm for the partially personalized formulation that offers significant benefits to all clients. In particular, it breaks the curse of data heterogeneity in several settings, such as training with local steps, asynchronous training, and Byzantine-robust training.