PeFLL: Personalized Federated Learning by Learning to Learn

TL;DR

PeFLL introduces a novel personalized federated learning algorithm that enhances model accuracy, reduces client computation, and offers theoretical guarantees for generalization to unseen clients, using a learning-to-learn approach with embedding and hypernetworks.

Contribution

It proposes a new learning-to-learn based method for personalized federated learning that generalizes to future clients and reduces on-device computation.

Findings

Achieves state-of-the-art performance on personalized federated learning benchmarks.

Provides theoretical guarantees for generalization to unseen clients.

Reduces client-side computation and communication requirements.

Abstract

We present PeFLL, a new personalized federated learning algorithm that improves over the state-of-the-art in three aspects: 1) it produces more accurate models, especially in the low-data regime, and not only for clients present during its training phase, but also for any that may emerge in the future; 2) it reduces the amount of on-client computation and client-server communication by providing future clients with ready-to-use personalized models that require no additional finetuning or optimization; 3) it comes with theoretical guarantees that establish generalization from the observed clients to future ones. At the core of PeFLL lies a learning-to-learn approach that jointly trains an embedding network and a hypernetwork. The embedding network is used to represent clients in a latent descriptor space in a way that reflects their similarity to each other. The hypernetwork takes as input such descriptors and outputs the parameters of fully personalized client models. In combination, both networks constitute a learning algorithm that achieves state-of-the-art performance in several personalized federated learning benchmarks.