Multi-Model Federated Learning

TL;DR

This paper extends federated learning to simultaneously train multiple unrelated models across clients, proposing new client assignment policies that outperform or match traditional single-model approaches.

Contribution

Introduces a multi-model federated learning framework with novel client selection policies based on local losses, enhancing performance over standard FedAvg.

Findings

Multi-model policies outperform single-model FedAvg in experiments.

Client selection based on local losses improves training efficiency.

Proposed policies are effective on synthetic and real-world data.

Abstract

Federated learning is a form of distributed learning with the key challenge being the non-identically distributed nature of the data in the participating clients. In this paper, we extend federated learning to the setting where multiple unrelated models are trained simultaneously. Specifically, every client is able to train any one of M models at a time and the server maintains a model for each of the M models which is typically a suitably averaged version of the model computed by the clients. We propose multiple policies for assigning learning tasks to clients over time. In the first policy, we extend the widely studied FedAvg to multi-model learning by allotting models to clients in an i.i.d. stochastic manner. In addition, we propose two new policies for client selection in a multi-model federated setting which make decisions based on current local losses for each client-model pair. We compare the performance of the policies on tasks involving synthetic and real-world data and characterize the performance of the proposed policies. The key take-away from our work is that the proposed multi-model policies perform better or at least as good as single model training using FedAvg.