Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning

TL;DR

This paper introduces a personalized federated learning method that combines locally fine-tuned models with server-learned models to improve real-time prediction accuracy in dynamic, non-stationary environments.

Contribution

It proposes a novel algorithm for personalized federated learning that adaptively combines local and federated models, supported by theoretical analysis and real dataset experiments.

Findings

Enhanced prediction accuracy in non-stationary environments

Effective model adaptation through online fine-tuning

Theoretical guarantees of the proposed method's performance

Abstract

Federated learning is renowned for its efficacy in distributed model training, ensuring that users, called clients, retain data privacy by not disclosing their data to the central server that orchestrates collaborations. Most previous work on federated learning assumes that clients possess static batches of training data. However, clients may also need to make real-time predictions on streaming data in non-stationary environments. In such dynamic environments, employing pre-trained models may be inefficient, as they struggle to adapt to the constantly evolving data streams. To address this challenge, clients can fine-tune models online, leveraging their observed data to enhance performance. Despite the potential benefits of client participation in federated online model fine-tuning, existing analyses have not conclusively demonstrated its superiority over local model fine-tuning. To bridge this gap, the present paper develops a novel personalized federated learning algorithm, wherein each client constructs a personalized model by combining a locally fine-tuned model with multiple federated models learned by the server over time. Theoretical analysis and experiments on real datasets corroborate the effectiveness of this approach for real-time predictions and federated model fine-tuning.