Flexible Personalized Split Federated Learning for On-Device Fine-Tuning of Foundation Models

TL;DR

This paper introduces FlexP-SFL, a flexible split federated learning approach that enables personalized fine-tuning of foundation models on resource-constrained clients, improving efficiency and accuracy.

Contribution

The paper proposes a novel flexible personalized split federated learning framework that adapts to client resources and enhances personalized model performance.

Findings

FlexP-SFL outperforms baseline models in accuracy.

FlexP-SFL improves fine-tuning efficiency.

The alignment strategy boosts global data performance.

Abstract

Fine-tuning foundation models is critical for superior performance on personalized downstream tasks, compared to using pre-trained models. Collaborative learning can leverage local clients' datasets for fine-tuning, but limited client data and heterogeneous data distributions hinder effective collaboration. To address the challenge, we propose a flexible personalized federated learning paradigm that enables clients to engage in collaborative learning while maintaining personalized objectives. Given the limited and heterogeneous computational resources available on clients, we introduce \textbf{flexible personalized split federated learning (FlexP-SFL)}. Based on split learning, FlexP-SFL allows each client to train a portion of the model locally while offloading the rest to a server, according to resource constraints. Additionally, we propose an alignment strategy to improve personalized model performance on global data. Experimental results show that FlexP-SFL outperforms baseline models in personalized fine-tuning efficiency and final accuracy.