Task-Centric Personalized Federated Fine-Tuning of Language Models

TL;DR

FedRouter is a clustering-based personalized federated learning method that creates task-specific models to improve robustness and generalization in federated language model training.

Contribution

It introduces a novel clustering approach with adapters and an evaluation router to enhance task-centric personalization and robustness in federated learning.

Findings

FedRouter outperforms existing methods by up to 6.1% under task interference.

Achieves up to 136% relative improvement in generalization.

Demonstrates strong resilience in challenging federated learning scenarios.

Abstract

Federated Learning (FL) has emerged as a promising technique for training language models on distributed and private datasets of diverse tasks. However, aggregating models trained on heterogeneous tasks often degrades the overall performance of individual clients. To address this issue, Personalized FL (pFL) aims to create models tailored for each client's data distribution. Although these approaches improve local performance, they usually lack robustness in two aspects: (i) generalization: when clients must make predictions on unseen tasks, or face changes in their data distributions, and (ii) intra-client tasks interference: when a single client's data contains multiple distributions that may interfere with each other during local training. To tackle these two challenges, we propose FedRouter, a clustering-based pFL that builds specialized models for each task rather than for each client. FedRouter uses adapters to personalize models by employing two clustering mechanisms to associate adapters with specific tasks. A local clustering that associate adapters with task data samples and a global one that associates similar adapters from different clients to construct task-centric personalized models. Additionally, we propose an evaluation router mechanism that routes test samples to the best adapter based on the created clusters. Experiments comparing our method with existing approaches across a multitask dataset, FedRouter demonstrate strong resilience in these challenging scenarios performing up to 6.1% relatively better under tasks interference and up to 136% relative improvement under generalization evaluation.