FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

TL;DR

FDLoRA introduces a personalized federated learning framework using dual LoRA tuning for large language models, enabling effective knowledge sharing across clients while reducing costs and improving performance.

Contribution

It proposes a novel dual LoRA module approach in federated learning for LLMs, enhancing personalization and efficiency over existing methods.

Findings

Outperforms six baselines in various metrics

Improves robustness and stability of federated LLM training

Reduces communication and computation costs

Abstract

Large language models (LLMs) have emerged as important components across various fields, yet their training requires substantial computation resources and abundant labeled data. It poses a challenge to robustly training LLMs for individual users (clients). To tackle this challenge, the intuitive idea is to introduce federated learning (FL), which can collaboratively train models on distributed private data. However, existing methods suffer from the challenges of data heterogeneity, system heterogeneity, and model size, resulting in suboptimal performance and high costs. In this work, we proposed a variant of personalized federated learning (PFL) framework, namely FDLoRA, which allows the client to be a single device or a cluster and adopts low-rank adaptation (LoRA) tuning. FDLoRA sets dual LoRA modules on each client to capture personalized and global knowledge, respectively, and only the global LoRA module uploads parameters to the central server to aggregate cross-client knowledge. Finally, an adaptive fusion approach is employed to combine the parameters of the dual LoRAs. This enables FDLoRA to make effective use of private data distributed across different clients, thereby improving performance on the client without incurring high communication and computing costs. We conducted extensive experiments in two practice scenarios. The results demonstrate that FDLoRA outperforms six baselines in terms of performance, stability, robustness, computation cost, and communication cost.